RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
Authors: Haolin Li
Tianjie Dai
Zhe Chen
Siyuan Du
Jiangchao Yao
Ya Zhang
Yanfeng Wang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Fusion
Medical Imaging
Interpretability
Disease Diagnosis
Primary area: machine_learning_for_sciences
TL;DR: We propose Retrieval-Augmented Diagnosis (RAD), a holistic method that explicitly integrates disease-centered medical knowledge into multimodal diagnosis models.

Abstract: Clinical diagnosis is a highly specialized discipline requiring both domain expertise and strict adherence to rigorous guidelines. 
While current AI-driven medical research predominantly focuses on knowledge graphs or natural text pretraining paradigms to incorporate medical knowledge, these approaches primarily rely on implicitly encoded knowledge within model parameters, neglecting task-specific knowledge required by diverse downstream tasks.
To address this limitation, we propose **R**etrieval-**A**ugmented **D**iagnosis (RAD), a novel framework that explicitly injects external knowledge into multimodal models directly on downstream tasks.
Specifically, RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guideline-enhanced contrastive loss that constrains the latent distance between multi-modal features and guideline knowledge, and the dual transformer decoder that employs guidelines as queries to steer cross-modal fusion, aligning the models with clinical diagnostic workflows from guideline acquisition to feature extraction and decision-making.
Moreover, recognizing the lack of quantitative evaluation of interpretability for multimodal diagnostic models, we introduce a set of criteria to assess the interpretability from both image and text perspectives. Extensive evaluations across four datasets with different anatomies demonstrate RAD's generalizability, achieving state-of-the-art performance. Furthermore, RAD enables the model to concentrate more precisely on abnormal regions and critical indicators, ensuring evidence-based, trustworthy diagnosis. Our code is available at https://github.com/tdlhl/RAD.

RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
Haolin Li Tianjie Dai Zhe Chen Siyuan Du
Jiangchao 2a3 Ya Zhang2.4. Yanfeng Wang
College of Computer Science and Artificial Intelligence, Fudan University "Shanghai AI Laboratory *CMIC, Shanghai Jiao Tong University Schoool of Artificial Intelligence, Shanghai Jiao Tong University
Institute of Artificial Intelligence for Medicine, Shanghai Jiao Tong University School of Medicine
Abstract
Clinical diagnosis is a highly specialized discipline requiring both domain exper- tise and strict adherence to rigorous guidelines. While current AI-driven medical research predominantly focuses on knowledge graphs natural text pretraining paradigms to incorporate medical knowledge, these approaches primarily rely on implicitly encoded knowledge within model parameters, neglecting task-specific knowledge required by diverse downstream tasks. To address this limitation, we propose
Diagnosis (RAD), novel framework that explicitly injects external knowledge into multimodal models directly on downstream tasks. Specifically, RAD operates through three key mechanisms: retrieval and refine- ment of disease-centered knowledge from multiple medical sources, guideline- enhanced contrastive loss that constrains the latent distance between multi-modal features and guideline knowledge, and the dual transformer decoder that employs guidelines as queries steer cross-modal fusion, aligning the models with clin- ical diagnostic workflows from guideline acquisition to feature extraction and decision-man -aking.. Moreover, recognizing the lack of quantitative evaluation of inter- pretability for multimodal diagnostic models, we introduce set of criteria to assess the interpretability from both image and text perspectives. Extensive evaluations across four datasets with different anatomies demonstrate RAD's generalizability, achieving state-of-the-arr performance. Furthermore, RAD enables the model concentrate more precisely abnormal regions and critical indicators, ensuring wvidence-based trustworthy diagnosis. Our code is available at this repository
Introduction
The rapid development of multimodal learning [37 46) has revolutionized numerous fields by enabling models to process and integrate diverse data types, including images, texts, audio, and structured records 4 II (77. Biomedical applications particularly benefit from these advancements, given that diagnostic workflows inherently depend multimodal evidence, ranging from radiographic imaging and reports to electronic health records (EHR) [66] 52 75). For instance, radiologists integrate X-ray or MRI scans with textual pathology reports, while clinicians combine electronic health records, vital signs, and even genomic data to form comprehensive patient profiles. Accordingly. recent research efforts have increasingly focused on developing multimodal architectures tailored healthcare challenges, seeking to enhance diagnostic precision through cross-modal synergy [ 32 61 68
While these approaches demonstrate significant progress in integrating data from different modalities, they often overlook the foundational principles governing clinical decision-making.
"Equal Contribution. i Correspondence to Jiangchao Yao (Sunarker@ (ww.eewsssu.uuueee and Yanfong Wang (wangyanfeng62226 @ www.eeeeee.eeu..u..uee.o.oeoe sjju.edu.cn).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RAD: Hollstic Injection
[CLS] eeportt Ending compesed perssous miboomphh,
GCT Si < empprt findings compare to the medlonghh there méorant change right hemideghhogg with billlees) relatively estendive jarnecyy myyma opacities
relerant change.. elevation of the night beradd with bilateral relatively on onteneeve pomnehymmal opacities no pieural effusions no pretmothorax chr the pelient is 64 year- old (emale the leb reulls Minire aminccmpnntonn (alt) with valur
Krnwledge
pleural effusioas no gre greunotiones. < eht bi the po bent 64 vear- old (enale the leb resulss include Danine aminstrandenma alt) with a value
Guidelme:
Enoode Aliim
Docisoon
espernde annnn ened ast wi valle 0. beoophils with value of 0, [[lliinan total with a value of 0 calculated looal 002 with
albllize plasppanase
Anitstrasttemse
I L.Alazine
sey indrcatons
of 5. dlaaiie phosppalase with value of O. smminrrnraoeee
lasopials
value
belimsinn toral
valuo 6
2. 2.Aegarits
with a value of S. calculated toka ond with value of 4. It homotocrt with a volue of
Lagut
Prstmin
Paramater Prtor
valure of 0 benationn will value O. lsetate with vallne of 0, lactate dely dehydrogensee value
3. Senum
Amiisttrsleeooe
lastane with valln of 0. lastate debyyroggnase
agmtmmt
Encoder
of phoophate
Stinhn
( with a valtae of 0, phooppare with vallo
[Gomact coun both vollne
Piausot Courc
Stat coum
of
Prior Methods
pobasttum with a varm of 3..0 [SEP]
polassium willi
(SEP)
Singloieew Iegiction
w/o RAD
RAD
(a)
(b)
The design motivation of RAD. Left: Previous methods mostly focus on enhancing a single aspect of the diagnostic process, whereas our approach is holistic. Right: Visualization of model attention to textual content. Color intensity reflects attention magnitude, with red highlighting disease-critical indicators mentioned in the guideline. Models without explicit knowledge guidance exhibit limited focus on key indicators, whereas our model can make evidence-based diagnoses.
Figure 1:
Medical analysis fundamentally differs from natural scene understanding in its strict adherence to evidence-based principles, relying heavily structured protocols [31 [5)] Clinical decisions must be grounded in standardized diagnostic criteria derived from patient-specific symptoms, imaging findings, and laboratory results. This inherent rigor poses a critical challenge for black-box neural networks, whose vague decision-making mechanisms hinder trustworthy and practical deployment in clinical settings 19] 4 [53. Consequently, there has been growing interest in integrating medical knowledge
into Al models to simultaneously improve model performance and interpretability
g 9 73).
Existing approaches primarily focus on knowledge injection during pretraining phases. Researchers enhance the text encoders by pretraining them on large-scale medical corpora 30 47 or leveraging
structured knowledge graphs imbue models with semantic relationships between biomedical enti- ties 36 48). While effective in expanding the semantic coverage of text encoders, these approaches often struggle to explicitly integrate ine-grained knowledge tailored for downstream diagnostic tasks. To this end, we argue that effective knowledge integration requires task-centric, holistic alignment with disease-level knowledge throughout the entire diagnostic pipeline. As illustrated in Figure D a), our framework systematically integrates refined knowledge to guide input augmentation, feature extraction, and modality fusion, contrasting with prior methods confined to single perspective. Fig- ure [Tb) presents case of the model's 's attention distribution over the input text. The previous model fails to concentrate on critical indicators. but focuses on obvious disease terms in the reports. While the RAD model can not only attend to these terms but also consider other quidelin-rrcommeent key indicators. The explicit knowledge guidance enables RAD to prioritize critical indicators tailored for the current disease, making trustworthy diagnoses aligned with clinical standards.
In this paper, we propose holistic cnowledge-injectiie framework RAD, which operates through three synergistic components spanning the entire diagnostic workflow. RAD begins by retrieving and refining disease-specific guidelines from diverse sources, flexibly adapting downstream tasks in different scenarios. We then employ two modality encoders coupled with seidellnne-nnancee contrastive loss that explicitly aligns the modality-specific feature with the corresponding disease- guideline prototypes in the joint latent space. A dual decoder network is further developed to steer the cross-modal fusion process, which simultaneously incorporates discase labels and their corresponding guidelines to interact with fused multimodal features for final predictions. Through this systematic knowledge infusion paradigm, our framework achieves performance gains while establishing a traceable decision pathway grounded in clinical guidelines a critical step toward clinically actionable AI. We further establish an evaluation system for model interpretability. which quantitatively assesses the model's 's adherence to guidelines through both textual indicators and visual localization. Combined with qualitative visualization, this system provides measurable evidence that RAD's decisions are driven by the injected knowledge. In summary, our contributions are three-fold:
We propose RAD to systematically inject external medical knowledge into multimodal diagnosis models. RAD incorporates a guideline-enhanced loss and a dual-decoder structure to explicitly steer multimodal feature extraction and cross-modal fusion with discase-guidelin prototypes.
2

---

A dual-axis evaluation system for the interpretability of diagnosis models is developed. formulating both textual and visual metrics. This system enables quantitative analyses of the model's adherence to clinical guidelines, demonstrating the transparency and explainabilityy brought by RAD.
We aligned MIMIC-CXR 28 and MIMIC-IV [29 to construct the MIMIC-ICDS! dataset, covering three modalities with 53 types of disease. Extensive experiments on our dataset and three other public datasets demonstrate the superiority of RAD over SOTA baselines across various metrics.
2 Related Work
2.1 Multimodal Learning Medicine
Recent years have witnessed significant advancements in the field of multimodal learning, with models such CLIP [46). BLIP (33, and LLaVA (37 exhibiting remarkable capabilities in natural domains. These developments have spurred increasing interest in extending multimodal frameworks the medical field, where the integration of diverse data modalities demonstrates prominent potential in diagnostic tasks. Current research focus lies in multimodal pretraining methods, which focus on cross-modal alignment between imaging and textual data to improve the representation transferabil- ity [7]. ConVIRY [72] and GLoRIA (26) pioneered the application of CLIP-style architectures in the medical domain by constructing image-text pairs from radiology datasets. MedCLIP [56] and BiomedCLIP [70] addressed the scarcity of paired medical image-text data by leveraging multi- source datasets, achieving state-of- the-art performance. Beyond pretraining methods, multimodal fusion approaches aim at integrating information from different modalities for diagnostic applica- tions (20 57 66). MedFuse [21] introduced an LSTMM-based temporal fusion method of series data and X-ray images. HEALNet [23] proposed hybrid early-fusion method to learn from data sources with different structures. While these works have made significant strides, they often operate without explicit guidance from medical knowledge when addressing specific diagnosis tasks. In contrast, considering the evidence- task-specific knowledge to guide both representation extraction and multimodal fusion processes.
nature medicine [44, [1], RAD explicitly incorporates
2.2 Medical Knowledge Injection
Injecting professional knowledge into Al models is a prevalent strategy to improve their domain- specific capabilities 40 65
Various techniques have been investigated incorporate medical knowledge into the models. Pretraining-base approaches train the text encoder on extensive
medical domain corpora, such as PubMedBERT [18] and HUATUO-GPT [69]. Other methods like KAD [7I] and DRAGON [67] leverage structured knowledge graphs of medical entities for pre-training to enhance the text encoder's 's comprehension of medical terminology. While showing empirical effectiveness, these knowledge integration methods remain primarily confined to the pre- training phase, providing only implicit guidance during the subsequent diagnostic stage. With the rapid development of large language models (LLMs) [1 35 54 various Retrieval-Augmented Generation (RAG) methods have been proposed to enhance the generation process medical These methods dynamically retrieve external medical knowledge to improve the performance of LLMS on question-ansswrrin (QA) tasks (34
LLMs 1 61
60). Building upon this foundation, multimodal RAG approaches further retrieve similar data samples (e.g.. image-report pairs) for visual answering (VQA) (59 (74. In contrast to RAG methods that online retrieve knowledge to augment input for generative QA/VQA tasks, our framework adopts structured approach for discrim- inative tasks by performing offline retrieval of discase-specific knowledge, which is systematically incorporated to guide model training.
3 Method
In this section, we first present the problem formulation, followed by the detailed introduction of our proposed method, Retrieval-Augmentt Diagnosis (RAD), which consists of guideline retrieval and refinement, guideline-enhanced feature constraint, and dual diagnostic network. Finally, we introduce our interpretability evaluation system. The overall framework of RAD is illustrated in Figure
3

---

Multi-Source Knowledge
Image
Comesponding
Label
Imagee
iisual Fambeddding
Corpus
Queey
Koy de Vilue
Mulla AuatAitttiin
Mal:-1 Malt- Hnd Melt Atwettn
Adiit Noem
Adl
9
Wiki
Pundas
Biski MRRI
Dersatoogyy
Chess X Ray
Guideline Retrieval Refinement
[
Decoder
Guideline
Research
Query
BCB
Summartze
Text Enooder
LECC
GUIDE
Quey
Report Normalized EHR
Mat Saee Altrtton
Guidelines
compared previous radiograph, the known left lower lobe, paeumonia has unntmmally moreased. -.... The patient is 15-year-old old male. His lab
Key
Embodding
glidte
results Lymphocytos 5. blood sugar with value of 9.....
Books
Guideline Deroder
Figure 2: The overview of our Eetrieval-Augmenne Diagnosis framework, including multi- medical knowledge retrieval and refinement, multimodal representation learning under the guideline constraint, and the dual diagnosis network. @ represents the concatenation operation.
3.1 Problem Formulation
Given a training set of samples, D D= ((2,, (4 y:) | where æ; represents a radiology image, 4 is the report and electronic health records, and Vi (0, 1 is the corresponding multi-label vector indicating the presence of m diseases. The multi-source medical knowledge corpus is denoted as P {pr i = 1, 2, .......... s), where p, denotes the i-th source and denotes the number of sources. The guideline corresponding to the disease label is derived from multi-source retrieval and refinement. We denote this guideline g {9: = 2.... m). The objective is to develop multimodal model trained on D, capable of accurately predicting the disease for any given multimodal sample.
3.2 Retrieval-Augmented Diagnosis
3.2.1 Guideline Retrieval and Refinement
Knowledge-corpus. To retrieve disease-related diagnostic knowledge, collect medical knowl- edge from four distinct sources: "Wiki", "Research", "Guideline", and "Book". Wiki provides comprehensive descriptions of target discases, such formal medical definitions, and clinically relevant subcategories. Research incorporates the latest research articles from PubMed (a premier database of biomedical literature). These articles provide cutting-edge findings in disease mecha- nisms, diagnostic criteria, and therapeutic interventions. Guideline includes 45K clinical practice guidelines from 13 sources, providing rigorously vetted diagnostic criteria and treatment protocols for medical practitioners. Book consists of diverse medical textbooks, covering basic medical knowledge in surgery, medical imaging, and drugs, etc. More details of the corpus be found in Appendix B.
Disease Knowledge Retrieval. For given dataset with m diseases, our objective is to retrieve the most relevant knowledge from the knowledge corpus P. including but not limited associated symptoms, imaging characteristics, and critical examination//lboratooy indicators. We adopt Med- CPT (2 a dual-encoder model optimized for medical scenarios, as the retriever. Specifically, the article encoder R-(-)) (-) is utilized to convert the corpus P into dense vectors for retrieval. The disease names E are used the input query of the query encoder R-)-). The obtained embeddings are then used caleulate the similarity Sim( E, P) RQ (E) (PP)). For each discase with name E E, we preserve the top-k retrieved documents as:
C, = Top- & Sim(ei.ps).
pyEP
LLM Refinement. Given that retrieved documents C, may contain content irrelevant to the diag- nosis the current disease and exhibit cross-source redundancy, directly combining the retrieved documents the final guideline is suboptimal. In addition, the total document length often exceeds

---

the context window of the diagnosis model. To address these challenges, we employ large language models (LLMs) to perform automated summarization and refinement of C, The final refined guideline De of disease can be obtained by:
9: LLM( C1,1, .,,E)).
(2)
where Cr C2 is the j-th document, This process yields standardized, well-structured diagnostic guidelines that preserve critical clinical information while eliminating noise and redundancy. In practice, we choose Qwen2.5-72B (64] as the LLM. Examples of the guideline and prompt templates
are provided in Appendix B.2
3.2.2 Guideline-enhanced Feature Constraint
For multimodal downstream tasks, our framework utilizes two modality-speciffe encoders to sepa- rately learn visual and textual representations. The refined guideline g obtained in Section 3.2. is employed here as the feature constraint of both textual and visual representation.
Given sample (z,, f4 yt), embeddings V; from Zę- The text encoder 4 (-xx(-) is employed to obtain the textual embeddings T,, Meanwhile, the refined guideline g is also encoded by the text encoder for subsequent feature alignment. The encoding process is summarized as follows:
the vision encoder denoted as Ping() (-) to extract the visual
V, imgg (I:) E Rexxxxd
T, (4)) E Rixdd
G == E R.xxiid
where h, w are the height, width of the image, / is the max token length of the text encoder, is the number of disease types, and d is the embedding dimension. These embeddings with spatial information are then used as the input of the dual decoder in Section[322.3] for multimodal fusion and final diagnosis. Here, we perform pooling on the extracted embeddings and use the pooled features for subsequent feature alignment. Specifically, we apply adaptive pooling operations get the visual feature V. E Rd, and directly the embedding of the [CLS] token the textual feature T Rd. The corresponding pooled disease-guideling prototypes are G' {G; E R2 1,2,..... .., m...
To align the extracted features with diagnostic criteria, we propose a guideline-enhancee enhanced feature constraint strategy. Specifically, disease-guudelle prototypes utilized as anchor to pull both image and text features closer to them. To achieve this, we introduce a Guideline -naiinee-- Contrastive Loss (GECL) for feature extraction under the guideline constraint. For sample i with the disease label yi. the guideline features G' are split into P; and N,, where P {G, E G' [gt- = 1} is the set of guideline features corresponding to positive disease labels, N is the set of guideline features with negative disease labels. To avoid using excessive negative samples, we sample a subset Q from N; that satisfies Q mén(r) [N,), where is the negative sampling ratio. The final guideline feature set S, P, Qi Then, we can formalize GECL as pross-entropy-biie based supervised contrastive learning objective:
(
(I) = [SI
Vij
o ((,) - log(1 + (
(4)
P
SS
LGECL CSupCon
CaaiooonTTTT (T, aLSupCon ( (V,, V.,s.) I[[P - > 0].
(5)
where I[:] is the indicator function. ST), I Sty/T is the similarity score between the modality-specifi feature and the guideline feature, t is the temperature hyperparameter, and a is the or hyperparameter. Note that the similarity score can be converted into probability via the Sigmoid function. As shown in Eq. (5). LaECI. aligns image features V and text features T¹ with disease guideline prototypes, which are the diagnostic criteria of each disease defined by the embedding of its guideline. Dynamically aligning sample features with their positive prototypes prevents representation collapse while enhancing model robustness in multi-label scenarios. Further- more, this approach induces the model to selectively focus on clinically relevant features that match the guidelines, improving the model performance and interpretability simultaneously. For detailed derivation from the standard cross cross-entropy form to Eq. please refer to Appendix B.3
5

---

3.2.3 Dual Diagnostic Network
Under the guideline constraint, we obtained enhanced visual and textual features. To achieve the final disease diagnosis, we develop a transformer-based cross-modal information fusion module, which has dual decoder architecture. In the first decoder, the guideline g employed the query, and the concatenated modality embeddings V, + T, are used as the key and value. After forward through the fusion structure $ we obtain the logits corresponding each disease:
of quuide
$
@ text (9), V + Tú V + T.).
(6)
To further enhance the performance, we symmetrically utilize the second similar structure D where the query is replaced with the disease names, while keeping the key and value unchanged. This sym- metric operation gets e! = & (w (E), V, @ T,, V, + T,) E R"", ensuring comprehensive feature integration. Finally, compute the truth. Thus, the total training loss of RAD is:
loss both logits with the ground
N Σ N
Liota
yo)
CECE (ahel y.) LGBCL
(7)
N
guideline
label branch
where 3 represents the trade-off hyperparameter between LaCR and LGECI
3.3
Interpretability Evaluation System
To validate the wridence-based diagnosis of RAD, we introduce a dual-axis interpretability evaluation system that quantitatively measures the model's s adherence to injected guidelines through both textual and visual metrics. Formal definitions of the metrics for each input modality are presented below.
3.3.1 Textual Recall of Indicators
The Guideline Recall designed to quantify the model's explicit compliance with disease- specific diagnostic standards. The refined guide- line of each disease contains set of key lab- oratory indicators that are considered valuable for diagnosing this disease. The extent which a model attends to these indicators can reflect its adherence to the guideline. Formally, when the input text contains indicators mentioned in assess the model's 's attention to these indicators by aggregating the attention weights of the corresponding tokens (derived from the antss-attnntion maps in the transformer
Algorithm 1 Guideline Recall
1: Input: Guideline G, text token sequence T, at- tention weights A, threshold 0
2: 2 Extract indicators from G
attended 0, total
4: for each u E 24 do
5: Matched + Tokens in T matching u if Matched * 0 then
6:
total -tttall
the guideline,
if mean(AA > then aslended = attended + 11 return allended/ if total 0 else
9:
10:
decoders). When the aggregated attention weights exceed predefined threshold 6, this provides quantitative evidence that the model exhibits statistically significant attention to the corresponding indicator. The detailed computation process is outlined in Algorithm
3.3.2 Visual Attention Grounding Ability
For visual explainability, an attention-derivel localization metric is employed to measure the align- ment between model-attended regions and pathological abnormalities. Given expert-annotated bounding boxes for lesions, we compute the overlap between top-activated regions in the attention map and these ground truths. Specifically, we use the Intersection over Union IoU |NB as the
metric, where A is the model localization derived from the attention map and B is the ground truth.
These two metrics formally establish dual-modality interpretability evaluation system. Through the systematic analysis of how the injected knowledge explicitly intervenes in the model's decision- making, this system provides quantitative evaluation for explainable multimodal medical AL.

---

Table 1: Performance across four datasets of different anatomies. The values of "Ace" and "Ace-S" on Maid are the same since the dataset has only one disease. Subscript with arrows represents the absolute difference between RAD and the recond-best method. A is the variance of RAD.
Datasct
Method
F1
Precision
Recall
AUC
mAP
Ace
Acc-S
Avg
MedFuse
34,46
31.36
45.04
90.85
31.77
95.34
41.44
52.89
BiomedCLIP
32.99
29,56
45.04
88.71
29.91
94.72
39,83
51.54
KAD
36.32
33.80
48.33
91.95
33.54
95.12
40.27
54.19
MIMIC-ICDS3 (Chest)
DrFuse
34.10
33.70
45.34
89.50
31.19
94.68
38.25
52.39
HEALNet
35.42
32.76
47.95
88.80
31.97
94.90
40.10
53.13
RAD
39.71
39.07
54.74 411
93.00, n51
36.74, 201
95.40%.00
42.33 son
57.28, 091
A
L.00101
0.0099
0.0016
0.0103
0.0116
0.0050
1. 0,0228
0.0089
MedFuse
81.33
76.13
87.29
87.99
88.
79.50
79.50
83.50
BiomedCLIP
81.27
72.87
91.88
87.69
87.62
78.35
78.35
83.
KAD
81.18
73.92
90.03
88.62
88.88
78.65
78.65
83.55
LMid (Eye)
DrFuse
81.69
73.72
91.59
89.33
90.38
79.00
79.00
84.29
HEALNet
81.80
75.22
89.64
89.60
90.45
79.60
79.60
84.39
RAD
84.30 501
77.52,
92.38, 5ot
91.32 721
91.88; 431
82.40, sOT
82,40 sot
86.63
A
0.0028
J. 0,0077
J. 0,0005
1. 0,0126
1. 0.0014
1. 0,0080
J. 0,0080
1. 0.0060
MedFuss
79.25
85.96
77.99
96.50
73.61
99.34
74.36
83.86
BiomedCLIP
81.49
87.13
81.41
97.22
79.22
99.11
74.36
85.71
KAD
82.06
86.79
81,27
80.40
99.25
75.46
86.
SkinCAP (Skin)
DrFuse
81.18
85.70
79.64
94.92
76.42
99.29
77.66
84.97
HEALNet
82,20
88.69
81,18
92.68
77.97
99.37
78.39
85.79
RAD
85.48,
89.48,
83.23
97.97
83,55 15t
99.48, 141
81.32;
88.64
A
0.0678
0,0750
0.0136
0.0356
0.0639
0,0159
0.0474
0.0407
MedFuse
31.53
25.59
68.36
85.50
24.49
87,
58.45
$4.48
BiomedCLIP
34.36
29.02
66.95
84.00
26.03
88.80
$8.21
55.34
KAD
35,09
29.68
64.49
85.88
27.73
89.69
57.86
55.77
NACC (Brain)
DrFuse
34.11
27.86
68.96
82.88
27.88
87.99
51.31
54.43
HEALNet
35,91
28.92
67.33
85.04
26.13
89.55
56.79
55.67
RAD
37.65 741
36.24,
321
05, 05.78
87.11 p.07t
30.03 901
90.36, s11
59.64,
58.12 451
0.0015
0,0003
0.0019
0.00023
0.0078
0.00220
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate RAD on four multimodal medical datasets with different anatomies, including MIMIC-ICDS53 SrrvaardFFaiVLLMe
Table 2: Detailed information of the datasets.
Dataset
AAatomyy
Modality
|Label] |Sample
MIMIC-ICDS3
Chest
X-say Image & Report & EHR (Lab Results)
53
51830
Harvard-Fair
Eye
Fundus Image & Report Demographics
10000
41I.
SkinCAP
Skin
Dermatology Image & Report
50
2526
NACC
Brain
MRI Image EHR (Lab Results)
4199
SkinCAP (76), and NACC [5]
MIMIC-CCD33 is constructed through the alignment and integration of MIMIC-XX [28] and
MIMIC-IV (29). comprising chest X-ray images, corresponding reports, and EHRs, annotated with 53 diseases under the ICD [43) standard. For laboratory indicators in the EHR, quantified the numerical results on a scale of to 10 based on the upper and lower limits of their normal range. We will release the dataset PhysioNet [42). Details of dataset construction are provided in Appendix C.I.I Harvard-Fair Vard SkinCAP, and NACC are multimodal datasets focusing on eyes, skin, and brain, respectively. All patient data has been de-identified. More detailed statistics of datasets are presented Table 2]
Baselines. We select representative baseline methods in the medical field, including large-scale pre-training model BiomedCLIP 770). knowledge-enhanced pre-training method KAD [71], and state-of-the-art multimodal fusion methods MedFuse (21] DrFuse [66). and HEALNet 23
Evaluation Metrics. For the evaluation of model performance, we adopt widely used multi-label classification metrics including F1, Precision, Recall, AUC, mAP, and ACC. All metrics are the average of multiple labels. Since standard accuracy (ACC) aggregates predictions across all labels and thus may not adequately reflect the correctness for individual patients, we include an additional metric named sample-wise ACC (ACC-S). This metric considers prediction correct only all labels of patient are accurately classified, making more aligned with clinical scenarios.
7

---

Table Quantitative evaluation of Visual Explainability. We calculate the metries for each disease category and report both disease-averager (Avg-D) and patient-averager (Avg-P) values.
Visual Grounding (mloU)
Method
Consolidation Atelectasis Effusion Emphysema Fibrosis Fracture Mass Avg-D
Avg-P
w/o RAD
17.68 24.30
19.23
18.89
14.95
17.22
13.13
10.81
15.98
17.78 22.04
RAD
20.74
20.13
21.15
19.42
17.14
15.15
19.72
Table 4:
Quantitative evaluation of Textual Explainability. We present the guideline recall on representative laboratory indicators and the total average recall. The indicator names are abbreviated.
Guideline Recall
Method
PC
Bilirubin
ALT
IBC
WBC
AST
Total
w/o RAD
23.82
31.34
6.81
37.38
11.96
4,41
24.76
RAD
64.55
51.71
57.96
71.82
29.09
40.65
65.62
Implementation Details. In practice. Top-k in (q. is set to 10. All guidelines obtained by Eq. and the indicators used in Algorithm aar manually verified to avoid potential factual errors. The default backbone of the text image encoder is ClinicalBERT [55] and ResNet-50 [22]. respectively. The hyperparameters and B, which serve the balancing ratio between different losses, are set to be le 2 and le- ---1 1, respectively. All experiments are conducted on a single NVIDIA A100 GPU.
4.2 Diagnosis Performance
As demonstrated in Table our method consistently achieves superior performance across four benchmarks of diverse anatomies. Specifically, RAD outperforms the second- method with average improvements of 3.09%, 2.24%, 2.49%, and 2.45% MIMIC-ICDS3, FaidV Skin- CAP, and NACC datasets, respectively. The most substantial gains occur in MIMIC-ICD53, where RAD improves both precision and recall over 5%, suggesting strong robustness in handling com- plex, real-world clinical label distributions. This improvement is particularly noteworthy given the dataset's challenging nature, containing both fine-grained ICD labels and noisy clinical documenta- tion. Notably, the sample-wise accuracy (ACC-S) of all methods exhibits a significant degradation compared to macro-average accuracy (ACC), especially in datasets with extensive label spaces. This diserepancy highlights fundamental limitations in current models' capacity to handle multi-label problems, exposing challenges for real-world clinical deployment. Intriguingly, KAD, which injects medical knowledge during the pretraining phase, achieves strong performance on MIMIC-ICD53 but falls short on others. This is likely because its pretraining data concentrated on chest X-rays, limiting its ability to generalize to other anatomical regions. In contrast, our approach directly injects knowledge on downstream tasks, offering greater adaptability across distinct regions and modalities. These consistent improvements across diverse anatomies, data scales, and label complexities validate the versatility and scalability of RAD. Full baseline results with variance are in Appendix
4.3 Interpretability Evaluation
4.3.1 Interpretability from Textual Perspective
To quantitatively assess the impact of knowledge injection from the textual perspective, we calculate the guideline recall defined in Section 3 As presented in Table 4 incorporating knowledge via RAD prominently increases the recall value from 24.76% to 65.62%. This indicates that RAD indeed injects guideline-derived knowledge into the model, thereby enhancing its focus on key information mentioned in the guideline. Notably, the conventional model exhibits extremely low recall (<10%) on Alanine Aminotransferase (ALT) and Aspartate Aminotransferase (AST). This may stem from their inability to understand highly specialized, rare medical terms. In contrast, RAD explicitly highlights the importance of these indicators in the guideline, leading to significant recall improvement. This finding underscores the necessity of flexible knowledge adaptation for downstream tasks rather than statie pretraining paradigms. Overall, the enhanced guideline recall demonstrates that RAD enables the model to make reliable evidence-basee diagnoses according to guidelines. This improvement also aligns with the qualitative attention patterns observed in Figure [ To further substantiate the interpretability of RAD, we provide more and clearer visualization cases in Appendix C.5
8

---

94
58
42.5
38-
W/l RAB
Wib RAD
93
RAD
56
v/ RAD
40.0
mAP
35.0
V R
C
RFC R
RFC
Figure 4: Performance under different combinations modality encoder backbones on the MIMIC- ICD53 dataset. (R = ResNet, V VIT, C= ClinicalBERT, B BioClinicalBERT)
Table Ablation on each component of our method. "x" in the "Decoder" column means replacing our dual diagnostic decoder with a conventional MLP. The best results are in boldface.
rvision GECL
GECL
Decoder
F1
Precision
Recall
AUC
mAP
Acc
Acc-$
Avg
x
X
x
34.91
31.01
50.91
91.27
32.24
94.50
38.63
53.35
37.43
33.98
51.44
92.53
34.80
95.26
38.10
54.79
X
V
X
37.75
36.32
51.52
92.91
35.03
95.43
39.65
55.52
V
V
39.34
37.74
51.87
92.94
36.36
95.59
39.95
56.26
X
39.22
36.88
51.41
92.25
36.44
95.39
39.80
55.91
V
V
V
39.71
39.07
54.74
93.00
36.74
95.40
42.33
57.28
4.3.2
Interpretability from Visual Perspective
Symmetrically, we conduct zero-shot grounding experiments on the ChestX- Det dataset [38]. The results shown in Table 3 demonstrate a significant improvement in mloU scores for le- sion detection after the injection of re- fined guidelines. Besides, Figure 3 illustrates multiple cases of lesion grounding. For clearer visualization, overlay spectrum heatmaps the original CXR images, together with the ground truth bounding box high- lighted in red. A comparison between the lesions identified by the model and the bounding boxes marked by clinical experts reveals notable improvement in alignment when guidelines applied. This indicates that the model's focus more accurately directed toward clinically significant lesions, emphasizing RAD's enhanced diagnosis capabilities and interpretability under the guidance of external knowledge.
W/O RAD
RAD
Disease Atelectasis
Fibrosis Corsolication
Mass
Figure Visualization grounding results on four diseases.
4.4 Ablation Study
this subsection, we conduct ablation studies on each component of RAD and validate its generaliz- ability across different model architectures. All experiments conducted on MIMIC-ICD53.
Ablation on Each Component. As shown in Table 5 we evaluate the efficacy of each newly pro- posed component RAD. It evident that removing either the LGECI or the dual decoder negatively impacts model performance, highlighting the importance of the guideline in both representation learning and multimodal fusion. Notably, the removal of the Dual Decoder results the most substantial performance degradation, underscoring the necessity of leveraging guidelines to intervene in the final decision decision-mmaing process. We further compared the performance of the textual and visual branches of LGECL when used individually. The results show that the textual branch yiclds more significant improvements. This can be attributed to the fact that both the input text and the guideline belong to the same modality, allowing for more effective alignment.
Ablation on Different Backbones. To demonstrate the robustness and flexibility of our method, we verify RAD on different encoder backbone combinations. Specifically, we iteratively replaced the default visual encoder and text encoder with two other popular architectures, ViT [15) and

---

Avg
F1
Acc
F1
Aoc
Precision
Recall
Precision
Recall
Few-shot
SFT
RAD
Lingshu-7B
HuatuoGPT-Vision-77
Qwen2.5-VL-7B
Figure 5: Performance comparison with MILMs. We convert the single-label dataset FairVLMedd into the VQA format and evaluate MLLMs under both few-shot and supervised fine-tuning settings.
BioClinicalBERT 2] As illustrated in Figure RAD consistently offers substantial performance gain across all four combinations of backbones. This not only highlights the insensitivity of our approach to different backbone architectures but also underscores its robustness and generalizability. Specifically, ResNet and ViT exhibit comparable performance gains, while ClinicalBERT shows more pronounced improvement than BioClinicalBERT. Overall, RAD exhibits the robust ability to generalize to diverse data and model structures, ensuring reliable performance in various scenarios. Detailed results, more ablation studies, and hyperparameter analysis are presented in Appendix C.6
4.5 Discussion
Comparison with Multimodal Large Language Models Multimodal large language models exhibit remarkable capabilities in visual content understanding and generalization. To further validate the effectiveness of RAD, compare with satte-of-the-az MLLMs, including Owen2.5-VL-7B (3). HuatuoGPT- PiaiinGGG---iiiii----- [10], and Lingshu-7B [62]. As presented in Figure 5 our discriminative framework achieves superior performance with significantly lower computational cost. These results demonstrate that complex diagnostic tasks are better suited for specialized discriminative models than generative MI.LMs. The significant performance gap, observed the simplest single-label dataset, underscores the practical advantages of our approach in clinical applications where both accuracy and efficiency are critical.
5
Conclusion
This paper proposes RAD, which enhances the capabilities of multimodal diagnosis models by leveraging external medical knowledge. RAD operates via tri-fold methodology. consisting of offline retrieval and refinement of disease-centered external guidelines, multimodal feature alignment under the guideline constraint, and the dual diagnostic network. Extensive experiments on four datasets of different anatomies demonstrate the effectiveness of RAD. Furthermore, RAD exhibits dual-axis interpretability by simultaneously achieving precise lesion localization imaging data and prioritizing quideline-concordam concordant indicators in textual analysis. This evidence- podencee-ased explainability enhances clinical trustworthiness, offering the potential to inspire future research in medical AI.
Acknowledgement
This work is supported by the National Key R&D Program of China (No. 20222D0160703). National Natural Science Foundation of China (No. 62306178) and STCSM (No. 22D7?2220055), 111 plan (No. BP0719010) and Shanghai Artificial Intelligence Laboratory.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:/904.00 93.99004033,,, 2019.
2019.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang. Peng Wang. Shijie Wang. Jun Tang. et al. Qwen2. 5-v1 technical report. arXiv preprint (7Xiv:2502.13922 2025.
[4] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Phili-P Moreney. Multimodal machine lcarning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423 443, 2018.
[5] Duane L Beekly, Erin Ramos, William Lee, Woodrow D Deitrich, Mary E Jacka, Joylee Wu. Janene Hubbard, Thomas D Koepsell, John C Morris, Walter A Kukull, et al. The national alzheimer's is coordinating center (nacc) database: the uniform data set. Altheimer Disease & Associated Disorders, 21(3):249-25 258, 2007.
[6] Edgar A Bernal, Xitong Yang. Qun Li, Jayant Kumar, Sriganesh Madhvanath, Palghat Ramesh, and Raja Bala. Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors. IEEE Transactions on Multimedia, 20(1): 1077:117-118, 2017.
[7] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer. Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- et al. Making the most of text semantics improve biomedical vision-language processing. In European conference computer vision, pages 1-21. Springer, 2022.
[8] Cheng Chen, Qi Dou. Yueming Jin. Quande Liu, and Pheng Ann Heng. Learning with privileged multimodal knowledge for unimodal segmentation. IEEE transactions medical imaging, 41(3))621-633 2021.
[9] Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. Cod, towards an interpretable medical agent using chain of diagnosis. arXiv preprint arXiv:2407. 13301, 2024.
[10] Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang. Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal Ilms scale. arXiv preprint arXiv:2406. 14080.900000 2024.
[11] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A iision-audo-suubiile-eee omdilmodality foundation model and dataset. Advances in Neural Information Processing Systems, 36:72842-72866. 2023.
[12] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matto Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Mcditron-706 Scaling medical pretraining for large language models. arXiv preprint arXiv:231111607. 2023.
[13] Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, Yiqiu Guo, Yanfeng Wang, and Yu Wang. Towards omni-rag: Comprehensive relrenea generation for large language models in medical applications. arXiv preprint arXiv:2501.00 02460, 2025.
[14] Tianjic Dai, Ruipeng Zhang. Feng Hong, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classification. IEEE Transactions on Medical Imaging, 2024.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. / 19929, 2020.
[16] Qi Dou, Quande Liu, Pheng Ann Heng, and Ben Glocker. Unpaired multi-modal segmentation via knowledge distillation. IEEE transactions medical imaging, (9(7))2445-2422 2020.
[17] Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedelip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: FACL. 2023, pages 1181-1193 2023.
[18] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain -seaiicc language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1) 23, 2021.
11

---

[19] Bo Han, Jiangehao Yao, Tongliang Liu, Bo Li. Sanmi Koyejo, Feng Liu, et al. Trustworthy machine learning: From data models. Foundations and Trends@ in Privacy and Security, 7(2-3):74-246, 2025.
[20] Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):9,, 2019.
[21] Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. Medfuse: Multi modal fusion with clinical time- series data and chest x-ray images. In Machine Learning for Healthcare Conference, pages 479-503, PMLRR 2022.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[23] Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Healnet: Multimodal fusion for heteroge- neous biomedical data, Advances Neural Information Processing Systems, 37:64479-6444 64499, 2024.
[24] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumarr, Max Körfer, Shi Bin Hoo, Robin Tibor Schirmeister,, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319 026.9332622222 2025.
[25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
[26] Shih-Cheng Huang, Liyue Shen, Matthew P PI Lungren, and Serena Yeung. Gloria: A multimodal global- local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3942-3951, 2021.
[27] Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. Medept: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 99(11):btad651 2023.
[28] Alistair EW Johnson, Tom J Pollard, Seth Berkowitz, Nathaniel R Greenbaum, Matthew Lungren, Chih-ying Deng, Roger G Mark, and Steven Horn. Mimic-cxi, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
[29] Alistair FW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
2023.
[30] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234- 12404, 2020.
[31] Karim Lekadir, Richard Osuala, Catherine Gallin, Noussair Lazrak, Kaisar Kushibar, Gianna Tsakou, Susanna Aussó, Lconor Cerdá Alberich, Kostas Marias, Manolis Tsiknakis, et al. Future-ai: guiding principles and consensus recommendations for trustworthy artificial intelligence medical imaging. arXiv preprint arXiv:2109.06655 2021.
[32] Haolin Li, Yuhang Zhou, Ziheng Zhao, Siyuan Du, Jiangchao Yao, Weidi Xie. Ya Zhang, and Yanfeng Wang. Lorkd: Low-rank knowledge decomposition for medical foundation models. arXiv preprint arXiv:2409. 19540, 2024.
[33] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language -magge pre- pre-training with frozen image encoders and large language models. In International conference on machine learning. pages 19730-19742. PMLR, 2023.
[34] Mingchen Li, Halil Kilicoglu, Hua Xu, and Rui Zhang. Biomedrag: A retrieval augmented large language model for biomedicine. Journal of Biomedical Informatics, 162: 104769, 2025.
[35] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprint 24Xiv:2412.19433 2024.
[36] Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Xu Sun, el al. Auto-encoding knowledge graph for unsupervised medical report generation. Advances in Neural Information Processing Systems, 34:16266. 16279, 2021.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
12

---

[38] Jingyu Liu, Jie Lian, and Yizhou Yu. Chestx-det10: chest x-ray dataset on detection of thoracic abnormali- ties. arXiv preprint arXiv:2006.10550 2020.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
international conference computer vision, pages 10012-10022, 2021.
[40] Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. Finhert: A pre-trained financial language representation model for financial text mining. In Proceedings of the twenty-ninth international conference international joint conferences artificial intelligence, pages 4513 4519, 2021.
[41] Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. Fairclip: Harnessing fairesss in dision-languag learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 12301, 2024.
[42] George B Moody. Physionet. In Encyclopedia of Computational Neuroscience, pages 2022.
Springer,
[43] World Health Organization et al. International classification of diseases-icd. 2009.
[44] Yifan Peng, Justin F Rousseau, Edward H Shortliffe, and Chunhua Weng. Ai-generated text may have role in evidence-based medicine. Nature medicine. 29(7):1593-1594 2023.
[45] Fernando Pérez-Gareía, Harshita Sharma, Sam Bond-Taylor Kenza Bouzid, Valentina Salvatelli, Maxim- ilian Ilse, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Matthew P Lungren, et al. Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence, pages 1-12, 2025.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[47] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine, 4(1):86, 2021.
[48] Maya Rotmensch, Yoni Halpern, Abdulhakim Tlimat, Steven Horng, and David Sontag. Learning& a health knowledge graph from electronic medical records. Scientific reports, 7(1):5994, 2017.
[49] Zohaib Salahuddin, Henry C Woodruff, Avishek Chatterjee, and Philippe Lambin. Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Computers in biology and medicine, 140:105111 I., 2022.
[50] Gregor Stiglic, Primoz Koebek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Inter- pretability of machine learning- based prediction models in healtheare. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(5) 2020.
[51] Vivek Subbiah. The next generation of evidence-basee medicine. Nature medicine, 29(1):49-555 188999558558888 2023.
[52] Heung-Il Suk, Seong- Whan Lee, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, Hier- archical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. Neurolnage, 101:569-582, 2014.
[53] Jesse Sun, Fatemeh Darbehani, Mark Zaidi, and Bo Wang. Saunet: Shape attentive u-net for interpretable medical image segmentation. In International conference on medical image computing and computer- assisted intervention, pages 797-806. Springer, 2020.
[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini ar02v: 130..1911..1 2023.
[55] Guangyu Wang, Xiaohong Liu, Zhen Ying. Guoxing Yang, Zhiwei Chen, Zhiwen Liu. Min Zhang, Hongmei Yan. Yuxing Lu, Yuanxu Gao, et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: proof-of-concept trial. Nature Medicine, 29(10):23332264 2022. 2023.
[56] Zifeng Wang. Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2022, page 3876, 2022.
13

---

[57] Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer's Disease Neuroimaging Initiative. et al. Daft: A universal module to interweave tabular data and 3d images cnns. Neurolmage, 260: 19505, 2022.
[58] Chaoyi Wu, Pengcheng Qiu, Jinxin I Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards evaluating and building versatile large language models for medicine. npj Digital Medicine, 8(1): 58, 2025.
[59] Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang. Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag Versatile multimodal rag system for medical vision language models. arXiv preprint Xiv:2241 3085, 2024.
[60] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024, pages 6233-6255 2024.
[61] Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Bowen Jin, May Wang. Joyce C Ho, and Carl Yang. Ram-ehr: Retrieval augmentation meets clinical predictions on electronic health records. arXiv preprint Stti:2400300081333 2024.
[62] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun I Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint ar 2r0ivv2506.000 07004, 2025.
[63] Chonghua Xue, Sahana S Kowshik, Diala Lteif, Shreyas Puducheri, Varuna H Jasodanand, Olivia T Zhou, Anika S Walia. Osman B Guney, J Diana Zhang, Serena T Pham, et al. Ai- based differential diagnosis of dementia etiologies multimodal data, Nature Medicine, 30(10):2977 2989, 2024.
[64] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li. Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511:2412.. 2024.
[65] Yue Yang, Mona Gandhi, Yufci Wang, Yifan Wu, Michael Yao, Chris Callison-Buree James Gec, and Mark Yatskar. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. Advances in Neural Information Processing Systems, 37:90683-9077 13, 2024.
[66] Wenfang Yao, Kejing Yin, William K Cheung. Jia Liu, and Jing Qin. Drfuse: Learning disentangled repre- sentation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 16416-16424, 2024.
[67] Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy S Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. Advances in Neural Information Processing Systems, 35:37309 37323, 2022.
[68] Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang. Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of- of-experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[69] Hongho Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangho Wu, Zhiyi Zhang. Qingying Xiao, et al. Huatuogpt. towards taming language model to be doctor. arXiv preprini xrx::22305. 15075, 2023.
[70] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint 2.03:22333.001... 2023.
[71] Xiaoman Zhang, Chaoyi Wa, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual- language pre-training on chest radiology images. Nature Communications, 14(1):4542, 2023.
[72] Yuhao Zhang. Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference, pages 2-25. PMLR, 2022.
[73] Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6428 0228-6336, 2017.
[74] Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding. Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for augmented generation: A survey, arXiv preprint arXiv:2303.1088 2023.
14

---

[75] Hong- Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via sups-ruupeiisionn between images and free-text radiology reports. Nature Machine Intelligence, 4(1):32-40, 2022.
[76] Juexiao Zhou, Liyuan Sun, Yan Xu, Wenbin Liu, Shawn Afvari, Zhongyi Han, Jiaoyan Song. Yongzhi Ji. Xiaonan He, and Xin Gao. Skincap: modail dermatology dataset annotated with rich medical captions. arXiv preprint arXiv:2405. 2024.
[77] Yuhang Zhou, Siyuan Du, Haolin Jiangchao Yao, Zhang, and Yanfeng Wang. Reprogramuing distillation for medical foundation models. In International Conference on Medical Image Computing and Computer-Asisste Intervention, pages 533 543. Springer, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section explains our method. Section presents experimental results to verify RAD on different datasets with diverse anatomies (Section 4.2) and different model structures (Section 4.4)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitation is discussed in Appendix A.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
There are no included theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
We have included a detailed experimental setup in Section 4.1 including dataset and implementation details (configuration like metrics, hyperparameters. etc) The preprocessing of the datasets is provided in Appendix [.I.I] The construction of the retrieval corpus provided in Appendix B.1 And the dataset and code will be publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code is available https://githut com/talhh/RAD With the exception of MIMIC-ICDS3, all datasets are publicly available. We will also make our constructed MIMIC-ICDS3 available PhysioNet upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
hhblltt/guilaes/oddeeuubmmssss hot.eeeouubbmssioooooooooo... for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctipss.cc/pullic/guiii guides/ CodeeSummiee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included a detailed experimental setup in Section 4.1 including dataset and implementation details (configuration like metrics, hyperparameters, etc). And the details can also be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the variance of our method over runs in Table [ The full results with variance of all basclines are presented in Appendix C33
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the experiments are conducted on a single NVIDIA A100 GPU, and we have reported the compute sources in the implementation details of Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httess/sttt////////////p///////t
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts have been discussed in Appendix A.I
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no safeguard risk in this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited the original papers and followed their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
/RAD Due to licens- ing restrictions, our new dataset (MIMIC-ICD53) will be available only on PhysioNet upon publication. The construction details of this dataset have been included in Appendix C.I.I
Justification: The code is available at:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
We utilized LLMs for the guideline refinement, which has been thoroughly detailed in Section 3.2. The function of LLM here is serve powerful long-context text summarization tool.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Appendix
Further Discussion
24
A.1 Broader impact
24
A.2 Limitations
24
Method Details
24
B.1 Knowledge-corpus Construction
24
B.2 Details for LLM Refinement
24
B.3 Derivation the Gaideline-Enhanee Contrastive Loss
25
C Experimental Details
27
C.I Details of Datasets
................................
27
C.2 Benchmarking MIMIC-ICD53
................................
29
C,3 The Variance of Baselines..
................................
30
C.4 Label-wise Analysis of MIMIC-ICD53
................................
31
C55 Interpretability Cases
C.6 Ablation Study
33
C.7 Cost Analysis.
34
C.8 Hyper-parameter analysis
34
23

---

A Further Discussion
A.1 Broader impact
The method proposed in this paper can effectively enhance the diagnostic capability of multimodal medical models. With the integration the guidelines, RAD is optimized through intervention in accordance with the guidelines. This not only improves diagnostic accuracy but also strengthens the model's s interpretability, making its decision -aaiiiin-mmainggg process more transparent and deployable in real- world clinical scenarios. Specifically, the systematic integration of multimodal data (imaging, text, and structured records) enables RAD to capture disease manifestations from multiple perspectives, potentially advancing personalized medicine through comprehensive patient profiling, However, the use of multimodal clinical data, including sensitive patient records and imaging features, necessitates stringent compliance with relevant regulations to prevent misuse or unintended leakage of private health information.
A.2 Limitations
limitation the current implementation is the static retrieval knowledge corpus. While medical guidelines undergo periodic updates (e.g., every 3-5 years) to incorporate new evidence and diseases, RAD relies on fixed knowledge base that may require manual updates to reflect revised diagnostic standards. This temporal mismatch could be addressed by regular updates of guidelines and further fine-tuning of the models, which would enhance long-term clinical relevance without compromising current performance.
B
Method Details
B.1
Knowledge-corpus Construction
For retrieving disease-related diagnostic knowledge, we collect medical knowledge from four distinct sources: "Wiki", "Research", "Guideline", and "Book". Wikipedia provides comprehensive and general descriptions of target diseases, such standard disease nomenclature, formal medical definitions, and clinically relevant subcategories. The processed data are obtained from Huggingface" Research incorporates the latest research articles from PubMed premier database of biomedical literature). These articles provide cutting-edge findings in disease mechanisms, diagnostic criteria, and therapeutic interventions. We utilize the 2024 PubMed baseline") which is complete snapshot of Pub.edd data. We filter the valid data through their paper titles and corresponding abstracts. Guideline includes 45K clinical practice guidelines from 13 sources. The guidelines provide rigorously vetted diagnostic criteria and treatment protocols for medical practitioners, serving as a critical component for reliable decision support. We employ the Clinical Guidelines dataset [12] and use the provided scripts to crawl non-redistributabe portions of the data. Book. consists of diverse medical textbooks. These books cover organized basic medical knowledge in surgery, medical imaging, and drugs, etc. We follow MedOmniKB [13) to collect 18K PDF documents from online medical libraries and academic publishers. Then, deduplicate and filter these books to obtain the final retrieval database.
B.2 Details for LLM Refinement
An example of the final guideline is presented in Figure For more guidelines, please refer to our GitHub repository.
The detailed prompt template for LLM refinement is shown Figure 7. In the prompt, (dis- ease_icd ) is the disease name e,, (topk) is the number of preserved documents, and (re- sreve_passage_sttr str) is the content of the document.
https://ftp.ncbi. nlm. mih wwt.ggggw/ubbebbbballeenee http://pw.........io.oooo//aaaaa
24

---

The guideline of "bronchitis":
Summary of Key Diagnostic Features for Bronchitis
Disease Description: Bronchitis is an inflammation of the bronchi, the air passages in the lungs. It can be classified into two main types: acute and chronic. Acute bronchitis typically a self-limiting condition characterized by a cough that may produce sputum and is often caused by viral infections. Chronic bronchitis, on the other hand, is long-term condition defined by productive cough lasting for at least three months in two consecutive years, often associated with chronic obstructive pulmonary disease (COPD). The primary risk factor for chronic bronchitis is tobacco smoking, with other factors including air pollution and occupational exposures.
Important Lab Tests and Values: Acute Bronchitis: White Blood Cell Count (WBC): Usually normal or slightly elevated. C-reactive Protein (CRP): May be slightly elevated but not typically high. Sputum Culture: Not routinely necessary, but can be useful if bacterial infection is suspected. Chronic Bronchitis: Pulmonary Function Tests (PFTs): Reduced FEV I/FVC ratio, indicating airflow obstruction. Sputum Analysis: Increased mucus production, often with neutrophil infiltration. Blood Gas Analysis:** ** May show hypoxemia and hypercapnia in advanced cases.
Key Radiological or Clinical Findings: Acute Bronchitis: Chest X-ray: Usually normal, but may show hyperinflation peribronchial thickening. Physical Examination: Wheezing, crackles, and rhonchi on auscultation. Chronic Bronchitis: Chest X-ray: May show hyperin- flation, increased bronchovaseular markings, and signs of emphysema. CT Scan: Can reveal bronchial wall thickening and mucus plugging. Physical Examination: and signs of cor pulmonale in advanced cases.
Barrel chest, cyanosis,
Initially dry, then becomes productive with clear yellowish sputum. Fever: Usually mild or absent; high fever suggests pneumonia. Fatigue and Body Aches: Common but generally mild. Wheezing and Shortness of Breath: May be present, especially in patients with underlying asthma. Chronic Bronchitis: Cough: Persistent, productive cough with sputum, often for at least three months in consecutive years. Dyspnea: Shortness of breath, especially on exertion. Wheezing: Common, especially in the morning. Chest Pain: May occur due to prolonged coughing. Fatigue and Malaise:
Diagnostic Symptoms or Relevant Clinical Features: Acute Bronchitis: Cough:
Persistent, often due chronic hypoxemia.
Figure 6: Examples of the guidelines.
B.3 Derivation of the Euideline-Enhaneed Contrastive Loss
Here, we derive the form of the most basic cross-entropy loss to the form in Eq. (4). We demon- strate the equivalence between the sigmoid-based cross-entropy formulation and the logit-style implementation of our supervised contrastive loss.
The equation in cross- cross-entropy form with explicit sigmoid terms is defined as:
LSupCon (I,
ykj P.
log ((is))))) - (1 Vij
log(1- -o(o)))
log(1 (lg(1-o(ol)
(8)
S
P.
jES,
where diÿ is short for S.)) IT Siy/T. the similarity score between the modality-specific feature and the guideline feature. (--) is the Sigmoid function, the logit of the similarity score is ( *., The difference between Eq. (8) and the standard cross-entropy loss is that we use the similarity score logits, and we add the normalization coefficient |.. to balance the gradient contribution of each positive label multi label scenarios. The following is step-by-step derivation. First, substitute the sigmoid function into Eq. (8):
25

---

Prompt for L.IM Refinement:
Your task is to help filter and summarize the relevant information from multiple sets of retrieved external knowledge (from 4 different sources) to support my multi-modal disease classification model in diagnosing the specific disease associated with the provided ICD disease description. For the given disease, identify the critical symptoms, lab indicators, and radiological features that most strongly associated with the disease diagnosis. Discard any information that is unrelated or irrelevant to disease classification. You do not need to focus on treatment options, but instead, concentrate on factors that would help in diagnosing the disease. You will be provided with set of documents (containing topk retrieved documents, each approximately 2000 characters), containing a range of medical information. Your job is to:
Review the retrieved documents and determine which information is directly relevant to diagnosing the disease, based on its ICD code and description. Eliminate any information that unrelated to diagnosis or classification, such as treatment options, management strategies, or irrelevant clinical details. Focus identifying key diagnostic features, including symptoms, laboratory test results, and imaging findings that help confirm the presence the disease. Evaluate the importance of each retrieved documents: Some documents may provide more critical or reliable information than others. Prioritize information that is most relevant and useful for diagnosing the disease, even if it means excluding less relevant details from certain documents. Summarize the most relevant content into a single cohesive summary of approximately 2000 characters (or 500 words). The summary should include the essential diagnostic criteria, including lab values, clinical features, and radiological findings.
The summary should emphasize:
A brief explanation or description of the disease, including any variations or related conditions under the same ICD code. Important lab tests and values (e.g., white blood cell count, C- reactive protein). Key radiological or clinical findings associated with the disease's presence (e.g., lung opacity, pleural effusion). Any diagnostic symptoms or relevant clinical features. Discard information that is not useful for diagnosing the disease.
Please ensure that the summary concise and directly related diagnosis, omitting irrelevant details.
Disease description (ICD name): (disease [od__nmme
Retrieved passages: (retrieve (retrieve_passages passages_str
Figure 7: Prompt for LLM refinement.
SSppCoon
1
Voj log a (dig) (1 -1-11/1 P
Dkj
log(1
(dv)))
[8,
P.
(Substitute sigmoid identities: log ( log(1 + e²),
log( o = -log(1 + e0))
Usj
(Øi - llo((1 + cQu))
V/ij
(- log(1 + +
S
P.
P
Vij
bij
Vig P
log(1 + E
Mii P
log(1 + t
P.
1
Vij tij log(1 + edis)
Yij [P
Vig
}
+1
[
P.
P.
1
Σ
dis -log(1 + t000) Pa
Vki
S
The final equation is:
Csu (I,, S.)
Vij
OLI, Siz) - -lo((1 + ((sl)
(9)
S
P
S., ES,
We use this form directly in the main body Section 3.2.2
26

---

600
Figure 8: Label distribution of MIMIC-ICD53, The X-axis represents the formal disease names under the ICD-10 standard.
C Experimental Details
C.1 Details of Datasets
C.1.1 Construction Process of MIMIC-ICDS3
First, merged and aligned the ED. HOSP, and ICU parts of MIMIC-IV [29). Subsequently, we aligned the processed MIMIC-IV dataset with the MIMIC-CXR-JPP dataset [28]. We utilized patient_ id and study_ id to align the datasets the patient level. Given that temporal patient dynamics were not considered, we selected the most recent radiological examination for each patient, including the associated images and reports. For temporal alignment, extracted EHR data and ICD disease codes from MIMIC-IV within a three-day window following the radiological examination. After excluding instances with missing modalities or labels, we obtained final sample size of 51830. For disease labeling, we standardized the granularity of diagnoses according to the ICD-10 classification. using the format Xab (where X is letter and ab are digits), resulting in over 2000 unique labels. To refine and further clean the dataset, we consulted LLMs and then physicians to identify and select 53 critical disease categories that were related thoracic and cardiovascular conditions or could be identified using laboratory indicators in the EHR. For the numerical indicators in the EHR, we quantized each indicator on a 0-10 integer scale based on corresponding normal range limits. A value of 4-7 is considered normal, 0-3 indicates too low, and 8- 10 signifies too high. The label distribution of MIMIC-ICDS3 is shown in Figure The training set and test set are randomly divided in ratio of 4:1. The final processed dataset, termed MIMIC-ICD53, will be made publicly available on PhysioNet after publication. (The MIMIC dataset requires that all datasets developed based on MIMIC can only be released on PhysioNet.)
C.1.2 Preprocess of NACC
The National Alzheimer's Coordinating Center (NACC) dataset [5] is a large, standardized resource comprising clinical and neuropathological data collected from individuals assessed Alzheimer's Disease Research Centers (ADRCs) across US, which consists various neurodegenerative diseases, like Alzheimer's disease, Parkinson's disease, vascular dementia, and other forms of cognitive impair- ment. We follow [63] to organize the dataset, resulting in 11 labels including: "Normal cognition" (NC), "Mild cognitive impairment" (MCI), "Dementia" (DE), " "Alzheimer's disease" (AD), "Vascular dementia, vascular brain injury and vascular dementia" (VD), "Lewy body dementia, including dementia with Lewy bodies and Parkinson's disease dementia" " (LBD), "Psychiatric conditions including schizophrenia, depression, bipolar disorder, anxicty and posttraumatic stress disorder"
27

---

(PSY), "Frontotemporal lobar degeneration and its variants, including primary progressive aphasia, corticobasal degeneration and progressive supranuclear palsy, and with without amyotrophic lateral sclerosis" (FTD), "Systemic and environmental factors including infectious diseases (HIV included), metabolic, substance abuse alcohol, medications, systemic disease and delirium" (SEF), "Other dementia conditions, including neoplasms, Down syndrome, multiple systems atrophy, Huntington's s disease and seizures" (ODE), and "Moderate/severe traumatic brain injury, repetitive head injury and chronic traumatic encephalopathy" (TBI). The label distribution of NACC is shown in Figure 9, Given that NACC contains over 800 distinct EHR variables, selecting the most relevant features for analysis was critical step in study. To ensure both scientific validity and clinical interpretability. we first utilized LLM for cleaning and double-checked with several physicians.
Finally, we distilled the original set down to a final list of 36 key EHR variables. The selected variables include height, weight, body mass index (BMI), systolic blood pressure, diastolic blood pressure, corti- cal atrophy (Alzheimer's disease marker), small vessel disease (vascular dementia re- lated), left motor cortex vascular lesion, right motor cortex vascular lesion, nor- mal pressure hydrocephalus gait, parkin- sonian signs (tremor/rggidity), bradykine- sia (Parkinsonian symptom), neck rigidity (dystonia), gait disturbance, history of hy- pertension, history of diabetes, history of cardiovascular disease, history of stroke, history of Parkinson's 's disease, sleep apnea, REM sleep behavior disorder (RBD), his- tory of traumatic brain injury (TBI), delusions, hallucinations, depressive symptoms, agitation or aggression, unti-dementia medication (e.g.), Parkinson's 's disease medication (e.g.), anticoagulant use (stroke prevention). antidepressant medication, postural instability (Parkinson's or Lewy body dementia), APOE e4 allele (Alzheimer's disease risk), hypercholesterolemii (vascular risk). amy- otrophic lateral sclerosis (ALS) signs, left visual cortex functional impairment, and right visual cortex functional impairment. The final processed dataset randomly divided in ratio of 4:1 for training and testing.
180
FTD
sBF
1B
Figure Label distribution of NACC.
C.1.3 Details Harvard-FairVI.Mor and SkinCAP
The Harvard-FairVL.Mer dataset 41, sourced from the Department of Ophthalmology at Harvard Medical School, contains 10,000 multimodal samples (7,000 train, 1,000 val, 2,000 test) with paired clinical notes, diagnostic labels, and detailed demographic attributes (race, gender, ethnicity, language). The dataset is publicly available under the CC BY-NC-ND 4.0 license at Githul We directly used the original dataset.
skin disease images with rich natural language descriptions [76]. The dataset combines cases from diverse dermatology image datasets, all annotated by board-certified dermatologists to ensure clinical accuracy. It is publicly available under an open license at HuggingFace To address class imbalance, we removed tail categories with too few positive samples, resulting in a filtered dataset of 2,526 samples with 50 disease labels. The final dataset was partitioned into a 4:1 train-test split.
SkinCAP is multimodal dermatology dataset containing 4,000
28

---

Table 6: Benchmarking performance of each modality on MIMIC-ICD53 dataset.
Modality
Model/Method
F1
Precision
Recall
AUC
mAP
Acc
Acc-S
Avg
R8
14.39
10.49
32.94
71.47
9.66
87.60
21.94
35.50
ResNet50
14.49
10.99
33.09
72.35
9.92
87.68
22.41
35.85
ViT-Base
15.49
11.59
34.90
73.93
10.81
88.00
20.66
36.75
Image
Swin-Base
16.93
13.61
33.97
75.12
12.11
89.03
18.94
37.10
UniChest
19.30
16.68
31.45
78.16
14.18
92.00
21.63
39.06
RadDino
17.26
13.79
29.90
75.34
11.75
90.89
13.87
36.11
MLP
19.14
15.15
38.52
72.57
14.81
89.31
0.00
35.64
TabFPN
9.20
5.85
53.32
52.70
5.36
59.24
0.00
26.52
ClinicalBERT
14.96
9.99
50.41
79.80
9.65
84.16
18.03
38.14
EHR
BioClinicalBERT
11.90
8.57
42.57
72.52
7.21
80.51
18.98
34.61
PubMedBERT 10.18
6.31
78.48
67.16
5.87
53.35
19.88
34.46
LLaMa-3.2-1B
22.00
18.94
35.86
84.47
16.52
91.92
15.85
40.79
LLaMa-3.1-8h
22.53
18.55
38.84
84.98
16.66
92.28
17.65
41.64
MMedS-88 8B
21.88
18.00
34.47
84.72
15.99
92.47
19.98
41.07
ClinicalBERT
24.29
21.06
39.95
83.23
18.45
91.52
22.66
43.02
BioClinicalBERT
29.12
25.13
45.18
84.21
23.03
91.59
17.29
45.08
PubMedBERT
15,46
12.06
36.86
73.80
9.65
87.07
26.59
37.36
Report
L.L.Ma-.22-1B
30.86
28.91
42.67
86.81
25.46
93.83
22.53
47.30
I.LMMa33.1-8B
32.53
31.84
42.03
87.54
27.30
94.14
25.29
48.67
MMedS-8B
32.39
29.52
43.73
86.78
27.08
94.17
22.97
48.09
ClinicaBBRRT
27.13
23.43
46.32
89.22
22.20
92.47
31.32
47.44
BioClinicalBERT
28.18
23.62
46.96
89.28
22.28
92.26
29.26
47.41
Report
PubMedBERT
10.10
6.02
82.24
66.50
5.71
50.46
0.11
31.59
+EHR
LLaMa-3.2-IB
33.68
30.57
46.89
91.90
29.65
94.75
36.50
51.99
LLaMa-3.1-8B
32.84
28.69
47.94
90.91
28.20
94.09
32.35
50.72
MMedS-8B
33.27
31.80
47.58
91.51
28.96
94.44
32.87
51.49
C.2 Benchmarking MIMIC-ICDS3
To further evaluate the quality of our constructed dataset MIMIC-ICD53, we employed various unimodal methods train and test its performance. For the visual modality, selected ResNet-18, ResNet-50 [22], VIT [15), and Swin Transformer [39), along with two SOTA CXR-specific pretrained models, UniChest [14] and RadDino [45) as the baselines. Based on both computational efficiency and data leakage prevention considerations. we ultimately designated ResNet-50 as the standard visual backbone for the main experiments. ViT is also investigated in ablation studies in Section C.6,
For electronic health record (EHR) data, we first leveraged its inherent tabular structure by treating each EHR attribute respective dimensions. We experimented with MLP and a SOTA tabular data process method TabFPN [24), but both exhibited suboptimal performance. Consequently,
input dimension, with corresponding numerical values assigned to their
reformatted the EHR data into natural language text using the following template: "Laboratory values within the 4-7 range indicate normal levels, values 0-3 suggest clinically low levels, and values 8-10 denote elevated levels. The current panel includes [ATTRIBUTE] with the discretized value of [VALJE]...." We then evaluated the reformatted EHR data using classic backbone ClinicalBERT [55), BioCLinicalBERT (2). and PubMedBERT [18]. We also included natural and medical LLMs LLaMa [54] and MMed- S [58]. Specifically, we replace the last layer of LLMs with classification head to adapt to the text classification task. Due to the high consumption of computing resources, we only use LoRA [25] to fine-tune the LLM--baed models. We further conducted experiments using the same text encoders for the report modality alone and reports combined with EHR data. As shown in Table unimodal performance analysis reveals that the report modality achieves the highest diagnostic results, followed by the EHR modality. Combining the two modalities in text form (Report+EHR) can bring significant performance gains. For the visual modality, the performance gap between different backbone architectures is relatively small. While for the text-based modality, LLM-based
29

---

Table 7: Performance with variance on four datasets of different anatomies. All results calculated over independent runs.
Datasct
Method
FFI
Precision
Recall
AUC
mAP
Acc
Acc-S
Avg
ModFuse
34.46200077
31 3636600002
45 15.0410000
90 10.855300088
31.77s00084
95.34200127
41.44=90299
S2. 02.89-000005
BiomedCLIP
29.5640.0033 32.99:000058
45.04400077
88.7143.001
29.91:00061
39.83+00087
MIMIC-ICDS3
KAD
36.3220.00on
33 80
48 19.33200019
91 95
33. 13.54gamm1
95.12200m
40.27aamS4
0.000
(Ches)
Drl'use
34. 10:000067
33. 73.7020.0077
45 15.34000044
50.5003.0277
31. 31.19:0.073
94.68:00650
38.25:00259
52.09-0:0085
Net
32.76100009
47.95
31.97:00081
94.902:00m4
40.10
53. 0.0005
RAD
39.71
39.07:00000
54.74
93.00:3..0m
36.743001 16
95,00:00000
42.33gamm2
[7,
0.0000
MedFuse
81.33:000010 16.00201
87 87.29100000
87.9910.0034
88.76::00009
79,50200024
79. 19.50
83.50000.00000 0.0000
BiomedCLIP
#1.27:00014
72.87:000034
91 88::0000
87.69=3.0041
87.62:0003
78.35:0004
78
78. 78.35=a0044
0..28
KAD
18 73.9240.000)
90:032000000
87
S8
18.65.00m
78.65200001
83.55-000066
(Pye)
DiFuse
20.69:0
73.72:000090
91 59
89.33
90.38:1:0204
79.00=00021
79.00sa0121
84.29
HeALNet
11.80100001
75. 22:2.000
89 642600000
89.60180000
90.45
90.45100041
79.602000
79 79.60190022
84.39 00009
04.30:0.003
77.52.00000
92.38:0000m
91.322000026
91.88
82.40=00000
82.40.000000
866.630000000
MedFuuse
79. 18
85 96-60.00888
77.99:00006
96. 06.5003..0044
73.61190m61
99 99.34g00m
74.3619.0:4
83.86-
BiomedCLLIP
$1.49:0.1073
87.13+0.1228
97.22:0.0351
19.2240.1114
74.36+0.114
SkinCAP
KAD
82.06ao 1028
86. 79 1790
81 27:277077777
97. 97,800494454
80 80.40gat 1066
99:25gmmm
75.46 10MM
86. 0.006
(Skin)
DiFuse
81. 1820.009
85. 70:000000700
79 79.641200000
94.92:33.115
76.42200.065
99 09.299:00158
77.66:20.22
84.97-0.0208
HEALNet
22.20.000000
88 69.0.1130
88:180000000
92.681a.22
77.972/002s
99.37a00070
0.075
RAD
00.48
89.4819.0730
83.23100036 36
97.97.3009
83.551o0w
99:48
81.32
88.64 0.000
31.5310.000
25. 591000000
36100051
05.500:0:008
87.44200010 to 28:45
0.000
BionedCLIP
11366000033
29 02
66.95
66
84,002220043
26.03
88 08.80000000
58,
=8.214
55, 0,000
KAD
35.09
29. 68160000
85.88190081
27.733:uu
89.69.m001
37.86170071
-5.70000000
(Rmin)
DiFuse
34.11200000
27. 86:60,00022
68.96:100085
82.88:200000
27.88:200024
87.992000191
$1.30
54,00-000005
HEALNet
35.91 =0.000
28. 92+0.0044
67.33:00009
85.04422.057
26. 13::0000
89.55=a0000
56.79 26.79::0.001
55.67=00.000
37.65500005 36.24:0.0099 65.78:0000 87.11#3.0019 30.03:00023
59.6420.0088 18.12200.000
models generally outperform BERT-based models. Among the BERT-baaed models, ClinicalBERT consistently achieves the best performance. Considering model size and practicality, we selected ClinicalBERT as the default text encoder in our RAD framework.
C.3 The Variance of Baselines
Due to space limitations. we do not show the variance of the baselines in Table Here we add the variance of all baselines in Table 7 can be observed that the overall variance of SkinCAP the largest among all datasets, Mcanwhile, there is no significant gap between the variance of different methods, all methods exhibit stable performance across the four datasets.
30

---

hypertension (2580)
diabetes mellitus (1753)
pulmonary collapse (1588)
Inchycardia (1541)
hyperosmolaliny (1355)
heart failure (1244)
complications (1159)
pulmonasy congestion (1127)
lung opacity (1007)
pacumioiia (938)
surgical instruments (925)
occlusion of coronary actery
hypotension (592)
myocardial infarction (556)
chronic ischemic heart discase (515)
arrial fibrillation (459)
chronic airway obstruction (296)
pleutay (226)
primary pulmonary hypertension (195)
pocumaoumiosis (189)
rib(s) (184)
scule postemontaggic snemin (172)
poeumotherax (164)
other congalation defocts
endomyocardial fibrosis (134)
consolication (112)
malignant neoplasm of trachea (99)
atheroselerosis of aorta (94)
bronchitis (81)
pocumenitis (74)
curdoomyoputhy (71)
abnormal enzyme levels (68)
pulmonary etaphysema (62)
palmonary (61)
other pleursl concltions (58)
angina docubitas (46)
eardiac authythiiss (43)
vitamin d deficiency (43)
esophingeal variees (41)
bronchiectasis (36)
pulmonary embolism (32)
poisoning (20)
bacterial endocarditia (20)
polyarterilis nodosa (18)
abscess of lung (17)
scule and subucute endocarditis (11)
arterial thrombosis (10)
Medflee
MadPue
ihalessemia (8)
(8)
KAD
KAD
phlcbitis Ihrombophiebiiis
DuFise
DuFure
acute bronchiolitis (3)
HEALNs
D7
HEALNe
farmers' lung
0.6
0.7
0.8
0.9
1.0
1.1
0.0 0.1
0.2 0.3
0.4
0.5
0.6 0.7 0.8 0.9
AUC
F1
Figure 10: Detailed AUC and F1 for each class in MIMIC-ICDS3. The y-axis is the disease name. The numbers in brackets represent the number of samples with this disease.
C.4 Label-wise Analysis of MIMIC-ICDS3
In addition to evaluating the overall performance of RAD in Section 4.2 comprehensive performance across all categories on MIMIC-ICDS3, As illustrated in Figure I0 our method achieved the highest scores in 41 out of 53 categories across both AUC and Fl metrics. Furthermore, the long-tail categories (classes with fewer than 100 positive samples), our method outperformed the previous SOTA by 1.60% in AUC and by 4.44% in F1. Importantly, the performance gains in these long-tal tail categories exceeded the average improvements observed across all categories, underscoring the robustness and practical utility of RAD under real-world clinical settings.
we also investigated its
C.5 Interpretability Cases
In this subsection, we further explore the textual interpretability of RAD by presenting additional visualization cases. addition, the full names of the abbreviations in Table aree given here. In Table 4 the "PC" is short for Platelet Count, "Bilirubin" is Serum Bilirubin, "ALT" is Alaninc
31

---

[CLS]
conpatios
[TLS) trsaly allend meveta sostae mome
loacyy altend nettl soos, anoss poeumotia palmuracy edettia fradies: pa and litora viows of dhe were pooridad the luags are clear without local cocsolidation, effusion, or peeumotboras cantionediesiine sillooelle appeers within nomeel limits the bony free imperssion no acute Intrathoracie process eerre the is a your odd formale the ao tossdd Levves ladici aotid ieih vulus Dia
sugissas axpttiio scute tstobbooc grooss oer x Bovkow
year-
Fedeeee natral
are
right henidigghegn.
Gundaline:
levels indess int lnndy antvales boca B adii kibleeeb coallo onion pop with of a beutesato
cacaus,
usd with value of 0. ci E 6.0 Eresting drass (ck) wilh a an
indicatons
value 3. C.
cre crevinine with valuue 9. O. folate with of 6. garee with 51 Eamaleen
and with
crudine nb
cel Caeas
unewillool
SdtundaeedE bccoolataa withed 3 0, l pt) with a ayalan of 0. lactate value of 0 votenss
Socezyise wit
Inlacotth oro. 0,
Sodiac:
Unae
magagssi with of ghath 6 0. 8.gthwtth oodee 3. 0. phorghads of S pfplesdet ooa wodae af poleeee ctatue 6. Du anhond 5 i. and blood tels with a volue of 2.0 [odrum)
with vi ot3. 9. R.ghopphee with woww B D, plater tou with or of
of Whiieaeee (ustoiibb33 with of pain and bor both 7
with of i. red
with vollas of 6.0. thymodd stimslating homonn with a value of 7. O. una nitragen with a value of 10. O, vilamin h12 wilh a value of 4 0. Sutn bood on with of D 2491
[SEP]
w/o RAD
RAD
il cepurto fiail rppert moneretce: chee stegi view indicator: yof with babrsiosis and heap- warsening effudios? P with babesiosis and hrop- worsening att alors Pachsaaee portable spoigt
creport o- final report connitation cheet single view indication with baheesinsi and heag- warsening eTuasioes? f yof with bubessonis and heap- worsening ellusions? uchiique portable up sprngn comparison
yat
29 PAP
tidiegs compand the pres fim them sonsing
feeding corqund gree vow film thes
andectete tha have no pulmonary vascular congestion pm haart is nortrll <dtr the jest 62
size dedestai
cat improsion: slght iumas firld pinal
year- old tumale to valaes
dl brover 4 and indacao anr loods
Guideline:
valass toos to indiata low levda, cnd vákas fron to 12 iadiaa tash levis to
lab in indede absolute hosephil count with of of absolate oosinoghil
Key indicsionn
Alerzm
Londa
to tadiallla lon levils and nukas toe bie 10 indaad hiph lovil dhe buap but coast vulie ef dooikdo aopld
with svatce of 5.0, abeolue lymphosyle aboolate nestrophil count
with volun L,
dlalun morseyle with
Alpazzze
o812 B. endrppi cartt with 10, blomne aninetmasseees
(ake) with malae of 10. O, alia phoephatsse with ovaaee of D.arioo
smirotressfrasa) oos)
5.3.3 3. beaapkis wil
4. D. and wis of aaian loal with with anal
Ialinika
camt widi now al
dinect with realue of 10, o [nlmbar aooi with of
ad Dooocet wit analaoof 4. fador viii with of facte wnd Eidtaggm fancteed witha rolamiel with
value
Facturni with andaod 4. factor wita viise factura
O hehheeoelooee valavet D. oothaw to 0
w/o RAD
RAD
[CLS] eopic cepet evoluction..
lad worseting respiratory distress, dtddngg
comprrt firel regert midiograph inchosion worssning
them no
respinaory distress, ev evalusiion comporison: Eaaiigg ss oompond
miierunt change. devotan of Ea right be aetaac with Islated adutesly valanae pereecty hymel epecties the epporsce of then spectiie and her oertuattonn sayeess rnllctt de oelee thin praumceis. borforline size the cardla si silhonete. tortaosity of the thorcii accte. pleural elfisions. preurothooas. 60 your old female. the
the prevauus zudiograph Caze no retevant change devatoo to bilzord ielativdy
accidaa these opecitee thei dustrbttoo
parodtonal spactbes the
pneumonia bertldine sime of cartias silboutte loctuosily the thoracic acota, no pleara effudoos no yceumothomxx <ar > the potieet
the
Guidelize:
wel a focerde en
lo and
ab) vita of 5. 3 dhalie
phind badh
Key indicaton::
Alazza
is o3 poar and tomale voeeeses ia resis benvern and 7 tadocato somnal
anda, vulum from ta Bidia lew bash val an fine io 10 indonnoligh
planiie aminotranseeres
RE Xid G O uakian unai mell eeteeeta 0. calcuiated basi t22 dhbendeettbbb... andoeee cmntaate
Aguersin
5.0. Sootin
ot 32200 gaa will dfd
wlblbaa ladaasda d lacae debylippesmn II a U.
Sersn Eibrkie
esparate aminol lecase ast) with vulus of 6. beopaldeea wnth andaa a2 [UlneE total with value 5 vii o. plasso vila a valeco 3. B.
koaphay as vots of 3.3 redh vobhaaak o raacunta cr a ia witu aestoptlss
of 3. 2. i.mmapppl
bem senateent with valen of b. benaghdan with avaka I 1. acdida with antai of id.
D. the wht (SPP
acrate dnbydrogenas (1) with Gf 6. G.bayylwyris will aolaov 2 eelostkk
of
with (SEP)
w/o RAD
RAD
luatrry pustap day after whipple proondure, contieed shormnss of bmoch. ing ingrrsiion: tpp Casit compandd llag valures rasaa qute du adsarrd tibcila limt stall anauad plourl fluad cn both sides the axlechere pneumothorus beat zilily enlarged before, but Is no pulmonary edemta righteenticlaar trmovrenus paicr lead follows the expected course. vor in the pelient a 79- yoar- old male, the and
[CLS) us flxs report up thett
tis rpport Eraa report ched, B
hideery pollop
after whipple procadure compared to
Iepresder up
mrute lov dae comeeets bivd pudiaration andectaees mal sesene of parat nid in beb disec thet purite ao
preumothorax bast oaady enlarge as Indano, there to padzaceary edema.
ngitt vontiicala tracsy esous pacer follows expected course peinet 75- year- mude
Guideline:
medd bewwee
3 kogd Boci roda adado abenia with andae 2/2 mace
with vit
Senze
indicalone
roads cncaa albumin with a value of D as puratt mas of decabouute
10
of
Rentis
wibarotaaess S. D. calcamm mad with 4f O. chartt at of 3 monrioo 6. digosin wilh value of Furi value of gfocoe with a value 10. beroteent with vodoe heroglabis
at digustn of
Sceax Inu and
avdde 3 201 wit valaz 5.5 Mibaaduu 3.3.0 boxin
Capuciyy
load with 5. ends anddace D. wndeowth arees 9XN v ploqhas
with vulue 3 jm| with oflee o Imm anaia 3. 0 Tmme
of
landdin capaciby total ad cas wits
cade
value at O eolieoaaii of 0. mageesin with a ralees 6. 0. ptwit saleedd
5. D. beskon wIth of d (SEP)
phoughata with value 3. poosse
of wwtsslex--maaaaaaa 5. of
eda with
C Laadeon with avdaa Drier)
w/o RAD
RAD
Figure 11: Visualization of model attention to textual content. Both font size and color intensity reflect attention magnitude, with red highlighting disease- critical indicators mentioned in the guideline.
Aminotransferase, "IBC" is Iron-Binding Capacity, "WBC" is White Blood Cell Count, and "AST" is Aspartate Aminotransferasee. Figure TT presents longer and clearer cases of interpretability on the textual data. The third row is the complete content of Figure I and the other rows are other cases
32

---

Table 8: Performance across different combinations of encoder backbones MIMIC-ICDS Sub-
script with arrows represents the absolute improvement. Our method is highlighted with shading.
Backhone
Method
F1
Precision
Rocall
AUC
mAP
Acc
Acc-S
Avg
ResNett
w/o RAD
34.91
31.01
50.91
91.27
32.24
94.50
38.63
53.35
ClinicaBERT
RAD
39.71..50r
39.07
54.74a.6at
93.00
36.74,.50r
95.40
42.33a.70r
$7.283.037 0031
w/o RAD
37.22
35.77
45.64
91.01
34.05
95.63
41.04
54.34
ChiicalBERT
3.022
41.146.371
51.89
92.70,.07-
37.24..19r
95.78 16t
41.970.931
57.423.081
ResNett
w/o RAD
36.71
33.76
49.99
92.03
34.31
95.02
38.86
54.38
BioCInicalBERr
RAD
39.95 3,9
39.90,.14f
51.72 733t
92.59
36.34, oon
95.89
42.29
56.95 2.11
VIT+
w/o RAD
36.32
34.35
48.99
92.08
33.39
95.18
39.77
54.30
BioClinicalBERT
RAD
40.00..att
39,58,
50.70,.715
92.25
36.38, seo
96.01p. s30
42.52
56.78,
Table 9: Ablation on LLM refinement of RAD on the MIMIC-ICD53,
LLM-refine
Precision
Recall
AUC
mAP
Acc
Acc-S
Avg
38.73
36.94
53.24
92.99 93.00
36.56
95.34
40.43
56.32
39.71
39.07
54.74
36.74
95.40
42.33
57.28
Table 10: Ablation on retrieval knowledge sources. "Ours" is the default setting with four knowledge sources. "+ Google Search" means adding source based on "Ours". Random Drop" means randomly removing one knowledge source for each guideline.
Source
F1
Precision
Recall
AUC
mAP
Acc
Ace-S
Avg
Wiki
39.77
39.11
47.05
93.14
36.93
96.02
41.67
56.24
Research
38.54
36.35
51.41
93.01
36.26
95.47
40.69
55.96
Single Source
Guideline
39.79
39.17
50.32
93.03
37.12
96.02
41.42
56.70
Book
39.49
39.14
47.65
93.11
36.84
96,20
42.04
56.35
Random Drop
40.18
40.15
49.34
93.05
37.35
96.24
43.32
57.09
Multi Source
Ours
39.71
39.07
54.74
93.00
36.74
95.40
42.33
57.28
+ Google Search
40.56
40.01
50.56
92.84
36.97
96.15
42.89
57.14
with different indicators. It can be observed that RAD enables the model to dynamically focus on indicators valuable for the current diagnostic goal based on the retrieved guidelines.
C.6
Ablation Study
In this part, we conduct a comprehensive ablation study to systematically evaluate the impact of architectural backbones, key components, and hyperparameter configurations in RAD.
Ablation on different backbones. In Section 4.4, we demonstrated the impact of RAD model performance when replacing different modality backbones, as reflected in the average metrics, AUC, F1, and mAP. To provide Table 8, such as Precision, Recall, Accuracy, and Acc-S, which collectively illustrate the holistic enhancement of the model in diagnostic tasks.
comprehensive evaluation, have included additional metrics in
Ablation LLM refinement of the retrieved knowledge. To assess the necessity of LLM refinement Section 3.2.1] we further conducted an ablation study by comparing RAD with and without this step. Specifically, we constructed baseline guidelines through direct concatenation of top-k retrieved documents and evaluated the performance on MIMIC-ICDS: The results in Table demonstrate that all metrics have decreased after removing the LLM filtering step, underscoring the importance of regularizing the retrieved text. The LLM refinement not only performs semantic filtering eliminate irrelevant contexts but also standardizes heterogencous medical knowledge into actionable diagnostic guidelines critical enabler for effective downstream knowledge infusion.
Ablation on Knowledge Sources. To investigate the effect of modifying the knowledge base on model performance, we compare each knowledge source's individual performance, as well as the performance of adding or removing sources based on our default setting. As presented in Table 10, clinical guidelines provide the most valuable knowledge, as they directly encode established
33

---

58
58
58-
56
56-
56
AV8
AV8
Ave
54
54
54
52
52
52
50
1.0
0.1
0.01
0.001
0.0001
50
1.0
0.5
0.1
0.05
0.01
50
15
20
95
95
95
94
93-
AUC
AUC
91
8
k
90
1.0
0.1
0.01
0.001
0.0001
1.0
0.5
0.1
0.05
0.01
10
15 20
Figure 12: Analysis of hyper-parameters on MIMIC-ICDS3.
diagnostic criteria, key indicators, and decision pathways specifically designed for clinical practice. Research papers show the lowest contribution, as they often focus on novel discoveries, experimental treatments, or specialized cases rather than established diagnostic standards. For well-established diseases, diagnostic criteria have become a consensus, making cutting-edge research less useful. When applying multiple knowledge sources, the performance of RAD remains stable across different source counts (+0.2 Avg), demonstrating RAD's robustness to knowledge base modifications.
C.7 Cost Analysis.
To evaluate the practical feasibility our framework, we analyze the additional cost of RAD brought by the guideline acquisition process. Since only perform retrieval the label level, which avoids the prohibitive cost of sample-wise retrieval. The retrieval process incurs negligible computational overhead. The additional cost primarily occurs during the LLM refinement phase, where the retrieved documents are processed by LLMs for each label of the dataset. When using Qwen2.5-72D model, the average processing time is 33.83s per label. The total preprocessing time for guideline retrieval and refinement on MIMIC-ICD53 is around 31 minutes. The cost can be further reduced using smaller LLMs. When expanding new datasets, the linear growth of retrieval cost (Ndisaa))) ensures efficient scalability, as it grows significantly slower than patient samples O(Neample in real-world scenarios. Furthermore, the retrieval and refinement steps are executed once per dataset during preprocessing, eliminating runtime delays during clinical deployment. In general, RAD achieves knowledge infusion with minimal practical overhead.
C.8 Hlyper-parameter analysis
To evaluate the impact of hyperparameters RAD, we conduct experimental analysis on the three key hyperparameters a, 3, and top-k. The hyperparameter a determines the weight of the guideline- enhanced contrastive learning for visual and text features. And 3 determines the weight of binary cross-entropy loss and the quideline-enhanced contrastive loss. Top-k controls the number of retrieved documents for each disease (label). Figure [2 presents the performance trends as these parameters vary. As 3 decreases, the model performance initially improves before declining. This pattern arises because an excessively high weight over-prioritizes optimization of the primary classification loss. On the contrary, very low weight also leads to performance degradation, underscoring the utility of the guideline in refining multi-modal feature representations. exhibits similar pattern. The optimal values for and 3 are le 2 and le 1.
auxiliary loss, disrupting
34

---

respectively. Regarding the top-k hyperparameteer the model achieves worst performance at it = 1, with gradual improvement as increases. However, performance plateaus after reaching a threshold (I: = 10 here). When retrieving too few documents, limited informative content leads to suboptimal results. Conversely, retaining excessive documents beyond the threshold primarily introduces noisy knowledge, as core disease-related information has already been captured within the top-ranked documents.
35


Reviews:
[{"review_id": "jHKLirp1ip", "reviewer": "Reviewer_QEAp", "summary": "This paper introduces Retrieval-Augmented Diagnosis (RAD), a framework designed to enhance credible multimodal clinical diagnosis. Current AI-driven medical research often relies on implicitly encoded knowledge, ignoring the needs of specific tasks. RAD solves this problem by explicitly injecting external medical knowledge through the following three key mechanisms: retrieving and optimizing disease-centric guidelines, applying a guideline-enhanced contrastive loss function to align features, and using a dual diagnosis decoder to guide cross-modal fusion. The paper also constructs the MIMIC-ICD53 dataset and conducts experiments on 4 datasets to verify the effect. The paper has sufficient work, but may need to be strengthened in theory.", "questions": "1.Introduce the innovation of the retrieval process and give an explanation from a theoretical level.\n\n2.Analyze cases where retrieved reports are incorrect.\n\n3.72B Offline models may introduce bias and should be supplemented with manual verification instructions or ablations of different large model refinements.\n\n4.The comparative experiments lack comparison results with large language models, such as Qwen-vl, llava-med, and huatuogpt-vision.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. This paper systematically integrates multi-source medical knowledge into the diagnostic pipeline, addressing the limitation of implicit knowledge encoding in prior methods.\n\n2. The MIMIC-ICD53 dataset is constructed to provide a reliable testing benchmark.\n\n3. Extensive experiments are conducted on four datasets to effectively verify the model.\n\nWeaknesses\n\n1. No contribution in search strategy. Search strategy is the most important part of RAD, but this article is not innovative enough in search method and directly uses MedCPT as a search engine.\n\n2. Retrieval mainly focuses on text modality and lacks the utilization of multimodal information.\n\n3. The LLM-based knowledge refinement and dual decoder structure may introduce higher computational costs, though the authors note retrieval is label-level and manageable.\n\n4. The framework relies on a fixed knowledge base and lacks experimental effects of modifying the knowledge base or discussions on knowledge base updating algorithms.\n\n5. There is a lack of discussion on the correctness of the retrieved results. If the wrong report is retrieved, it may lead to further error accumulation. \n\n6. The class labels of MIMIC-ICD53 are too unbalanced, which may bring serious obstacles to model training.\n\n7. The data refinement process directly uses the 72B offline model and lacks manual verification, which may lead to data deviation.", "was_revised": true, "final_justification": "The authors have mostly addressed my questions.", "rebuttal": "We sincerely thank Reviewer QEAp for your valuable time and constructive feedback. Below, we provide our responses to each point.\n\n>Weakness & Question 1: No contribution in search strategy/retrieval process.\n\nWe apologize for some potential misunderstanding, and would like to clarify that our **research focuses on how to effectively inject retrieved knowledge into multi-modal discriminative models, rather than designing novel search algorithms**. Specifically, the novelty of RAD lies in the effective injection of disease-specific diagnostic knowledge acquired by the widely used retriever MedCPT[1,2] with the disease name as the query.\n\nAlthough search strategy innovation is important for dynamic retrieval in RAG systems, it is not necessary for our setting. Our retrieval goal is simply obtaining disease-specific knowledge. This purpose is sufficiently addressed by standard retrieval with MedCPT, since disease names already provide precise semantics for guideline retrieval.\n\nWe will revise Section 3.2 of our paper to clarify this research focus and reduce any potential misunderstandings about our contribution.\n\n[1] Benchmarking retrieval-augmented generation for medicine\n\n[2] Improving retrieval-augmented generation in medicine with iterative follow-up questions\n\n>Weakness 2: Retrieval lacks the utilization of multimodal information.\n\nEssentially, aligning the machine diagnostics with the clinical guidelines makes the model more trustworthy. This is the target of injecting the clinical guideline knowledge into the model learning instead of multimodal information, which is beyond our scope. Diagnostic guidelines—by their nature—are primarily textual knowledge that defines standardized diagnostic criteria.\n\nFor our framework, medical knowledge injection requires the knowledge sources to be authoritative and standardized (e.g., \"The levels of amyloid and tau proteins in cerebrospinal fluid are valuable for the diagnosis of Alzheimer's disease\"). This differs fundamentally from case-based retrieval in multimodal RAG, which seeks similar patient examples. To clearly differentiate our method from standard RAG and multimodal RAG approaches, we have included a comparative analysis table, to which we refer the reviewer in our response to Reviewer UQga's Weakness 1.\n\n>Weakness 3: The LLM-based refinement and dual decoder may introduce higher computational costs.\n\n**LLM-refinement**:\n\nAs detailed in Appendix C.7 of our submission, the computational overhead of retrieving is minimal due to our offline label-level retrieval strategy. The cost scales with O(N_disease) rather than O(N_sample), making it highly efficient. The LLM refinement takes 33.83 seconds per disease label. Total preprocessing time takes only 31 minutes for the largest dataset MIMIC-ICD53. More importantly, these operations are one-time preprocessing steps that introduce no runtime delays during model training/deployment. And using smaller LLMs (e.g., Qwen2.5-7B) can further reduce this cost with minimal performance drop (result table in our response to Weakness 7).\n\n**Dual-Decoder**:\n\nThe computational impact of our dual-decoder is negligible, as shown by component-level analysis:\n\nComponent|Params|FLOPs\n-|-|-\nVision Encoder|31.33M (16%)|87.82G (80.13%)\nText Encoder|135.92M (71%)|21.77G (19.86%)\nDual Decoder|25.22M (13%)|12.65M (0.01%)\n\nDespite accounting for 13% of parameters, the dual decoder contributes only 0.01% of total FLOPs. Because the decoder structure is simply a transformer layer that operates on already-processed embeddings (not raw inputs).\n\nThis confirms that both components introduce small computational overhead while delivering considerable improvements.\n\n>Weakness 4: Experimental effects of modifying the knowledge base or discussions on knowledge base updating algorithms.\n\nThank you for the suggestion. In the following, we conducted experiments that modify the number of knowledge sources to evaluate RAD's robustness:\n- Default: Original version with 4 sources\n- \\- Random Drop: For each disease, randomly remove one knowledge source for guideline retrieval\n- \\+ Google Search: Adding \"Google Search\" as a new retrieval source\n\nSources|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nDefault|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\n\\- Random Drop|40.18|40.15|49.34|93.05|37.35|96.24|43.32|57.09\n\\+ Google Search|40.56|40.01|50.56|92.84|36.97|96.15|42.89|57.14\n\nThe performance of RAD remains stable across different source counts (±0.2 Avg), demonstrating RAD's robustness to knowledge base modifications. As shown in our response to the Question 1 of Reviewer TyCK, even single-source implementations (clinical guidelines alone) achieve strong performance (56.70 Avg), significantly outperforming baselines.\n\nAdditionally, in clinical practice, medical guidelines represent **gold standards** that have become common consensus. Such knowledge evolves slowly (especially for most common diseases), making frequent knowledge base updates unnecessary for most scenarios.\n\n>Weakness 5 & Question 2: Lack of discussion on the correctness of the retrieved results. If the wrong report is retrieved, it may lead to further error accumulation.\n\nTo address this, we conducted controlled experiments where we intentionally replaced 10% and 20% of retrieved guidelines with irrelevant disease guidelines during training:\nWrong%|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\n0|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\n10%|39.91|39.56|49.30|92.94|36.94|95.95|41.97|56.65\n20%|38.66|37.42|49.20|92.89|36.24|95.59|40.09|55.73\nLabel-branch|40.00|39.78|50.80|92.59|36.35|95.88|42.29|56.81\n\nAs shown in the above table, the*wrong knowledge only leads to little performance degradation, demonstrating RAD's built-in error resilience. This **robustness stems from our dual-decoder architecture**: The label branch operates independently using only disease labels as query. Using the label branch alone can achieve 56.8 average performance. Thus, even when the retrieved guidelines are wrong, RAD can still provide robust results.\n\nThis architecture can further prevent error accumulation by prioritizing the label decoder when guideline information is unreliable, maintaining performance around 56.8 Avg.\n\n>Weakness 6: The unbalanced labels of MIMIC-ICD53.\n\nFirst, the class imbalance indeed brings obstacles. However, we would like to clarify **why choosing an imbalanced dataset**. In healthcare settings, disease distributions naturally follow long-tail patterns (e.g., pneumonia is much more common than farmers' lung/pneumoconiosis). Thus, we intentionally preserved the original distribution of MIMIC to align with real-world medical settings.\n\nBesides, we also conducted an experiment on a balanced dataset to demonstrate the effectiveness of RAD. Please kindly refer to our response to Reviewer TyCK's Question 2.\n\n>Weakness 7 & Question 3: The data refinement process lacks manual verification. 72B models should be supplemented with manual verification instructions or ablations of different large model refinements\n\nWe apologize for not clearly detailing our validation process. All final guideline versions underwent rigorous verification by junior doctors from a major hospital. In fact, three critical aspects are checked by physicians, including: MIMIC-ICD53 construction process, final guideline correctness, and double-checking the indicators for Guideline Recall calculation.\n\nFurthermore, we compare the performance with/without manual verification and summarization using a smaller model:\nModel|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nQwen2.5-72B w/ manual|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\nQwen2.5-72B w/o manual|40.24|40.15|50.23|92.72|37.01|96.17|43.33|57.12\nQwen2.5-7B w/o manual|40.20|39.89|49.94|93.16|37.16|96.09|42.92|57.05\n\nEven without manual check, the model only shows a negligible performance drop. This is because the LLM only needs to perform simple summarization tasks on retrieved knowledge rather than generating novel content. And the minimal performance drop between 7B and 72B model demonstrates that 7B model is enough for the summarization task. We will use smaller models in future implementations to lower the costs.\n\n>Question 4: Experiments lack comparison results with large language models, such as Qwen-vl, llava-med, and huatuogpt-vision.\n\nWe have conducted comprehensive comparisons with state-of-the-art multimodal large language models (MLLMs) by converting the data into VQA format. The results are on the FairVLMed dataset since it is the only single-label dataset suitable for VQA. For each MLLM, we evaluated both few-shot and supervised fine-tuning (SFT) settings:\n\nModel|Setting|F1|Precision|Recall|Acc|Avg\n-|-|-|-|-|-|-\nQwen2.5-VL-7B|Few-shot|57.46|73.88|47.01|63.60|60.49\n-|SFT|67.70|72.33|63.63|68.95|68.15\nHuatuoGPT-Vision-7B|Few-shot|65.19|61.39|69.50|62.05|64.53\n-|SFT|69.74|71.64|67.93|69.85|69.79\nLingshu-7B|Few-shot|28.91|73.60|17.98|64.85|46.34\n-|SFT|71.91|65.93|79.08|68.40|71.33\nOurs|-|84.30|77.52|92.38|82.40|84.15\n\nSignificant performance gaps exist between RAD and all MLLMs (>12 Avg), even though the single-label dataset is already the simplest for MLLMs. (Multi-label datasets require the model to complete classifications for all labels at once.)\n\nFrom the results, we can find that complex diagnostic tasks are better suited for discriminative models than generative MLLMs, as evidenced by the substantial performance gap. Moreover, RAD (and all our baselines) can achieve superior performance with significantly lower resource cost than MLLMs.\n\nRegarding llava-med that the reviewer mentioned, we did not include it as it's outdated (2023) with poor capability. Instead, we add Lingshu-7B, a new medical MLLM released in June 2025. We will add more clarification in our model selection."}, {"review_id": "JC4DWO6IK2", "reviewer": "Reviewer_UQga", "summary": "This paper introduces RAD, a Retrieval-Augmented Diagnosis framework that enhances multimodal clinical diagnosis by injecting refined disease-specific external knowledge (e.g., clinical guidelines, medical textbooks, research papers) into the diagnostic process. RAD uses a three-pronged approach: (1) retrieving and refining guideline knowledge, (2) embedding this knowledge in multimodal representations via a contrastive loss, and (3) integrating it through a dual diagnostic decoder. The authors also propose a dual-axis interpretability evaluation framework.", "questions": "1. How does RAD handle errors or outdated content in the knowledge corpus or LLM-generated summaries? Have you analyzed the effect of injecting incorrect or incomplete guidelines on downstream accuracy or interpretability? \n\n2. Attention weights are used to proxy adherence to diagnostic indicators. Have you considered augmenting your interpretability evaluation with external expert review or ground-truth decision rationales?\n\n3. Could you expand on the differences between your method and retrieval-augmented generation or multimodal RAG approaches?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths \n- RAD introduces a systematic method to inject disease-specific guidelines into the entire diagnostic pipeline, including input augmentation, feature extraction, and decision fusion, rather than limiting knowledge use to pretraining stages.\n- Across four anatomically diverse datasets (MIMIC-ICD53, FairVLMed, SkinCAP, NACC), RAD consistently outperforms strong baselines like BiomedCLIP and KAD in terms of F1, AUC, and sample-wise accuracy.\n\n\nWeaknesses\n1. The main novelty lies in the explicit combination and systematic integration of prior methods (retrieval, LLM summarization, contrastive alignment, and dual decoder); however, the technical advances in each are modest individually. There is limited comparison or differentiation from conventional retrieval-augmented generation or discriminative frameworks. The distinction from RAG approaches is only briefly justified.\n2. The approach assumes that a suitable disease guideline exists, that LLM summaries are reliable, and that retrieved content truly covers all relevant diagnostic edge cases. The refinement of retrieved documents via Qwen2.5-72B is central to RAD, yet the quality and consistency of LLM-generated summaries are not critically evaluated.\n3. While improved averages are shown, the paper does not analyze or visualize errors where adherence to retrieved guidelines leads to mistakes or misses patient-specific context.\n4. While interpretability evaluation is a valuable addition, the reliance on attention weights (text and image) as evidence of adherence is controversial.\n5. The pipeline for retrieving, refining, and updating guidelines is resource-intensive (uses Qwen2.5-72B), potentially limiting reproducibility in resource-constrained or privacy-sensitive settings. \n6. The paper benchmarks against baselines in multimodal medical AI, but omits RAG-style methods tailored to clinical reasoning or explanation. Explanation for specific baseline choices or additional comparisons would strengthen the empirical claim.\n7. Occasional jargon (“dual diagnostic transformer decoder”, “feature prototypes”), ambiguous notation in equations (Section 3.2.2), and dense referencing to appendices can break the narrative flow. \n8. While the manuscript emphasizes clinical workflows, an analysis of clinician utility, end-user validation, or prospective trials in realistic deployment environments is missing.", "was_revised": true, "final_justification": "The authors have satisfactorily addressed most of my concerns, so I am increasing my rating accordingly.", "rebuttal": "We sincerely thank Reviewer UQga for your valuable time and constructive feedback. Below, we provide our responses to each point.\n\n>W1 & Q3: The main novelty lies in the explicit combination and systematic integration of prior methods (retrieval, LLM summarization, contrastive alignment, and dual decoder); technical advances in each are modest individually. Limited comparison from conventional RAG/multimodal RAG/discriminative frameworks.\n\nWe sincerely apologize for any confusion caused to the reviewer, and would like to clarify the novelty of RAD, which is not merely a combination of existing techniques.\n\n- First, we acknowledge that retrieval and LLM summarization are not our pioneering contributions. Our retrieval goal is simple: obtaining disease-specific knowledge. This purpose is sufficiently addressed by standard retrieval with the disease name as the query. Disease names already provide precise semantics for retrieving relevant knowledge. We did not propose novel query optimization or search strategy, since existing technology is enough to meet our needs.\n\n- Second, RAD is a novel method for effectively injecting guidelines into the diagnosis model. Specifically, we propose utilizing guidelines as anchors and design the GECL loss to simultaneously align text and images, aiming to reduce the distance between multimodal representations and corresponding guidelines. To further utilize guidelines to steer the fusion of multimodal information, we design a dual-decoder structure to explicitly leverage guidelines for cross-modal interaction. **The GECL loss and the dual-decoder are our novel design tailored for medical knowledge injection**.\n\n- Finally, another major contribution of RAD lies in advancing trustworthy medical AI by designing **a comprehensive evaluation framework and metrics to measure the model's interpretability**, which doctors favour.\n\nIn the following, we enumerate the major differences between RAD and other methods:\nMethod|RAD|RAG|Multimodal RAG|Discriminative Baselines\n-|-|-|-|-\nTask Type|Discriminative|Generative (QA)|Generative (VQA)|Discriminative\nTask-specific Knowledge|√|√|√|×\nRetrieval Goal|Diagnosic Guidelines|Dynamic Knowledge|Similar Cases|-\nRetrieval Granularity|Label-level|Sample-level|Sample-level|-\nRetrieval Difficulty|Easy|Hard|Hard|-\nInjection Approach|Input augmentation (text), feature learning, and decision-making|Input augmentation (text)|Input augmentation (text/image)|-\nTechnical Focus|Inject knowledge to guide training|Search strategy, query optimization...|Retriever optimization, Reranking...|Pretraining, multimodal fusion\n\nFor RAG & multimodal RAG, the fundamental difference lies in the goal of retrieval. RAG methods usually dynamically retrieve textual knowledge for current questions; Multimodal RAG primarily retrieves similar data samples(e.g., image-report pairs), especially for medical multimodal RAG[1]. While our method retrieves diagnostic standards once to constrain training on all samples. And we incorporate the knowledge to guide multimodal representation learning, modality fusion, participating in entire training pipeline. Our method also has major differences with RAG in retrieval difficulty, retrieval complexity, etc.\n\nDiscriminative methods typically do not inject domain-specific knowledge on downstream tasks and often focus solely on pre-training on extensive corpus for text encoders. While our approach internalizes guidelines not only within text encoders but also impacts all parameters, including visual encoders and dual decoders.\n\nWe will revise Section 2 (Related Work) to better explain these distinctions.\n\n[1] MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models\n\n>W2: The quality and consistency of LLM-generated summaries are not critically evaluated.\n\nThank you for the suggestion. In fact, all final guideline versions underwent rigorous verification by junior doctors from a major hospital. Three critical aspects are checked by doctors, including: MIMIC-ICD53 construction process, final guideline correctness, and double-checking the indicators for Guideline Recall calculation. Specifically, for MIMIC-MM53, a total of 53 guidelines were reviewed, and physicians made moderate modifications to only 6 of them.\n\nWe further compare the performance with/without manual verification and refinement via a smaller LLM:\nModel|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nQwen2.5-72B w/ manual|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\nQwen2.5-72B w/o manual|40.24|40.15|50.23|92.72|37.01|96.17|43.33|57.12\nQwen2.5-7B w/o manual|40.20|39.89|49.94|93.16|37.16|96.09|42.92|57.05\n\nEven without manual check, the model only shows a negligible performance drop. This is because the LLM only needs to perform simple text summarization tasks rather than generating novel content.\n\n>W3 & Q1: Analyze errors where adherence to guidelines leads to mistakes or misses patient-specific context. The effect of injecting incorrect/incomplete guidelines on downstream accuracy\n\nThank you for the suggestion. Please refer to experiments and discussion in response to Reviewer QEAp's Weakness 5.\n\n>W4 & Q2: The reliance on attention weights (text and image) as evidence is controversial. Have you considered augmenting your interpretability evaluation with external expert review or ground-truth decision rationales?\n\nAlthough attention-based methods have some limitations, we should note that\n\n1. Attention weight-based methods like Grad-CAM[1] have been widely accepted as standard interpretability tools[2,3]. In medical tasks, attention-based localization is widely used, with validation against physician-annotated ground-truth[4,5].\n2. Our visual interpretability metrics in Table 4 directly align with physician-verified lesion annotations (which are the gold standard in diagnosis, enough to serve as ground-truth decision rationales). This high performance provides clinical validation beyond pure attention visualization.\n3. In clinical practice, medical guidelines represent gold standards that have become common consensus. Only highly recognized diagnostic criteria can be included in textbooks and guidelines. The RAD model can exhibit high-level alignment with such guidelines, demonstrating that its interpretability is relatively reliable.\n\n[1]Grad-cam: Visual explanations from deep networks via gradient-based localization\n\n[2]Transformer interpretability beyond attention visualization\n\n[3]Attention-based Interpretability with Concept Transformers\n\n[4]A foundation model for generalizable disease detection from retinal images\n\n[5]Large-scale pancreatic cancer detection via non-contrast CT and deep learning\n\n>W5: Retrieving, refining guidelines is resource-intensive (uses Qwen2.5-72B), limiting reproducibility in resource-constrained/privacy-sensitive settings.\n\nAs detailed in Appendix C.7 of our submission, the computational overhead of retrieving is minimal due to our offline label-level retrieval strategy. The cost scales with O(N_disease) rather than O(N_sample), making it highly efficient. The LLM refinement takes 33.83 seconds per disease label. Total preprocessing time takes only 31 minutes for the largest dataset. More importantly, these operations are one-time preprocessing steps that introduce no runtime delays during model training/deployment. And using smaller LLMs (e.g., Qwen2.5-7B) can further reduce this cost with minimal performance drop (result table in our response to Weakness 2).\n\nFor privacy-sensitive settings, we use the LLM to summarize the retrieved knowledge for certain diseases, not the patient context. Thus, the LLM usage will not disclose patients' privacy. \n\n>W6: Omits RAG-style methods. Explanation for specific baseline choices or additional comparisons would strengthen the claim.\n\nTo address your concern, we implemented RAG baselines on MLLMs like Qwen2.5-VL. The experiments are on the single-label dataset FairVLMed. Specifically, we use the same guidelines used by our model as the RAG input for fair comparison.\nMethod|Model|F1|Precision|Recall|Acc|Avg\n-|-|-|-|-|-|-\nw/o RAG|Qwen2.5-VL-7B|57.46|73.88|47.01|63.60|60.49\nw/ RAG|Qwen2.5-VL-7B|59.68|75.00|49.56|65.55|62.45\nw/o RAG|HuatuoGPT-Vision-7B|65.19|61.39|69.50|62.05|64.53\nw/ RAG|HuatuoGPT-Vision-7B|66.66|58.59|77.32|60.45|65.76\nw/o RAG|Lingshu-7B|28.91|73.60|17.98|64.85|46.34\nw/ RAG|Lingshu-7B|29.90|77.20|18.54|64.95|47.65\n-|Ours|84.30|77.52|92.38|82.40|84.15\n\nEven with RAG, the performance gaps between MLLMs and discriminative models are still significant. This indicates that adding the guideline in input of generative models does not offer significant improvement, since diagnosis requires fundamentally different knowledge integration than generative tasks.\n\n>W7: Jargon (“dual diagnostic transformer decoder”), ambiguous notation (Section 3.2.2), and dense referencing to appendices break the narrative flow.\n\nWe sincerely apologize for the confusion. The manuscript will be revised to replace jargon with clear descriptions (e.g., from \"dual diagnostic transformer decoder\" to \"dual decoder\"). For a smooth narrative flow, we will put all references to appendix at the end of each section. We will also revise Section 3.2.2 to provide clearer notations.\n\n>W8: Analysis of clinician utility, end-user validation, or prospective trials in realistic deployment environments\n\nThank you for your valuable suggestion. As a primary method exploration, our ultimate goal is to promote clinical practice. We are currently collaborating with the thoracic surgery department at a major hospital to deploy RAD for auxiliary examination. However, due to patient privacy protection and the need to fulfill relevant procedures, the ethics approval process is still ongoing. We will include the results and analysis of clinical validation once all regulatory procedures are done. But we believe that, for NeurIPS, the current extensive validation on open datasets is still persuasive and fits this community."}, {"review_id": "tZas4sWONo", "reviewer": "Reviewer_jbMM", "summary": "This work introduces a framework (RAD) that explicitly injects disease-specific clinical guidelines into multimodal diagnostic models. The pipeline (i) retrieves disease-centric documents from four curated medical corpora and refines them with a large-language-model summariser, (ii) aligns image and text representations to these guidelines via a Guideline-Enhanced Contrastive Loss (GECL) term, and (iii) fuses modalities with a dual‐decoder transformer whose queries are the guideline text and the disease labels. A dual-axis evaluation scheme is proposed to quantify interpretability. RAD is evaluated against other competitors across four anatomies (chest, fundus, dermatology, and brain).", "questions": "- Figure 2 depicting the Dual Diagnostic Network appears asymmetric. The two branches seem to contain different amounts of information, with the guidance branch being inclusive/indicative of the label. It is unclear if this dual-branch design is strictly necessary, and an ablation study on a single-branch design would strengthen the argument for its necessity (though current study swaps to an MLP, not one-branch).\n\n- The training processes for other comparison methods are not clearly elaborated, making it difficult to ascertain the fairness of the comparison with existing methods. If RAD's improvements are primarily due to learning more external knowledge, which other methods might not have been exposed to, then the observed gains might stem largely from this additional knowledge rather than the pipeline novelty itself. If so, can RAG's knowledge injection strategy/framework be applied to other models—for example, BiomedCLIP?\n\n- Equation (7) is an important loss of RAD and is the base of the method. But it's not clear to me if the the text encoder, which is used to obtain embeddings for input text and guidelines, is optimized?  If the encoders are frozen, the GECL term can only shift the lightweight projection heads or prototypes, so its ability to “pull” patient text T toward guideline G seems limited.  Conversely, if the encoders are updated, both T and G are re-encoded each step. In this case, how to ensure the guideline embedding G still faithfully represent the original guideline semantics? \n\n \n- Providing a couple of complete examples of the crafted condition guidelines in the main paper (or a prominent appendix section) would greatly enhance reader understanding of what these guidelines look like and could help extend the research. The appendix shows the prompt template, not concrete examples.\n\n- While Figure 1 and Table 3 show that RAD pays more attention to representative laboratory indicators/keywords. The current evidence—higher guideline-recall scores and attention heat-maps—mainly shows correlation: GECL already pulls representations toward guideline tokens, so a larger recall is almost guaranteed even if the model’s final decision does not truly hinge on those tokens. To demonstrate that RAD’s predictions are causally driven by guideline evidence, consider adding a counter-factual test (e.g., mask or perturb key indicators at inference) to see if performance drops.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Overall, the motivation behind this work is clear and aligns with current calls for trustworthy medical AI, where expert knowledge is critical to ensuring reliable decision-making. The work is well-written and the method is clearly explained. \n\n- The experimental setup is robust, and the RAD method's applicability is successfully demonstrated across diverse medical modalities (chest, eye, skin, and brain). The extensive evaluations on four datasets validate its generalizability and superiority against state-of-the-art baselines.\n\n- The proposed Guideline-enhanced Feature Constraint (GECL) is empirically proven to be an effective component, contributing to the improved performance and interpretability of RAD.\n\nWeaknesses\n- It remains unclear that the dual-decoder design—one of the paper’s headline contributions—provides value beyond a simpler alternative. \n\n- Because training protocols for baselines are under-specified, the performance gap may partly reflect unequal access to external knowledge rather than genuine architectural superiority, casting doubt on fair comparison. \n\n- The optimisation scope of GECL, which is an important component of the proposed method, is ambiguous, making it hard to judge whether the claimed guideline-driven reasoning is robust or merely correlational.\n\n(Details are in the questions)", "was_revised": true, "final_justification": "The submission was a solid work but I had a few detailed technical questions about the method. The rebuttal addressed all my concerns and I support the acceptance of this submission.", "rebuttal": "We sincerely thank Reviewer jbMM for your valuable time and constructive feedback. We appreciate your recognition of our clear motivation and robust experimental validation. Below, we provide our responses to each point.\n\n>Weakness & Question 1: Necessity of the dual-decoder design.\n\nThank you for the suggestion. To address your concern, we have conducted an ablation study comparing four different decoder structures:\n- MLP: A single MLP head\n- Label-branch: Only the label decoder branch\n- Guideline-branch: Only the guideline decoder branch\n- Dual-branches: Our dual-decoder structure\n\nThe experimental results on MIMIC-ICD53 are as follows:\n\nModel|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nMLP|39.34|37.74|51.87|92.94|36.36|95.59|39.95|56.26\nLabel-branch|40.00|39.78|50.80|92.59|36.35|95.88|42.29|56.81\nGuideline-branch|39.65|38.96|53.60|92.76|36.43|95.42|41.37|56.88\nDual-branches|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\n\nKey findings:\n\n1. The dual-branch design achieves the best average performance. Neither single branch alone can match the performance of the dual-branch architecture (while still better than MLP). \n\n2. While the guideline branch's query contains label information, the two branches can still learn complementary information. \n\nAnalysis: The guideline branch achieves higher recall (53.60% vs 50.80%), which means it tends to predict positive cases more aggressively. While the label branch gives more conservative predictions, showing higher precision (39.78% vs 38.96%) and sample-wise accuracy (42.29% vs 41.37%). This is because the guideline branch can make bold diagnoses based on the symptoms/key indicators mentioned in the guideline, minimizing false negatives. While the label branch (with only the disease names as the query) develops stricter decision boundaries. Thus, **combining the two branches can further improve performance**.\n\nWe will include this ablation study in Section 4.4 of our revised manuscript.\n\n>Weakness & Question 2: Training protocols. If RAD's improvements are primarily due to learning more external knowledge, which other methods might not have, the observed gains might stem largely from this additional knowledge rather than the pipeline novelty itself. If so, can RAD's injection strategy be applied to other models—for example, BiomedCLIP?\n\nWe apologize for not clearly detailing the training protocols. To ensure fair comparison, all models were trained under identical conditions, with the same optimizer (AdamW,lr=1e-4, weight_decay=0.01), training epochs, and so on. Detailed configuration for each dataset is available in the config files of our anonymized code repository.\n\nTo address your concern about knowledge access and application to other models, we conducted the following experiments:\n- BiomedCLIP+Guideline: Adding the guideline (same as RAD) into the input of BiomedCLIP\n- BiomedCLIP+RAD: Adding RAD's knowledge injection strategy to BiomedCLIP\n\nDataset|Model|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-|-\n-|BiomedCLIP|32.99|29.56|45.04|88.71|29.91|94.72|39.83|51.54\nICD53|BiomedCLIP+Guideline|32.57|30.68|46.89|90.05|29.55|94.36|38.18|51.75\n-|BiomedCLIP+RAD|34.99|33.11|48.94|91.70|32.40|94.86|39.89|53.70\n-|BiomedCLIP|81.49|87.13|81.41|97.22|79.22|99.11|74.36|85.71\nSkin|BiomedCLIP+Guideline|81.60|86.95|79.49|97.62|79.72|99.38|78.20|86.14\n-|BiomedCLIP+RAD|84.70|88.87|82.66|98.00|82.48|99.44|80.59|88.11\n\nResults on two datasets show that BiomedCLIP+Guideline can only bring a slight improvement over the original model. While combining RAD on BiomedCLIP can markedly enhance performance (>2% Avg). This demonstrates that **RAD's gains are not solely from the external knowledge**. Its superiority stems from our novel pipeline design rather than unequal knowledge access. The pipeline is essential for effective knowledge integration.\n\n>Weakness & Question 3: The optimisation scope of GECL. It's not clear to me if the text encoder, is optimized? If the encoders are frozen, the GECL term can only shift the lightweight projection heads or prototypes, so its ability to “pull” patient text T toward guideline G seems limited. Conversely, if the encoders are updated, both T and G are re-encoded each step. How to ensure the guideline embedding G still faithfully represent the original guideline semantics?\n\nThank you for raising this insightful suggestion! In our original implementation, all components, including the text encoder are optimized. We agree that updating the text encoder may affect the faithfulness of G. Our implementation will be changed to the frozen version. We conducted experiments to investigate how training the text encoder influences the performance.\n\nMethod|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nTuneable|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\nFrozen|39.65|38.96|54.60|93.00|36.73|95.42|42.35|57.24\n\nResults show that the performance difference was minimal (only -0.04% Avg).\n\nWe think that lightweight projection heads have sufficient capacity to align patient text with guideline semantics without modifying the fundamental text representations. This is because the module operates on the embedding learned by the pretrained text encoder, which is sufficiently compact, semantical, and easily transformed through lightweight projector layers.\n\nMoreover, the contrastive learning objective in GECL primarily adjusts the relative positions in the embedding space rather than radically changing individual embeddings. This maintains the semantic integrity of guidelines while allowing for task-specific alignment.\n\nWe will revise this implementation detail of the frozen text encoder in future updates.\n\nRegarding your concern about whether RAD  is \"robust or merely correlational,\" please refer to our response to Question 5.\n\n>Question 4: Examples of the crafted guidelines.\n\nBelow is an example for the guideline of \"bronchitis\":\n\n\"### Summary of Key Diagnostic Features for Bronchitis\n\n**Disease Description:**\\nBronchitis is an inflammation of the bronchi, the air passages in the lungs. It can be classified into two main types: acute and chronic. Acute bronchitis is typically a self-limiting condition characterized by a cough that may produce sputum and is often caused by viral infections. Chronic bronchitis, on the other hand, is a long-term condition defined by a productive cough lasting for at least three months in two consecutive years, often associated with chronic obstructive pulmonary disease (COPD). The primary risk factor for chronic bronchitis is tobacco smoking, with other factors including air pollution and occupational exposures.\n\n**Important Lab Tests and Values:**\\n- Acute Bronchitis:\\n  White Blood Cell Count (WBC): Usually normal or slightly elevated.\\n  C-reactive Protein (CRP): May be slightly elevated but not typically high.\\nSputum Culture: Not routinely necessary, but can be useful if bacterial infection is suspected. Chronic Bronchitis:\\n  Pulmonary Function Tests (PFTs): Reduced FEV1/FVC ratio, indicating airflow obstruction.\\nSputum Analysis: Increased mucus production, often with neutrophil infiltration.\\nBlood Gas Analysis:** May show hypoxemia and hypercapnia in advanced cases.\n\n**Key Radiological or Clinical Findings:**\\nAcute Bronchitis:\\n Chest X-ray: Usually normal, but may show hyperinflation or peribronchial thickening.\\nPhysical Examination: Wheezing, crackles, and rhonchi on auscultation. Chronic Bronchitis:\\n Chest X-ray: May show hyperinflation, increased bronchovascular markings, and signs of emphysema.\\nCT Scan: Can reveal bronchial wall thickening and mucus plugging.\\n Physical Examination: Barrel chest, cyanosis, and signs of cor pulmonale in advanced cases.\n\n**Diagnostic Symptoms or Relevant Clinical Features:**\\nAcute Bronchitis:\\n Cough: Initially dry, then becomes productive with clear or yellowish sputum.\\n Fever: Usually mild or absent; high fever suggests pneumonia.\\n Fatigue and Body Aches: Common but generally mild.\\n Wheezing and Shortness of Breath: May be present, especially in patients with underlying asthma.\nChronic Bronchitis:\\n Cough: Persistent, productive cough with sputum, often for at least three months in two consecutive years.\\n Dyspnea: Shortness of breath, especially on exertion.\\n  Wheezing: Common, especially in the morning.\\n  Chest Pain: May occur due to prolonged coughing.\\n  Fatigue and Malaise: Persistent, often due to chronic hypoxemia.\"\n\nMore examples of full guidelines will be added to Appendix B.2.\n\n>Question 5: Counter-factual test of causality\n\nThank you for this insightful suggestion. We conducted a counter-factual test by randomly masking key indicators in the input text at inference time. We masked 0%, 10%, 30%, and 50% of the indicators of the MIMIC-ICD53 test set.\n\nmask|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\n0%|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\n10%|38.44|35.80|50.61|92.76|35.54|95.55|39.34|55.43\n30%|34.74|31.60|49.53|91.84|32.26|94.75|37.97|53.24\n50%|29.74|25.17|47.14|89.91|26.31|93.85|34.03|49.45\n\nAt lower masking ratios (10%), the model exhibits relatively small performance degradation. However, as the masking ratio increases, we observe a significant performance drop (10% F1 decrease at 50% masking). This demonstrates that model performance is highly sensitive to the presence of key indicators. (The model can still maintain certain performance even with 50% masking because RAD has multimodal inputs, including image, report, and EHR.)\n\nThe results confirm that **RAD's predictions are causally dependent on guideline-relevant information**, not merely correlational. Our model's decision-making process hinges on these specific clinical indicators identified by the guideline.\n\nWe will include this part in our revised manuscript, providing more evidence that RAD follows the guideline to focus on key indicators."}, {"review_id": "wXKZg85JZC", "reviewer": "Reviewer_TyCK", "summary": "The paper presents a model for clinical diagnosis based on images and medical records, guided by \"knowledge\" about a set of specific diseases. The overall model consists of visual and text encoders that feed corresponding embeddings into attention-based label and guideline decoders. The model is trained and evaluated on several datasets demonstrating improved performance versus other models across all datasets. The paper appears to be comprehensive, with relevant ablation studies, derivation of loss function, details of datasets, etc., however some model training details are omitted, such as the number of epochs.", "questions": "It would be helpful to understand which of the four sources used for knowledge retrieval are the most relevant to the performance of the model. For example, would just the clinical practice guidelines be sufficient?\n\nDo you think the results can be improved if a more balanced dataset is used? Or can some data augmentation be used to enrich the parts of the dataset where only a few samples are present?", "limitations": "The only limitation listed by the authors is due to the static retrieval knowledge corpus. I would not consider this to be a limitation as new knowledge can always be added to the knowledge base and the model can be re-trained. I think a bigger limitation of this work is a limited number of classes of diseases the model is trained to recognize as this is a classifier model. New diseases cannot be recognized even if there is an adequate knowledge about them is present in the knowledge base.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Overall quality of the work seems good, with good justification, detailed enough description of the proposed model and datasets, ablation studies, etc. Some additional details and experiments would strengthen the work, see \"Questions\" section of this review. While the work demonstrates clear improvements compared to other models, these improvements are still not sufficient to make this work practical as a medical diagnostics tool. \n\nWhile not entirely novel, model guidance through augmentation of relevant knowledge retrieved from four different sources seems to be helpful to achieve the stated performance. This is definitely a strength of this work.\n\nNice to see the construction of another dataset, MIMIC-ICD53. However this dataset seems to be hugely disbalanced. Not surprising, Figure 8 indicates model's low performance for many diseases that do not have sufficient number of samples.", "was_revised": true, "final_justification": "The authors provided sufficient new details that clarify my initial concerns.", "rebuttal": "We sincerely thank Reviewer TyCK for your valuable time and constructive feedback. We appreciate your recognition of our paper quality and the construction of the MIMIC-ICD53 dataset. In the following, we provide our responses point-by-point.\n\n>Weakness 1 and Limitation: Improvements are still not sufficient to make this work practical as a medical diagnostics tool. And a bigger limitation of this work is a limited number of classes of diseases the model is trained to recognize as this is a classifier model. New diseases cannot be recognized even if there is an adequate knowledge about them is present in the knowledge base.\n\n**New diseases**:\n\nWe would like to explain that a key advantage of RAD's architecture is its inherent zero-shot potential through the text-query mechanism:\n\nUnlike standard classifiers (e.g., MLP) with fixed output shapes, RAD uses text queries (both disease labels and guidelines) as inputs to the decoder. This means the model can flexibly process any disease name, regardless of whether it was seen during training. Given a new disease, we can directly retrieve the corresponding guideline with the disease name, and then perform zero-shot classification at the lowest cost.\nFollowing your question, we conducted zero-shot experiments on the Chest-Det dataset (which contains 10 new disease categories). We use checkpoints trained on the MIMIC-ICD53 dataset to perform zero-shot inference.\n\nChest-Det|Acc|AUC|mAP|F1|Avg\n-|-|-|-|-|-\nBiomedCLIP|38.93|56.73|22.57|31.39|37.41\nKAD|40.33|53.91|21.59|30.19|36.51\nOurs|54.63|60.26|24.47|32.16|42.88\n\nRAD demonstrates meaningful zero-shot capability, achieving 60.26 AUC on unseen diseases.\nAnd our method significantly outperforms baselines, proving that RAD can effectively leverage the knowledge base for novel diseases. For medical AI, even modest accuracy improvements can significantly reduce diagnostic errors when scaled across thousands of cases.\n\n**Practical Utility as a Diagnostics tool**:\n\nWe acknowledge that the current absolute performance is still not enough for full clinical application. However, we believe RAD offers meaningful advancement in its current form:\n\n1. The RAD model is **sufficient to work as a medical diagnostic tool on common diseases**:\n\n    Diseases|AUC|mAP|Acc|F1|Precision|Recall|Avg\n    -|-|-|-|-|-|-|-\n    Heart disease, cardiomegaly|98.68|89.61|82.5|75.5|90.94|95.69|88.82\n    Pulmonary collapse|96.38|84.44|79.38|77.85|80.98|93.56|85.43\n    Lung opacity|98.12|83.28|76.66|66.4|90.67|94.64|84.96\n\n    The above presents RAD's performance on some common chest diseases. The model already achieves performance comparable to senior clinicians, making it suitable as a diagnostic tool for these diseases.\n\n2. **Interpretability**: Unlike other baselines, RAD can perform evidence-based diagnosis by following clinical guidelines. RAD's knowledge-grounded approach provides transparent diagnosis process that clinicians can evaluate and trust - a critical factor for real-world adoption that pure performance metrics don't capture.\n\n3. **Adaptability**: As shown, the framework can incorporate new knowledge for zero-shot inference, surpassing existing baselines. Furthermore, as a small discriminative model, RAD can achieve strong performance on new diseases at the lowest cost by simple retraining.\n\nThese discussions can better elucidate RAD's real-world practicality, and we will include them in our revised manuscript as a ''Discussion'' section. We sincerely appreciate your insightful questions and suggestions!\n\n>Question 1: It would be helpful to understand which of the four sources used for knowledge retrieval are the most relevant to the performance of the model. For example, would just the clinical practice guidelines be sufficient?\n\nTo address this, we conducted ablation studies where we used only one knowledge source at a time:\n\nSource|F1|Precision|Recall|AUC|mAP|Acc|Acc-S|Avg\n-|-|-|-|-|-|-|-|-\nWiki|39.77|39.11|47.05|93.14|36.93|96.02|41.67|56.24\nResearch|38.54| 36.35| 51.41|93.01|36.26|95.47|40.69|55.96\nGuideline|39.79| 39.17|50.32|93.03|37.12|96.02|41.42|56.70\nBook|39.49|39.14|47.65|93.11|36.84|96.20|42.04|56.35\nAll|39.71|39.07|54.74|93.00|36.74|95.40|42.33|57.28\n\nResults on MIMIC-ICD53 demonstrate:\n1. **Clinical guidelines provide the most valuable knowledge** (Avg: 56.70), as they directly encode established diagnostic criteria, key indicators, and decision pathways specifically designed for clinical practice. This source aligns most with the gold standard for diagnosing specific diseases.\n2. **Research papers show the lowest contribution** (Avg: 55.96), as they often focus on novel discoveries, experimental treatments, or specialized cases rather than established diagnostic standards. For well-established diseases, diagnostic criteria have become consensus, making cutting-edge research less useful.\n3. **Multiple knowledge sources are complementary**: While guidelines alone already provide near-optimal results, the complete knowledge base (4 sources) can further boost the performance.\n\nFor resource/knowledge-constrained settings, clinical guidelines alone are nearly sufficient for maintaining high accuracy. However, for best practice, the full knowledge base is recommended.\n\nThis important ablation will be included in Section 4.4 of our revised manuscript.\n\n>Question 2: Do you think the results can be improved if a more balanced dataset is used? Or can some data augmentation be used to enrich the parts of the dataset where only a few samples are present?\n\nYes, we agree that class imbalance greatly affects performance on tail classes. If a more balanced dataset and data augmentation methods are used, the performance should be improved. In the following, we have conducted experiments as suggested:\n- Resample: Oversampling 10 tail classes with a sampling ratio of 5\n- Resample+Mixup: Further applying a popular data augmentation method Mixup[1] on tail classes based on the resampled dataset\n\nAll strategies are applied on the training set, and we use mixup with α=0.2 (suggested in the paper). The performance on tail classes is reported:\n\nDataset|F1|Precision|Recall|AUC|mAP|Avg\n-|-|-|-|-|-|-\nOriginal|5.53|4.07|45.42|91.41|1.32|29.55\nResample|9.18|10.33|35.85|91.31|3.20|29.97\nResample+Mixup|9.98|14.06|32.97|91.27|3.71|30.40\n\nAlthough the average performance shows little increase, F1, Precision, and mAP exhibit significant improvement. After resampling, mAP has increased by more than 100% (1.32 to 3.2). Mixup augmentation provided additional gains beyond simple resampling, particularly in precision.\n\nSince RAD is **orthogonal** to data-balancing methods, these methods can be easily integrated into RAD to further improve the performance on tail classes.\n\nAdditionally, we would like to clarify **why choosing an imbalanced dataset**. In healthcare settings, disease distributions naturally follow long-tail patterns (e.g., pneumonia is much more common than farmers' lung/pneumoconiosis). Thus, we intentionally preserved the original distribution of MIMIC to align with real-world medical settings.\n\nThe tail classes resampled here are: 'acute bronchiolitis and unspecified acute lower respiratory infection', 'farmers' lung and pneumoconiosis due to asbestos and mineral fibers', 'thalassemia', 'acute and subacute endocarditis', 'phlebitis and thrombophlebitis', 'flu due to avian flu virus', 'arterial embolism and thrombosis', 'acute and subacute bacterial endocarditis', 'polyarteritis nodosa', 'abscess of lung and mediastinum, pyothorax, mediastinitis'.\n\n[1]mixup: Beyond empirical risk minimization (ICLR 2018)"}]

Author rebuttal:

Meta-review: This paper proposes Retrieval-Augmented Diagnosis (RAD), a framework that integrates disease-specific clinical knowledge into multimodal diagnostic models through guideline retrieval and refinement, a guideline-enhanced contrastive loss, and a dual-decoder design. The strengths of the paper include a clear motivation rooted in trustworthy medical AI, systematic knowledge injection across the diagnostic pipeline, extensive evaluations on four diverse datasets with consistent improvements over strong baselines and multimodal LLMs, and thorough ablations clarifying the roles of different components. The rebuttal further strengthened the work by demonstrating zero-shot capability, robustness to noisy retrieval, ablations on knowledge sources (with guidelines shown to be most critical), fairness of baseline comparisons, and causal dependence on guideline features. 

Remaining weaknesses include limited novelty in retrieval strategy, dependence on textual guidelines, potential resource constraints, controversial reliance on attention-based interpretability, dataset imbalance, and absence of prospective clinical validation. Nonetheless, the reviewers converged on acceptance after rebuttal, agreeing that the methodological contribution, empirical validation, and interpretability advances justify inclusion at NeurIPS.
Decision: Accept (poster)