Title: RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Fusion

Medical Imaging

Interpretability

Disease Diagnosis
Abstract: Clinical diagnosis is a highly specialized discipline requiring both domain expertise and strict adherence to rigorous guidelines. 
While current AI-driven medical research predominantly focuses on knowledge graphs or natural text pretraining paradigms to incorporate medical knowledge, these approaches primarily rely on implicitly encoded knowledge within model parameters, neglecting task-specific knowledge required by diverse downstream tasks.
To address this limitation, we propose **R**etrieval-**A**ugmented **D**iagnosis (RAD), a novel framework that explicitly injects external knowledge into multimodal models directly on downstream tasks.
Specifically, RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guideline-enhanced contrastive loss that constrains the latent distance between multi-modal features and guideline knowledge, and the dual transformer decoder that employs guidelines as queries to steer cross-modal fusion, aligning the models with clinical diagnostic workflows from guideline acquisition to feature extraction and decision-making.
Moreover, recognizing the lack of quantitative evaluation of interpretability for multimodal diagnostic models, we introduce a set of criteria to assess the interpretability from both image and text perspectives. Extensive evaluations across four datasets with different anatomies demonstrate RAD's generalizability, achieving state-of-the-art performance. Furthermore, RAD enables the model to concentrate more precisely on abnormal regions and critical indicators, ensuring evidence-based, trustworthy diagnosis. Our code is available at https://github.com/tdlhl/RAD.

RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis

Abstract

Clinical diagnosis is a highly specialized discipline requiring both domain expertise and strict adherence to rigorous guidelines. While current AI-driven medical research predominantly focuses on knowledge graphs or natural text pretraining paradigms to incorporate medical knowledge, these approaches primarily rely on implicitly encoded knowledge within model parameters, neglecting task-specific knowledge required by diverse downstream tasks. To address this limitation, we propose R etrievalA ugmented D iagnosis (RAD), a novel framework that explicitly injects external knowledge into multimodal models directly on downstream tasks. Specifically, RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guidelineenhanced contrastive loss that constrains the latent distance between multi-modal features and guideline knowledge, and the dual transformer decoder that employs guidelines as queries to steer cross-modal fusion, aligning the models with clinical diagnostic workflows from guideline acquisition to feature extraction and decision-making. Moreover, recognizing the lack of quantitative evaluation of interpretability for multimodal diagnostic models, we introduce a set of criteria to assess the interpretability from both image and text perspectives. Extensive evaluations across four datasets with different anatomies demonstrate RAD's generalizability, achieving state-of-the-art performance. Furthermore, RAD enables the model to concentrate more precisely on abnormal regions and critical indicators, ensuring evidence-based, trustworthy diagnosis. Our code is available at this repository.

1 Introduction

The rapid development of multimodal learning [37, 46] has revolutionized numerous fields by enabling models to process and integrate diverse data types, including images, texts, audio, and structured records [4, 11, 77]. Biomedical applications particularly benefit from these advancements, given that diagnostic workflows inherently depend on multimodal evidence, ranging from radiographic imaging and reports to electronic health records (EHR) [16, 52, 75]. For instance, radiologists integrate X-ray or MRI scans with textual pathology reports, while clinicians combine electronic health records, vital signs, and even genomic data to form comprehensive patient profiles. Accordingly, recent research efforts have increasingly focused on developing multimodal architectures tailored to healthcare challenges, seeking to enhance diagnostic precision through cross-modal synergy [6, 32, 61, 68]. While these approaches demonstrate significant progress in integrating data from different modalities, they often overlook the foundational principles governing clinical decision-making.

(a)

(b)

Data

Encode & Align

Decision

RAD:

Single-view Injection

Input

Augment

Pretrain

Encoder

Parameter

Prior

Knowledge

Guideline:

…

Key indicators:

1. Alanine

Aminotransferase;

2. Aspartate

Aminotransferase;

3. Serum

4. Platelet Count

…

[CLS] < report > : findings : as compared to the

previous radiograph , there is no relevant change .

elevation of the right hemidiaphragm with bilateral

relatively extensive parenchymal opacities. no

pleural effusions . no pneumothorax . < ehr > : the

patient is a 64 - year - old female . the lab results

include alanine aminotransferase ( alt ) with a value

of 5 . 0 , alkaline phosphatase with a value of 5 . 0,

0 , basophils with a value of 4 . 0, bilirubin , total

with a value of 5 . 0, calculated total co2 with a

value of 4 . 0, hematocrit with a value of 1 . 0,

lactate with a value of 4 . 0 , lactate dehydrogenase

( ld ) with a value of 6 . 0, phosphate with a value

of 7 . 0 , platelet count with a value of 4 . 0 ,

potassium with a value of 5 . 0 [SEP]

[CLS] < report > : findings : as compared to the

previous radiograph , there is no relevant change .

elevation of the right hemidiaphragm with bilateral

relatively extensive parenchymal opacities. no

pleural effusions . no pneumothorax . < ehr > : the

patient is a 64 - year - old female . the lab results

include alanine aminotransferase ( alt ) with a value

of 5 . 0 , alkaline phosphatase with a value of 5 . 0,

0 , basophils with a value of 4 . 0, bilirubin , total

with a value of 5 . 0, calculated total co2 with a

value of 4 . 0, hematocrit with a value of 1 . 0,

lactate with a value of 4 . 0 , lactate dehydrogenase

( ld ) with a value of 6 . 0, phosphate with a value

of 7 . 0 , platelet count with a value of 4 . 0 ,

potassium with a value of 5 . 0 [SEP]

w/o RAD

RAD

Medical analysis fundamentally differs from natural scene understanding in its strict adherence to evidence-based principles, relying heavily on structured protocols [31, 50]. Clinical decisions must be grounded in standardized diagnostic criteria derived from patient-specific symptoms, imaging findings, and laboratory results. This inherent rigor poses a critical challenge for black-box neural networks, whose vague decision-making mechanisms hinder trustworthy and practical deployment in clinical settings [19, 49, 53]. Consequently, there has been growing interest in integrating medical knowledge into AI models to simultaneously improve model performance and interpretability [8, 9, 73].

Existing approaches primarily focus on knowledge injection during pretraining phases. Researchers enhance the text encoders by pretraining them on large-scale medical corpora [30, 47] or leveraging structured knowledge graphs to imbue models with semantic relationships between biomedical entities [36, 48]. While effective in expanding the semantic coverage of text encoders, these approaches often struggle to explicitly integrate fine-grained knowledge tailored for downstream diagnostic tasks. To this end, we argue that effective knowledge integration requires task-centric, holistic alignment with disease-level knowledge throughout the entire diagnostic pipeline. As illustrated in Figure 1(a), our framework systematically integrates refined knowledge to guide input augmentation, feature extraction, and modality fusion, contrasting with prior methods confined to a single perspective. Figure 1(b) presents a case of the model's attention distribution over the input text. The previous model fails to concentrate on critical indicators, but focuses on obvious disease terms in the reports. While the RAD model can not only attend to these terms but also consider other guideline-recommended key indicators. The explicit knowledge guidance enables RAD to prioritize critical indicators tailored for the current disease, making trustworthy diagnoses aligned with clinical standards.

In this paper, we propose a holistic knowledge-injection framework RAD, which operates through three synergistic components spanning the entire diagnostic workflow. RAD begins by retrieving and refining disease-specific guidelines from diverse sources, flexibly adapting to downstream tasks in different scenarios. We then employ two modality encoders coupled with guideline-enhanced contrastive loss that explicitly aligns the modality-specific feature with the corresponding diseaseguideline prototypes in the joint latent space. A dual decoder network is further developed to steer the cross-modal fusion process, which simultaneously incorporates disease labels and their corresponding guidelines to interact with fused multimodal features for final predictions. Through this systematic knowledge infusion paradigm, our framework achieves performance gains while establishing a traceable decision pathway grounded in clinical guidelines-a critical step toward clinically actionable AI. We further establish an evaluation system for model interpretability, which quantitatively assesses the model's adherence to guidelines through both textual indicators and visual localization. Combined with qualitative visualization, this system provides measurable evidence that RAD's decisions are driven by the injected knowledge. In summary, our contributions are three-fold:

We propose RAD to systematically inject external medical knowledge into multimodal diagnosis models. RAD incorporates a guideline-enhanced loss and a dual-decoder structure to explicitly steer multimodal feature extraction and cross-modal fusion with disease-guideline prototypes.

A dual-axis evaluation system for the interpretability of diagnosis models is developed, formulating both textual and visual metrics. This system enables quantitative analyses of the model's adherence to clinical guidelines, demonstrating the transparency and explainability brought by RAD.

Wealigned MIMIC-CXR [28] and MIMIC-IV [29] to construct the MIMIC-ICD53 dataset, covering three modalities with 53 types of disease. Extensive experiments on our dataset and three other public datasets demonstrate the superiority of RAD over SOTA baselines across various metrics.

2 Related Work

2.1 Multimodal Learning in Medicine

Recent years have witnessed significant advancements in the field of multimodal learning, with models such as CLIP [46], BLIP [33], and LLaVA [37] exhibiting remarkable capabilities in natural domains. These developments have spurred increasing interest in extending multimodal frameworks to the medical field, where the integration of diverse data modalities demonstrates prominent potential in diagnostic tasks. Current research focus lies in multimodal pretraining methods, which focus on cross-modal alignment between imaging and textual data to improve the representation transferability [7, 17]. ConVIRT [72] and GLoRIA [26] pioneered the application of CLIP-style architectures in the medical domain by constructing image-text pairs from radiology datasets. MedCLIP [56] and BiomedCLIP [70] addressed the scarcity of paired medical image-text data by leveraging multisource datasets, achieving state-of-the-art performance. Beyond pretraining methods, multimodal fusion approaches aim at integrating information from different modalities for diagnostic applications [20, 57, 66]. MedFuse [21] introduced an LSTM-based temporal fusion method of time-series data and X-ray images. HEALNet [23] proposed a hybrid early-fusion method to learn from data sources with different structures. While these works have made significant strides, they often operate without explicit guidance from medical knowledge when addressing specific diagnosis tasks. In contrast, considering the evidence-based nature of medicine [44, 51], RAD explicitly incorporates task-specific knowledge to guide both representation extraction and multimodal fusion processes.

2.2 Medical Knowledge Injection

Injecting professional knowledge into AI models is a prevalent strategy to improve their domainspecific capabilities [40, 65]. Various techniques have been investigated to incorporate medical knowledge into the models. Pretraining-based approaches train the text encoder on extensive medical domain corpora, such as PubMedBERT [18] and HUATUO-GPT [69]. Other methods like KAD [71] and DRAGON [67] leverage structured knowledge graphs of medical entities for pre-training to enhance the text encoder's comprehension of medical terminology. While showing empirical effectiveness, these knowledge integration methods remain primarily confined to the pretraining phase, providing only implicit guidance during the subsequent diagnostic stage. With the rapid development of large language models (LLMs) [1, 35, 54], various Retrieval-Augmented Generation (RAG) methods have been proposed to enhance the generation process of medical LLMs [13, 61]. These methods dynamically retrieve external medical knowledge to improve the performance of LLMs on question-answering (QA) tasks [34, 60]. Building upon this foundation, multimodal RAG approaches further retrieve similar data samples (e.g., image-report pairs) for visual question-answering (VQA) [59, 74]. In contrast to RAG methods that online retrieve knowledge to augment input for generative QA/VQA tasks, our framework adopts a structured approach for discriminative tasks by performing offline retrieval of disease-specific knowledge, which is systematically incorporated to guide model training.

3 Method

In this section, we first present the problem formulation, followed by the detailed introduction of our proposed method, Retrieval-Augmented Diagnosis (RAD), which consists of guideline retrieval and refinement, guideline-enhanced feature constraint, and dual diagnostic network. Finally, we introduce our interpretability evaluation system. The overall framework of RAD is illustrated in Figure 2.

Label

Image

…

Corresponding

As compared to the previous radiograph,

the known left lower lobe, pneumonia has

minimally increased. ……

The patient is a 35-year-old male. His lab

results include Lymphocytes with a value of

5

，

blood sugar with a value of

9

……

Report Text & Normalized EHR Value

Wiki

Guideline Retrieval & Refinement

Research

Guidelines

Books

Summarize

Image Encoder

Text Encoder

Multi-Source

Knowledge

Corpus

Label

Query

Brain MRI

Dermatology

Chest X-Ray

Fundus

Multi-Head Attention

Add & Norm

Multi-Head Attention

Add & Norm

Feed Forward

Add

Guideline Decoder

Query

Key & Value

Multi-Head Attention

Add & Norm

Multi-Head Attention

Add & Norm

Feed Forward

Add

Query

Key & Value

Guideline

ℒGECL

ℒGECL

ℒBCE

ො 𝑦

Label

ො 𝑦

Guide

Label

Visual Embedding

Text Embedding

7

GUIDE

3.1 Problem Formulation

Given a training set of N samples, D = { ( x i , t i , y i ) } N i =1 , where x i represents a radiology image, t i is the report and electronic health records, and y i ∈ { 0 , 1 } m is the corresponding multi-label vector indicating the presence of m diseases. The multi-source medical knowledge corpus is denoted as P = { p i | i = 1 , 2 , . . . , s } , where p i denotes the i -th source and s denotes the number of sources. The guideline corresponding to the disease label is derived from multi-source retrieval and refinement. We denote this guideline as g = { g i | i = 1 , 2 , . . . , m } . The objective is to develop a multimodal model trained on D , capable of accurately predicting the disease for any given multimodal sample.

3.2 Retrieval-Augmented Diagnosis

3.2.1 Guideline Retrieval and Refinement

Knowledge-corpus. To retrieve disease-related diagnostic knowledge, we collect medical knowledge from four distinct sources: 'Wiki', 'Research', 'Guideline', and 'Book'. Wiki provides comprehensive descriptions of target diseases, such as formal medical definitions, and clinically relevant subcategories. Research incorporates the latest research articles from PubMed (a premier database of biomedical literature). These articles provide cutting-edge findings in disease mechanisms, diagnostic criteria, and therapeutic interventions. Guideline includes 45K clinical practice guidelines from 13 sources, providing rigorously vetted diagnostic criteria and treatment protocols for medical practitioners. Book consists of diverse medical textbooks, covering basic medical knowledge in surgery, medical imaging, and drugs, etc. More details of the corpus can be found in Appendix B.1.

Disease Knowledge Retrieval. For a given dataset with m diseases, our objective is to retrieve the most relevant knowledge from the knowledge corpus P , including but not limited to: associated symptoms, imaging characteristics, and critical examination/laboratory indicators. We adopt MedCPT [27], a dual-encoder model optimized for medical scenarios, as the retriever. Specifically, the article encoder R A ( · ) is utilized to convert the corpus P into dense vectors for retrieval. The disease names E are used as the input query of the query encoder R Q ( · ) . The obtained embeddings are then used to calculate the similarity as Sim( E,P ) = R Q ( E ) ⊤ R A ( P ) . For each disease with a name e i ∈ E , we preserve the topk retrieved documents as:

LLMRefinement. Given that retrieved documents C i may contain content irrelevant to the diagnosis of the current disease and exhibit cross-source redundancy, directly combining the retrieved documents as the final guideline is suboptimal. In addition, the total document length often exceeds

the context window of the diagnosis model. To address these challenges, we employ large language models (LLMs) to perform automated summarization and refinement of C i . The final refined guideline g i of disease e i can be obtained by:

where c i,j ∈ C i is the j -th document. This process yields standardized, well-structured diagnostic guidelines that preserve critical clinical information while eliminating noise and redundancy. In practice, we choose Qwen2.5-72B [64] as the LLM. Examples of the guideline and prompt templates are provided in Appendix B.2.

3.2.2 Guideline-enhanced Feature Constraint

For multimodal downstream tasks, our framework utilizes two modality-specific encoders to separately learn visual and textual representations. The refined guideline g obtained in Section 3.2.1 is employed here as the feature constraint of both textual and visual representation.

Given a sample ( x i , t i , y i ) , we use the vision encoder denoted as Φ img ( · ) to extract the visual embeddings V i from x i . The text encoder Φ text ( · ) is employed to obtain the textual embeddings T i . Meanwhile, the refined guideline g is also encoded by the text encoder for subsequent feature alignment. The encoding process is summarized as follows:

where h, w are the height, width of the image, l is the max token length of the text encoder, m is the number of disease types, and d is the embedding dimension. These embeddings with spatial information are then used as the input of the dual decoder in Section 3.2.3 for multimodal fusion and final diagnosis. Here, we perform pooling on the extracted embeddings and use the pooled features for subsequent feature alignment. Specifically, we apply adaptive pooling operations to get the visual feature V ′ i ∈ R d , and directly use the embedding of the [CLS] token as the textual feature T ′ i ∈ R d . The corresponding pooled disease-guideline prototypes are G ′ = { G ′ i ∈ R d | i = 1 , 2 , . . . , m } .

To align the extracted features with diagnostic criteria, we propose a guideline-enhanced multi-modal feature constraint strategy. Specifically, disease-guideline prototypes are utilized as an anchor to pull both image and text features closer to them. To achieve this, we introduce a Guideline-Enhanced Contrastive Loss (GECL) for feature extraction under the guideline constraint. For sample i with the disease label y i , the guideline features G ′ are split into P i and N i , where P i = { G ′ j ∈ G ′ | y ij = 1 } is the set of guideline features corresponding to positive disease labels, N i is the set of guideline features with negative disease labels. To avoid using excessive negative samples, we sample a subset Q i from N i that satisfies | Q i | = min ( r | P i | , | N i | ) , where r is the negative sampling ratio. The final guideline feature set is S i = P i ∪ Q i . Then, we can formalize GECL as a cross-entropy-based supervised contrastive learning objective:

where I [ · ] is the indicator function. ϕ ( I i , S ij ) = I ⊤ i S ij /τ is the similarity score between the modality-specific feature and the guideline feature, τ is the temperature hyperparameter, and α is the trade-off hyperparameter. Note that the similarity score ϕ can be converted into a probability via the Sigmoid function. As shown in Eq. (5), L GECL aligns image features V ′ i and text features T ′ i with disease guideline prototypes, which are the diagnostic criteria of each disease defined by the embedding of its guideline. Dynamically aligning sample features with their positive prototypes prevents representation collapse while enhancing model robustness in multi-label scenarios. Furthermore, this approach induces the model to selectively focus on clinically relevant features that match the guidelines, improving the model performance and interpretability simultaneously. For detailed derivation from the standard cross-entropy form to Eq. (4), please refer to Appendix B.3.

3.2.3 Dual Diagnostic Network

Under the guideline constraint, we obtained enhanced visual and textual features. To achieve the final disease diagnosis, we develop a transformer-based cross-modal information fusion module, which has a dual decoder architecture. In the first decoder, the guideline g is employed as the query, and the concatenated modality embeddings V i ⊕ T i are used as the key and value. After forward through the fusion structure Φ g D , we obtain the logits corresponding to each disease:

To further enhance the performance, we symmetrically utilize the second similar structure Φ l D where the query is replaced with the disease names, while keeping the key and value unchanged. This symmetric operation gets ˆ y label i = Φ l D (Φ text ( E ) , V i ⊕ T i , V i ⊕ T i ) ∈ R m , ensuring comprehensive feature integration. Finally, we compute the binary-cross-entropy loss on both logits with the ground truth. Thus, the total training loss of RAD is:

where β represents the trade-off hyperparameter between L BCE and L GECL.

3.3 Interpretability Evaluation System

To validate the evidence-based diagnosis of RAD, we introduce a dual-axis interpretability evaluation system that quantitatively measures the model's adherence to injected guidelines through both textual and visual metrics. Formal definitions of the metrics for each input modality are presented below.

3.3.1 Textual Recall of Indicators

The Guideline Recall is designed to quantify the model's explicit compliance with diseasespecific diagnostic standards. The refined guideline of each disease contains a set of key laboratory indicators that are considered valuable for diagnosing this disease. The extent to which a model attends to these indicators can reflect its adherence to the guideline. Formally, when the input text contains indicators mentioned in the guideline, we assess the model's attention to these indicators by aggregating the attention weights of the corresponding tokens (derived from the cross-attention maps in the transformer

Algorithm 1 Guideline Recall

1: Input: Guideline G , text token sequence T , at- tention weights A , threshold θ

2:

U ←

Extract indicators from

G

3:

attended

= 0

,

total

= 0

4:

for

each

u

∈ U

do

5:

Matched

Tokens in

T

matching

u

6:

if

Matched

=

∅

then

7:

total

=

total

+1

8:

if

mean

(

A

Matched

)

> θ

then

9:

attended

=

attended

+1

10:

return

attended/total

if

total >

0

else

0

̸

decoders). When the aggregated attention weights exceed a predefined threshold θ , this provides quantitative evidence that the model exhibits statistically significant attention to the corresponding indicator. The detailed computation process is outlined in Algorithm 1.

3.3.2 Visual Attention Grounding Ability

For visual explainability, an attention-derived localization metric is employed to measure the alignment between model-attended regions and pathological abnormalities. Given expert-annotated bounding boxes for lesions, we compute the overlap between top-activated regions in the attention map and these ground truths. Specifically, we use the Intersection over Union IoU = | A ∩ B | | A ∪ B | as the metric, where A is the model localization derived from the attention map and B is the ground truth.

These two metrics formally establish a dual-modality interpretability evaluation system. Through the systematic analysis of how the injected knowledge explicitly intervenes in the model's decisionmaking, this system provides a quantitative evaluation for explainable multimodal medical AI.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate RAD on four multimodal medical datasets with different anatomies, including MIMIC-ICD53, Harvard-FairVLMed [41], SkinCAP [76], and NACC [5].

MIMIC-ICD53 is constructed through the alignment and integration of MIMIC-CXR [28] and MIMIC-IV [29], comprising chest X-ray images, corresponding reports, and EHRs, annotated with 53 diseases under the ICD [43] standard. For laboratory indicators in the EHR, we quantified the numerical results on a scale of 1 to 10 based on the upper and lower limits of their normal range. We will release the dataset on PhysioNet [42]. Details of dataset construction are provided in Appendix C.1.1. Harvard-FairVLMed, SkinCAP, and NACC are multimodal datasets focusing on eyes, skin, and brain, respectively. All patient data has been de-identified. More detailed statistics of datasets are presented in Table 2.

Baselines. We select representative baseline methods in the medical field, including large-scale pre-training model BiomedCLIP [70], knowledge-enhanced pre-training method KAD [71], and state-of-the-art multimodal fusion methods MedFuse [21], DrFuse [66], and HEALNet [23].

Evaluation Metrics. For the evaluation of model performance, we adopt widely used multi-label classification metrics including F1, Precision, Recall, AUC, mAP, and ACC. All metrics are the average of multiple labels. Since standard accuracy (ACC) aggregates predictions across all labels and thus may not adequately reflect the correctness for individual patients, we include an additional metric named sample-wise ACC (ACC-S). This metric considers a prediction correct only if all labels of a patient are accurately classified, making it more aligned with clinical scenarios.

Implementation Details. In practice, Topk in Eq.(1) is set to 10. All guidelines obtained by Eq.(2) and the indicators used in Algorithm 1 are manually verified to avoid potential factual errors. The default backbone of the text image encoder is ClinicalBERT [55] and ResNet-50 [22], respectively. The hyperparameters α and β , which serve as the balancing ratio between different losses, are set to be 1 e -2 and 1 e -1 , respectively. All experiments are conducted on a single NVIDIA A100 GPU.

4.2 Diagnosis Performance

As demonstrated in Table 1, our method consistently achieves superior performance across four benchmarks of diverse anatomies. Specifically, RAD outperforms the second-best method with average improvements of 3.09%, 2.24%, 2.49%, and 2.45% on MIMIC-ICD53, FairVLMed, SkinCAP, and NACC datasets, respectively. The most substantial gains occur in MIMIC-ICD53, where RAD improves both precision and recall over 5%, suggesting strong robustness in handling complex, real-world clinical label distributions. This improvement is particularly noteworthy given the dataset's challenging nature, containing both fine-grained ICD labels and noisy clinical documentation. Notably, the sample-wise accuracy (ACC-S) of all methods exhibits a significant degradation compared to macro-average accuracy (ACC), especially in datasets with extensive label spaces. This discrepancy highlights fundamental limitations in current models' capacity to handle multi-label problems, exposing challenges for real-world clinical deployment. Intriguingly, KAD, which injects medical knowledge during the pretraining phase, achieves strong performance on MIMIC-ICD53 but falls short on others. This is likely because its pretraining data concentrated on chest X-rays, limiting its ability to generalize to other anatomical regions. In contrast, our approach directly injects knowledge on downstream tasks, offering greater adaptability across distinct regions and modalities. These consistent improvements across diverse anatomies, data scales, and label complexities validate the versatility and scalability of RAD. Full baseline results with variance are in Appendix C.3.

4.3 Interpretability Evaluation

4.3.1 Interpretability from Textual Perspective

To quantitatively assess the impact of knowledge injection from the textual perspective, we calculate the guideline recall defined in Section 3.3.1. As presented in Table 4, incorporating knowledge via RAD prominently increases the recall value from 24.76% to 65.62%. This indicates that RAD indeed injects guideline-derived knowledge into the model, thereby enhancing its focus on key information mentioned in the guideline. Notably, the conventional model exhibits extremely low recall (<10%) on Alanine Aminotransferase (ALT) and Aspartate Aminotransferase (AST). This may stem from their inability to understand highly specialized, rare medical terms. In contrast, RAD explicitly highlights the importance of these indicators in the guideline, leading to significant recall improvement. This finding underscores the necessity of flexible knowledge adaptation for downstream tasks rather than static pretraining paradigms. Overall, the enhanced guideline recall demonstrates that RAD enables the model to make reliable evidence-based diagnoses according to guidelines. This improvement also aligns with the qualitative attention patterns observed in Figure 1. To further substantiate the interpretability of RAD, we provide more and clearer visualization cases in Appendix C.5.

4.3.2 Interpretability from Visual Perspective

w/o RAD

RAD

Disease

Mass

Fibrosis

Atelectasis

Consolidation

Symmetrically, we conduct zero-shot grounding experiments on the ChestXDet dataset [38]. The results shown in Table 3 demonstrate a significant improvement in mIoU scores for lesion detection after the injection of refined guidelines. Besides, Figure 3 illustrates multiple cases of lesion grounding. For clearer visualization, we overlay spectrum heatmaps on the original CXR images, together with the ground truth bounding box high- lighted in red. A comparison between the lesions identified by the model and the bounding boxes marked by clinical experts reveals a notable improvement in alignment when our guidelines are applied. This indicates that the model's focus is more accurately directed toward clinically significant lesions, emphasizing RAD's enhanced diagnosis capabilities and interpretability under the guidance of external knowledge.

4.4 Ablation Study

In this subsection, we conduct ablation studies on each component of RAD and validate its generalizability across different model architectures. All experiments are conducted on MIMIC-ICD53.

Ablation on Each Component. As shown in Table 5, we evaluate the efficacy of each newly proposed component in RAD. It is evident that removing either the L GECL or the dual decoder negatively impacts model performance, highlighting the importance of the guideline in both representation learning and multimodal fusion. Notably, the removal of the Dual Decoder results in the most substantial performance degradation, underscoring the necessity of leveraging guidelines to intervene in the final decision-making process. We further compared the performance of the textual and visual branches of L GECL when used individually. The results show that the textual branch yields more significant improvements. This can be attributed to the fact that both the input text and the guideline belong to the same modality, allowing for more effective alignment.

Ablation on Different Backbones. To demonstrate the robustness and flexibility of our method, we verify RAD on different encoder backbone combinations. Specifically, we iteratively replaced the default visual encoder and text encoder with two other popular architectures, ViT [15] and

Customized Radar Chart

Avg

F1

Precision

Recall

Acc

Avg

F1

Precision

Recall

Acc

RAD

Lingshu-7B

HuatuoGPT-Vision-7B

Qwen2.5-VL-7B

Few-shot

SFT

BioClinicalBERT [2]. As illustrated in Figure 4, RAD consistently offers substantial performance gain across all four combinations of backbones. This not only highlights the insensitivity of our approach to different backbone architectures but also underscores its robustness and generalizability. Specifically, ResNet and ViT exhibit comparable performance gains, while ClinicalBERT shows more pronounced improvement than BioClinicalBERT. Overall, RAD exhibits the robust ability to generalize to diverse data and model structures, ensuring reliable performance in various scenarios. Detailed results, more ablation studies, and hyperparameter analysis are presented in Appendix C.6.

4.5 Discussion

Comparison with Multimodal Large Language Models Multimodal large language models exhibit remarkable capabilities in visual content understanding and generalization. To further validate the effectiveness of RAD, we compare with state-of-the-art MLLMs, including Qwen2.5-VL-7B [3], HuatuoGPT-Vision-7B [10], and Lingshu-7B [62]. As presented in Figure 5, our discriminative framework achieves superior performance with significantly lower computational cost. These results demonstrate that complex diagnostic tasks are better suited for specialized discriminative models than generative MLLMs. The significant performance gap, observed on the simplest single-label dataset, underscores the practical advantages of our approach in clinical applications where both accuracy and efficiency are critical.

5 Conclusion

This paper proposes RAD, which enhances the capabilities of multimodal diagnosis models by leveraging external medical knowledge. RAD operates via a tri-fold methodology, consisting of offline retrieval and refinement of disease-centered external guidelines, multimodal feature alignment under the guideline constraint, and the dual diagnostic network. Extensive experiments on four datasets of different anatomies demonstrate the effectiveness of RAD. Furthermore, RAD exhibits dual-axis interpretability by simultaneously achieving precise lesion localization in imaging data and prioritizing guideline-concordant indicators in textual analysis. This evidence-based explainability enhances clinical trustworthiness, offering the potential to inspire future research in medical AI.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323 , 2019.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence , 41(2):423-443, 2018.

Duane L Beekly, Erin M Ramos, William W Lee, Woodrow D Deitrich, Mary E Jacka, Joylee Wu, Janene L Hubbard, Thomas D Koepsell, John C Morris, Walter A Kukull, et al. The national alzheimer's coordinating center (nacc) database: the uniform data set. Alzheimer Disease & Associated Disorders , 21(3):249-258, 2007.

Edgar A Bernal, Xitong Yang, Qun Li, Jayant Kumar, Sriganesh Madhvanath, Palghat Ramesh, and Raja Bala. Deep temporal multimodal fusion for medical procedure monitoring using wearable sensors. IEEE Transactions on Multimedia , 20(1):107-118, 2017.

Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision-language processing. In European conference on computer vision , pages 1-21. Springer, 2022.

Cheng Chen, Qi Dou, Yueming Jin, Quande Liu, and Pheng Ann Heng. Learning with privileged multimodal knowledge for unimodal segmentation. IEEE transactions on medical imaging , 41(3):621-632, 2021.

Junying Chen, Chi Gui, Anningzhe Gao, Ke Ji, Xidong Wang, Xiang Wan, and Benyou Wang. Cod, towards an interpretable medical agent using chain of diagnosis. arXiv preprint arXiv:2407.13301 , 2024.

Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406.19280 , 2024.

Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems , 36:72842-72866, 2023.

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079 , 2023.

Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, Yiqiu Guo, Yanfeng Wang, and Yu Wang. Towards omni-rag: Comprehensive retrieval-augmented generation for large language models in medical applications. arXiv preprint arXiv:2501.02460 , 2025.

Tianjie Dai, Ruipeng Zhang, Feng Hong, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Unichest: Conquer-and-divide pre-training for multi-source chest x-ray classification. IEEE Transactions on Medical Imaging , 2024.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Qi Dou, Quande Liu, Pheng Ann Heng, and Ben Glocker. Unpaired multi-modal segmentation via knowledge distillation. IEEE transactions on medical imaging , 39(7):2415-2425, 2020.

Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023 , pages 1181-1193, 2023.

Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) , 3(1):1-23, 2021.

Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, Feng Liu, et al. Trustworthy machine learning: From data to models. Foundations and Trends® in Privacy and Security , 7(2-3):74-246, 2025.

Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data , 6(1):96, 2019.

Nasir Hayat, Krzysztof J Geras, and Farah E Shamout. Medfuse: Multi-modal fusion with clinical timeseries data and chest x-ray images. In Machine Learning for Healthcare Conference , pages 479-503. PMLR, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Healnet: Multimodal fusion for heterogeneous biomedical data. Advances in Neural Information Processing Systems , 37:64479-64498, 2024.

Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature , 637(8045):319-326, 2025.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR , 1(2):3, 2022.

Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal globallocal representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3942-3951, 2021.

Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics , 39(11):btad651, 2023.

Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data , 6(1):317, 2019.

Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data , 10(1):1, 2023.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics , 36(4):1234-1240, 2020.

Karim Lekadir, Richard Osuala, Catherine Gallin, Noussair Lazrak, Kaisar Kushibar, Gianna Tsakou, Susanna Aussó, Leonor Cerdá Alberich, Kostas Marias, Manolis Tsiknakis, et al. Future-ai: guiding principles and consensus recommendations for trustworthy artificial intelligence in medical imaging. arXiv preprint arXiv:2109.09658 , 2021.

Haolin Li, Yuhang Zhou, Ziheng Zhao, Siyuan Du, Jiangchao Yao, Weidi Xie, Ya Zhang, and Yanfeng Wang. Lorkd: Low-rank knowledge decomposition for medical foundation models. arXiv preprint arXiv:2409.19540 , 2024.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning , pages 19730-19742. PMLR, 2023.

Mingchen Li, Halil Kilicoglu, Hua Xu, and Rui Zhang. Biomedrag: A retrieval augmented large language model for biomedicine. Journal of Biomedical Informatics , 162:104769, 2025.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024.

Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Xu Sun, et al. Auto-encoding knowledge graph for unsupervised medical report generation. Advances in Neural Information Processing Systems , 34:1626616279, 2021.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36:34892-34916, 2023.

Jingyu Liu, Jie Lian, and Yizhou Yu. Chestx-det10: chest x-ray dataset on detection of thoracic abnormalities. arXiv preprint arXiv:2006.10550 , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence , pages 4513-4519, 2021.

Yan Luo, Min Shi, Muhammad Osama Khan, Muhammad Muneeb Afzal, Hao Huang, Shuaihang Yuan, Yu Tian, Luo Song, Ava Kouhana, Tobias Elze, et al. Fairclip: Harnessing fairness in vision-language learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12289-12301, 2024.

George B Moody. Physionet. In Encyclopedia of Computational Neuroscience , pages 2806-2808. Springer, 2022.

World Health Organization et al. International classification of diseases-icd. 2009.

Yifan Peng, Justin F Rousseau, Edward H Shortliffe, and Chunhua Weng. Ai-generated text may have a role in evidence-based medicine. Nature medicine , 29(7):1593-1594, 2023.

Fernando Pérez-García, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Matthew P Lungren, et al. Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence , pages 1-12, 2025.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR, 2021.

Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital medicine , 4(1):86, 2021.

Maya Rotmensch, Yoni Halpern, Abdulhakim Tlimat, Steven Horng, and David Sontag. Learning a health knowledge graph from electronic medical records. Scientific reports , 7(1):5994, 2017.

Zohaib Salahuddin, Henry C Woodruff, Avishek Chatterjee, and Philippe Lambin. Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Computers in biology and medicine , 140:105111, 2022.

Gregor Stiglic, Primoz Kocbek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Interpretability of machine learning-based prediction models in healthcare. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 10(5):e1379, 2020.

Vivek Subbiah. The next generation of evidence-based medicine. Nature medicine , 29(1):49-58, 2023.

Heung-Il Suk, Seong-Whan Lee, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, et al. Hierarchical feature representation and multimodal fusion with deep learning for ad/mci diagnosis. NeuroImage , 101:569-582, 2014.

Jesse Sun, Fatemeh Darbehani, Mark Zaidi, and Bo Wang. Saunet: Shape attentive u-net for interpretable medical image segmentation. In International conference on medical image computing and computerassisted intervention , pages 797-806. Springer, 2020.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Guangyu Wang, Xiaohong Liu, Zhen Ying, Guoxing Yang, Zhiwei Chen, Zhiwen Liu, Min Zhang, Hongmei Yan, Yuxing Lu, Yuanxu Gao, et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof-of-concept trial. Nature Medicine , 29(10):2633-2642, 2023.

Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , volume 2022, page 3876, 2022.

Tom Nuno Wolf, Sebastian Pölsterl, Christian Wachinger, Alzheimer's Disease Neuroimaging Initiative, et al. Daft: A universal module to interweave tabular data and 3d images in cnns. NeuroImage , 260:119505, 2022.

Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards evaluating and building versatile large language models for medicine. npj Digital Medicine , 8(1): 58, 2025.

Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085 , 2024.

Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024 , pages 6233-6251, 2024.

Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Bowen Jin, May D Wang, Joyce C Ho, and Carl Yang. Ram-ehr: Retrieval augmentation meets clinical predictions on electronic health records. arXiv preprint arXiv:2403.00815 , 2024.

Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044 , 2025.

Chonghua Xue, Sahana S Kowshik, Diala Lteif, Shreyas Puducheri, Varuna H Jasodanand, Olivia T Zhou, Anika S Walia, Osman B Guney, J Diana Zhang, Serena T Pham, et al. Ai-based differential diagnosis of dementia etiologies on multimodal data. Nature Medicine , 30(10):2977-2989, 2024.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael Yao, Chris Callison-Burch, James Gee, and Mark Yatskar. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. Advances in Neural Information Processing Systems , 37:90683-90713, 2024.

Wenfang Yao, Kejing Yin, William K Cheung, Jia Liu, and Jing Qin. Drfuse: Learning disentangled representation for clinical multi-modal fusion with missing modality and modal inconsistency. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 16416-16424, 2024.

Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy S Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. Advances in Neural Information Processing Systems , 35:37309-37323, 2022.

Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, et al. Huatuogpt, towards taming language model to be a doctor. arXiv preprint arXiv:2305.15075 , 2023.

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915 , 2023.

Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visuallanguage pre-training on chest radiology images. Nature Communications , 14(1):4542, 2023.

Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference , pages 2-25. PMLR, 2022.

Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason McGough, and Lin Yang. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6428-6436, 2017.

Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for augmented generation: A survey. arXiv preprint arXiv:2303.10868 , 2023.

Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence , 4(1):32-40, 2022.

Juexiao Zhou, Liyuan Sun, Yan Xu, Wenbin Liu, Shawn Afvari, Zhongyi Han, Jiaoyan Song, Yongzhi Ji, Xiaonan He, and Xin Gao. Skincap: A multi-modal dermatology dataset annotated with rich medical captions. arXiv preprint arXiv:2405.18004 , 2024.

Yuhang Zhou, Siyuan Du, Haolin Li, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Reprogramming distillation for medical foundation models. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 533-543. Springer, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 3 explains our method. Section 4 presents experimental results to verify RAD on different datasets with diverse anatomies (Section 4.2) and different model structures (Section 4.4).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitation is discussed in Appendix A.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: There are no included theoretical results in this work.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included a detailed experimental setup in Section 4.1, including dataset and implementation details (configuration like metrics, hyperparameters, etc) The preprocessing of the datasets is provided in Appendix C.1.1. The construction of the retrieval corpus is provided in Appendix B.1. And the dataset and code will be publicly available.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available at: https://github.com/tdlhl/RAD . With the exception of MIMIC-ICD53, all datasets are publicly available. We will also make our constructed MIMIC-ICD53 available on PhysioNet upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have included a detailed experimental setup in Section 4.1, including dataset and implementation details (configuration like metrics, hyperparameters, etc). And the details can also be found in the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have reported the variance of our method over 5 runs in Table 1. The full results with variance of all baselines are presented in Appendix C.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All the experiments are conducted on a single NVIDIA A100 GPU, and we have reported the compute sources in the implementation details of Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impacts have been discussed in Appendix A.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no safeguard risk in this work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original papers and followed their license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code is available at: https://github.com/tdlhl/RAD . Due to licensing restrictions, our new dataset (MIMIC-ICD53) will be available only on PhysioNet upon publication. The construction details of this dataset have been included in Appendix C.1.1.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We utilized LLMs for the guideline refinement, which has been thoroughly detailed in Section 3.2.1. The function of LLM here is to serve as a powerful long-context text summarization tool.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

A Further Discussion

A.1 Broader impact

The method proposed in this paper can effectively enhance the diagnostic capability of multimodal medical models. With the integration of the guidelines, RAD is optimized through intervention in accordance with the guidelines. This not only improves diagnostic accuracy but also strengthens the model's interpretability, making its decision-making process more transparent and deployable in realworld clinical scenarios. Specifically, the systematic integration of multimodal data (imaging, text, and structured records) enables RAD to capture disease manifestations from multiple perspectives, potentially advancing personalized medicine through comprehensive patient profiling. However, the use of multimodal clinical data, including sensitive patient records and imaging features, necessitates stringent compliance with relevant regulations to prevent misuse or unintended leakage of private health information.

A.2 Limitations

A limitation of the current implementation is the static retrieval knowledge corpus. While medical guidelines undergo periodic updates (e.g., every 3-5 years) to incorporate new evidence and diseases, RAD relies on a fixed knowledge base that may require manual updates to reflect revised diagnostic standards. This temporal mismatch could be addressed by regular updates of guidelines and further fine-tuning of the models, which would enhance long-term clinical relevance without compromising current performance.

B Method Details

B.1 Knowledge-corpus Construction

For retrieving disease-related diagnostic knowledge, we collect medical knowledge from four distinct sources: 'Wiki', 'Research', 'Guideline', and 'Book'. Wikipedia provides comprehensive and general descriptions of target diseases, such as standard disease nomenclature, formal medical definitions, and clinically relevant subcategories. The processed data are obtained from Huggingface 2 . Research incorporates the latest research articles from PubMed (a premier database of biomedical literature). These articles provide cutting-edge findings in disease mechanisms, diagnostic criteria, and therapeutic interventions. We utilize the 2024 PubMed baseline 3 , which is a complete snapshot of PubMed data. We filter the valid data through their paper titles and corresponding abstracts. Guideline includes 45K clinical practice guidelines from 13 sources. The guidelines provide rigorously vetted diagnostic criteria and treatment protocols for medical practitioners, serving as a critical component for reliable decision support. We employ the Clinical Guidelines dataset [12] and use the provided scripts to crawl non-redistributable portions of the data. Book. consists of diverse medical textbooks. These books cover well-organized basic medical knowledge in surgery, medical imaging, and drugs, etc. We follow MedOmniKB [13] to collect 18K PDF documents from online medical libraries and academic publishers. Then, deduplicate and filter these books to obtain the final retrieval database.

B.2 Details for LLM Refinement

An example of the final guideline is presented in Figure 6. For more guidelines, please refer to our GitHub repository.

The detailed prompt template for LLM refinement is shown in Figure 7. In the prompt, {dis-ease_icd_name} is the disease name e i , {topk} is the number of preserved documents, and {re-trieve_passages_str} is the content of the document.

The guideline of "bronchitis":

Summary of Key Diagnostic Features for Bronchitis

Disease Description: Bronchitis is an inflammation of the bronchi, the air passages in the lungs. It can be classified into two main types: acute and chronic. Acute bronchitis is typically a self-limiting condition characterized by a cough that may produce sputum and is often caused by viral infections. Chronic bronchitis, on the other hand, is a long-term condition defined by a productive cough lasting for at least three months in two consecutive years, often associated with chronic obstructive pulmonary disease (COPD). The primary risk factor for chronic bronchitis is tobacco smoking, with other factors including air pollution and occupational exposures.

Important Lab Tests and Values: - Acute Bronchitis: White Blood Cell Count (WBC): Usually normal or slightly elevated. C-reactive Protein (CRP): May be slightly elevated but not typically high. Sputum Culture: Not routinely necessary, but can be useful if bacterial infection is suspected. Chronic Bronchitis: Pulmonary Function Tests (PFTs): Reduced FEV1/FVC ratio, indicating airflow obstruction. Sputum Analysis: Increased mucus production, often with neutrophil infiltration. Blood Gas Analysis:** May show hypoxemia and hypercapnia in advanced cases.

Key Radiological or Clinical Findings: Acute Bronchitis: Chest X-ray: Usually normal, but may show hyperinflation or peribronchial thickening. Physical Examination: Wheezing, crackles, and rhonchi on auscultation. Chronic Bronchitis: Chest X-ray: May show hyperinflation, increased bronchovascular markings, and signs of emphysema. CT Scan: Can reveal bronchial wall thickening and mucus plugging. Physical Examination: Barrel chest, cyanosis, and signs of cor pulmonale in advanced cases.

Diagnostic Symptoms or Relevant Clinical Features: Acute Bronchitis: Cough: Initially dry, then becomes productive with clear or yellowish sputum. Fever: Usually mild or absent; high fever suggests pneumonia. Fatigue and Body Aches: Common but generally mild. Wheezing and Shortness of Breath: May be present, especially in patients with underlying asthma. Chronic Bronchitis: Cough: Persistent, productive cough with sputum, often for at least three months in two consecutive years. Dyspnea: Shortness of breath, especially on exertion. Wheezing: Common, especially in the morning. Chest Pain: May occur due to prolonged coughing. Fatigue and Malaise: Persistent, often due to chronic hypoxemia.

B.3 Derivation of the Guideline-Enhanced Contrastive Loss

Here, we derive the form of the most basic cross-entropy loss to the form in Eq. (4). We demonstrate the equivalence between the sigmoid-based cross-entropy formulation and the logit-style implementation of our supervised contrastive loss.

The equation in cross-entropy form with explicit sigmoid terms is defined as:

where ϕ ij is short for ϕ ( I i , S ij ) = I ⊤ i S ij /τ , the similarity score between the modality-specific feature and the guideline feature. σ ( · ) is the Sigmoid function, the logit of the similarity score is σ ( ϕ ij ) = 1 1+ e -ϕ ij . The difference between Eq. (8) and the standard cross-entropy loss is that we use the similarity score as logits, and we add the normalization coefficient 1 | P i | to balance the gradient contribution of each positive label in multi label scenarios. The following is a step-by-step derivation. First, substitute the sigmoid function into Eq. (8):

Prompt for LLM Refinement:

Your task is to help filter and summarize the relevant information from multiple sets of retrieved external knowledge (from 4 different sources) to support my multi-modal disease classification model in diagnosing the specific disease associated with the provided ICD disease description. For the given disease, identify the critical symptoms, lab indicators, and radiological features that are most strongly associated with the disease diagnosis. Discard any information that is unrelated or irrelevant to disease classification. You do not need to focus on treatment options, but instead, concentrate on factors that would help in diagnosing the disease. You will be provided with a set of documents (containing topk retrieved documents, each approximately 2000 characters), containing a range of medical information. Your job is to:

Review the retrieved documents and determine which information is directly relevant to diagnosing the disease, based on its ICD code and description. Eliminate any information that is unrelated to diagnosis or classification, such as treatment options, management strategies, or irrelevant clinical details. Focus on identifying key diagnostic features, including symptoms, laboratory test results, and imaging findings that help confirm the presence of the disease. Evaluate the importance of each retrieved documents: Some documents may provide more critical or reliable information than others. Prioritize information that is most relevant and useful for diagnosing the disease, even if it means excluding less relevant details from certain documents. Summarize the most relevant content into a single cohesive summary of approximately 2000 characters (or 500 words). The summary should include the essential diagnostic criteria, including lab values, clinical features, and radiological findings.

The summary should emphasize:

Abrief explanation or description of the disease, including any variations or related conditions under the same ICD code. Important lab tests and values (e.g., white blood cell count, Creactive protein). Key radiological or clinical findings associated with the disease's presence (e.g., lung opacity, pleural effusion). Any diagnostic symptoms or relevant clinical features. Discard information that is not useful for diagnosing the disease.

Please ensure that the summary is concise and directly related to diagnosis, omitting irrelevant details.

Disease description (ICD name): {disease_icd_name} Retrieved passages: {retrieve_passages_str}

The final equation is:

We use this form directly in the main body Section 3.2.2.

hypertension

diabetes mellitus

pulmonary collapse

tachycardia

heart failure

complications and ill-defined descriptions of heart disease, cardiomegaly

pulmonary congestion, edema, and pleural effusion

lung opacity

pneumonia

surgical instruments, materials and cardiovascular devices

chronic total occlusion of coronary artery

hypotension

myocardial infarction

chronic ischemic heart disease

atrial fibrillation and flutter

extrinsic asthma, unspecified

chronic airway obstruction, not elsewhere classified

pleurisy without mention of effusion or current tuberculosis

pneumoconiosis

primary pulmonary hypertension

fracture of rib(s), sternum and thoracic spine

acute posthemorrhagic anemia

traumatic, spontaneous, and tension pneumothorax, air leak

endomyocardial fibrosis

other coagulation defects

other respiratory disorders, consolidation

malignant neoplasm of trachea

atherosclerosis of aorta

bronchitis

pneumonitis due to solids, liquids, and radiation

abnormal serum enzyme levels

cardiomyopathy

pulmonary fibrosis and other interstitial pulmonary diseases

other pleural conditions

other cardiac arrhythmias

vitamin d deficiency

esophageal varices

pulmonary embolism

angina decubitus

bronchiectasis

poisoning by, adverse effect of and underdosing of agents primarily affecting the cardiovascular system

abscess of lung and mediastinum, pyothorax, mediastinitis

acute and subacute bacterial endocarditis

arterial embolism and thrombosis

phlebitis and thrombophlebitis

acute and subacute endocarditis

thalassemia

flu due to avian flu virus

farmers' lung and pneumoconiosis due to asbestos and mineral fibers

acute bronchiolitis and unspecified acute lower respiratory infection

0

2000

4000

6000

8000

10000

12000

14000

# Sample

C Experimental Details

C.1 Details of Datasets

C.1.1 Construction Process of MIMIC-ICD53

First, we merged and aligned the ED, HOSP, and ICU parts of MIMIC-IV [29]. Subsequently, we aligned the processed MIMIC-IV dataset with the MIMIC-CXR-JPG dataset [28]. We utilized patient_id and study_id to align the datasets at the patient level. Given that temporal patient dynamics were not considered, we selected the most recent radiological examination for each patient, including the associated images and reports. For temporal alignment, we extracted EHR data and ICD disease codes from MIMIC-IV within a three-day window following the radiological examination. After excluding instances with missing modalities or labels, we obtained a final sample size of 51830. For disease labeling, we standardized the granularity of diagnoses according to the ICD-10 classification, using the format Xab (where X is a letter and ab are digits), resulting in over 2000 unique labels. To refine and further clean the dataset, we consulted LLMs and then physicians to identify and select 53 critical disease categories that were related to thoracic and cardiovascular conditions or could be identified using laboratory indicators in the EHR. For the numerical indicators in the EHR, we quantized each indicator on a 0-10 integer scale based on its corresponding normal range limits. A value of 4-7 is considered normal, 0-3 indicates too low, and 8-10 signifies too high. The label distribution of MIMIC-ICD53 is shown in Figure 8. The training set and test set are randomly divided in a ratio of 4:1. The final processed dataset, termed MIMIC-ICD53, will be made publicly available on PhysioNet after publication. (The MIMIC dataset requires that all datasets developed based on MIMIC can only be released on PhysioNet.)

C.1.2 Preprocess of NACC

The National Alzheimer's Coordinating Center (NACC) dataset [5] is a large, standardized resource comprising clinical and neuropathological data collected from individuals assessed at Alzheimer's Disease Research Centers (ADRCs) across US, which consists various neurodegenerative diseases, like Alzheimer's disease, Parkinson's disease, vascular dementia, and other forms of cognitive impairment. We follow [63] to organize the dataset, resulting in 11 labels including: "Normal cognition" (NC), "Mild cognitive impairment" (MCI), "Dementia" (DE), "Alzheimer's disease" (AD), "Vascular dementia, vascular brain injury and vascular dementia" (VD), "Lewy body dementia, including dementia with Lewy bodies and Parkinson's disease dementia" (LBD), "Psychiatric conditions including schizophrenia, depression, bipolar disorder, anxiety and posttraumatic stress disorder"

(PSY), "Frontotemporal lobar degeneration and its variants, including primary progressive aphasia, corticobasal degeneration and progressive supranuclear palsy, and with or without amyotrophic lateral sclerosis" (FTD), "Systemic and environmental factors including infectious diseases (HIV included), metabolic, substance abuse / alcohol, medications, systemic disease and delirium" (SEF), "Other dementia conditions, including neoplasms, Down syndrome, multiple systems atrophy, Huntington's disease and seizures" (ODE), and "Moderate/severe traumatic brain injury, repetitive head injury and chronic traumatic encephalopathy" (TBI). The label distribution of NACC is shown in Figure 9. Given that NACC contains over 800 distinct EHR variables, selecting the most relevant features for analysis was a critical step in our study. To ensure both scientific validity and clinical interpretability, we first utilized LLM for cleaning and double-checked with several physicians.

NC

MCI

DE

AD

VD

LBD

PSY

FTD

SEF

ODE

TBI

0

500

1000

1500

2000

2500

# Sample

Finally, we distilled the original set down to a final list of 36 key EHR variables. The selected variables include height, weight, body mass index (BMI), systolic blood pressure, diastolic blood pressure, cortical atrophy (Alzheimer's disease marker), small vessel disease (vascular dementia related), left motor cortex vascular lesion, right motor cortex vascular lesion, normal pressure hydrocephalus gait, parkinsonian signs (tremor/rigidity), bradykinesia (Parkinsonian symptom), neck rigidity (dystonia), gait disturbance, history of hypertension, history of diabetes, history of cardiovascular disease, history of stroke, history of Parkinson's disease, sleep apnea, REM sleep behavior disorder (RBD), his- tory of traumatic brain injury (TBI), delusions, hallucinations, depressive symptoms, agitation or aggression, anti-dementia medication (e.g.), Parkinson's disease medication (e.g.), anticoagulant use (stroke prevention), antidepressant medication, postural instability (Parkinson's or Lewy body dementia), APOE ϵ 4 allele (Alzheimer's disease risk), hypercholesterolemia (vascular risk), amyotrophic lateral sclerosis (ALS) signs, left visual cortex functional impairment, and right visual cortex functional impairment. The final processed dataset is randomly divided in a ratio of 4:1 for training and testing.

C.1.3 Details of Harvard-FairVLMed and SkinCAP

The Harvard-FairVLMed dataset [41], sourced from the Department of Ophthalmology at Harvard Medical School, contains 10,000 multimodal samples (7,000 train, 1,000 val, 2,000 test) with paired clinical notes, diagnostic labels, and detailed demographic attributes (race, gender, ethnicity, language). The dataset is publicly available under the CC BY-NC-ND 4.0 license at Github 4 . We directly used the original dataset.

SkinCAP is a multimodal dermatology dataset containing 4,000 expert-annotated skin disease images with rich natural language descriptions [76]. The dataset combines cases from diverse dermatology image datasets, all annotated by board-certified dermatologists to ensure clinical accuracy. It is publicly available under an open license at HuggingFace 5 . To address class imbalance, we removed tail categories with too few positive samples, resulting in a filtered dataset of 2,526 samples with 50 disease labels. The final dataset was partitioned into a 4:1 train-test split.

C.2 Benchmarking MIMIC-ICD53

To further evaluate the quality of our constructed dataset MIMIC-ICD53, we employed various unimodal methods to train and test its performance. For the visual modality, we selected ResNet-18, ResNet-50 [22], ViT [15], and Swin Transformer [39], along with two SOTA CXR-specific pretrained models, UniChest [14] and RadDino [45] as the baselines. Based on both computational efficiency and data leakage prevention considerations, we ultimately designated ResNet-50 as the standard visual backbone for the main experiments. ViT is also investigated in ablation studies in Section C.6.

For electronic health record (EHR) data, we first leveraged its inherent tabular structure by treating each EHR attribute as an input dimension, with corresponding numerical values assigned to their respective dimensions. We experimented with MLP and a SOTA tabular data process method TabFPN [24], but both exhibited suboptimal performance. Consequently, we reformatted the EHR data into natural language text using the following template: "Laboratory values within the 4-7 range indicate normal levels, values 0-3 suggest clinically low levels, and values 8-10 denote elevated levels. The current panel includes [ATTRIBUTE] with the discretized value of [VALUE]..." . We then evaluated the reformatted EHR data using classic backbone ClinicalBERT [55], BioClinicalBERT [2], and PubMedBERT [18]. We also included natural and medical LLMs LLaMa [54] and MMedS [58]. Specifically, we replace the last layer of LLMs with a classification head to adapt to the text classification task. Due to the high consumption of computing resources, we only use LoRA [25] to fine-tune the LLM-based models. We further conducted experiments using the same text encoders for the report modality alone and reports combined with EHR data. As shown in Table 6, unimodal performance analysis reveals that the report modality achieves the highest diagnostic results, followed by the EHR modality. Combining the two modalities in text form (Report+EHR) can bring significant performance gains. For the visual modality, the performance gap between different backbone architectures is relatively small. While for the text-based modality, LLM-based

models generally outperform BERT-based models. Among the BERT-based models, ClinicalBERT consistently achieves the best performance. Considering model size and practicality, we selected ClinicalBERT as the default text encoder in our RAD framework.

C.3 The Variance of Baselines

Due to space limitations, we do not show the variance of the baselines in Table 1. Here we add the variance of all baselines in Table 7. It can be observed that the overall variance of SkinCAP is the largest among all datasets. Meanwhile, there is no significant gap between the variance of different methods, all methods exhibit stable performance across the four datasets.

C.4 Label-wise Analysis of MIMIC-ICD53

In addition to evaluating the overall performance of RAD in Section 4.2, we also investigated its comprehensive performance across all categories on MIMIC-ICD53. As illustrated in Figure 10, our method achieved the highest scores in 41 out of 53 categories across both AUC and F1 metrics. Furthermore, in the long-tail categories (classes with fewer than 100 positive samples), our method outperformed the previous SOTA by 1.60% in AUC and by 4.44% in F1. Importantly, the performance gains in these long-tail categories exceeded the average improvements observed across all categories, underscoring the robustness and practical utility of RAD under real-world clinical settings.

C.5 Interpretability Cases

In this subsection, we further explore the textual interpretability of RAD by presenting additional visualization cases. In addition, the full names of the abbreviations in Table 4 are given here. In Table 4, the "PC" is short for Platelet Count, "Bilirubin" is Serum Bilirubin, "ALT" is Alanine

[CLS] < report > : final report chest radiograph performed on _ _ _ comparison : prior exam from _ _ _ .

clinical history : altered mental status , assess pneumonia or pulmonary edema .

findings : pa and lateral views of the chest were provided . the lungs are clear

without focal consolidation , effusion , or pneumothorax .

cardiomediastinal silhouette appears within normal limits . the bony

structures are intact . no free air below the right hemidiaphragm .

impression : no acute intrathoracic process . < ehr > : the patient is a 79 - year

- old female . the values of lab results between 4 and 7 indicate normal levels , values from 0 to 3

indicate low levels , and values from 8 to 10 indicate high levels . the lab results include anion gap

with a value of 5 . 0 , bicarbonate with a value of 6 . 0 , calcium , total with a value of 4 . 0 ,

chloride with a value of 6 . 0 , creatine kinase ( ck ) with a value of 3 . 0 , creatine kinase , mb

isoenzyme with a value of 4 . 0 , creatinine with a value of 9 . 0 , folate with a value of 6 . 0 ,

glucose with a value of 10 . 0 , hematocrit with a value of 3 . 0 , hemoglobin with a value of 3 . 0 , inr

( pt ) with a value of 7 . 0 , lactate with a value of 5 . 0 , mch with a value of 6 . 0 , mchc with a value

of 5 . 0 , mcv with a value of 6 . 0 , magnesium with a value of 5 . 0 , pt with a value of 6 . 0 , ptt

with a value of 3 . 0 , phosphate with a value of 5 . 0 , platelet count with a value of 5 . 0 ,

potassium with a value of 6 . 0 , rdw with a value of 5 . 0 , red blood cells with a value of 2 . 0 ,

sodium with a value of 6 . 0 , thyroid stimulating hormone with a value of 7 . 0 , urea nitrogen

with a value of 10 . 0 , vitamin b12 with a value of 4 . 0 , white blood cells with a value of 7 . 0 .

[SEP]

Guideline:

…

Key indicators:

1. Creatinine;

2. White Blood

Cell Count;

3. Serum Sodium;

…

[CLS] < report > : final report chest radiograph performed on _ _ _ comparison : prior exam from _ _ _ .

clinical history : altered mental status , assess pneumonia or pulmonary edema . findings : pa and

lateral views of the chest were provided . the lungs are clear without focal consolidation , effusion , or

pneumothorax . cardiomediastinal silhouette appears within normal limits . the bony structures are intact .

no free air below the right hemidiaphragm . impression : no acute intrathoracic process . < ehr > :

the patient is a 79 - year - old female . the values of lab results between 4 and 7 indicate normal

levels , values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high levels . the

lab results include anion gap with a value of 5 . 0 , bicarbonate with a value of 6 . 0 , calcium ,

total with a value of 4 . 0 , chloride with a value of 6 . 0 , creatine kinase ( ck ) with a

value of 3 . 0 , creatine kinase , mb isoenzyme with a value of 4 . 0 , creatinine with a

value of 9 . 0 , folate with a value of 6 . 0 , glucose with a value of 10 . 0 , hematocrit

with a value of 3 . 0 , hemoglobin with a value of 3 . 0 , inr ( pt ) with a value of 7 . 0 ,

lactate with a value of 5 . 0 , mch with a value of 6 . 0 , mchc with a value of 5 . 0 , mcv with a

value of 6 . 0 , magnesium with a value of 5 . 0 , pt with a value of 6 . 0 , ptt with a value of

3 . 0 , phosphate with a value of 5 . 0 , platelet count with a value of 5 . 0 , potassium with

a value of 6 . 0 , rdw with a value of 5 . 0 , red blood cells with a value of 2 . 0 , sodium

with a value of 6 . 0 , thyroid stimulating hormone with a value of 7 . 0 ,

urea nitrogen with a value of 10 . 0 , vitamin b12 with a value of 4 .

0 , white blood cells with a value of 7 . 0 . [SEP]

w/o RAD

RAD

[CLS] < report > : final report examination : chest single view indication : _ _ _ yof with

babesiosis and hcap - worsening effusions ? / / _ _ _ yof with babesiosis and

hcap - worsening effusions ? technique : portable ap upright comparison : _ _ _ .

findings : compared to the previous film there is increasing bilateral pleural effusions . _ _ _ be

atelectasis in the left base . no pulmonary vascular congestion pre heart is normal in

size impression : slight increase to the bilateral pleural effusions . < ehr > : the patient is a 62

- year - old female . the values of lab results between 4 and 7 indicate normal levels ,

values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high levels . the

lab results include absolute basophil count with a value of 3 . 0 , absolute eosinophil

count with a value of 3 . 0 , absolute lymphocyte count with a value of 4 . 0 ,

absolute monocyte count with a value of 10 . 0 , absolute neutrophil count with a

value of 10 . 0 , alanine aminotransferase ( alt ) with a value of 10 . 0 , alkaline phosphatase with

a value of 9 . 0 , anion gap with a value of 4 . 0 , asparate aminotransferase ( ast ) with a value of 10 .

0 , bands with a value of 5 . 0 , basophils with a value of 4 . 0 , bicarbonate with a value of 4 . 0 ,

bilirubin , direct with a value of 10 . 0 , bilirubin , total with a value of 9 . 0 , calcium , total with a

value of 2 . 0 , chloride with a value of 4 . 0 , creatinine with a value of 5 . 0 , eosinophils with a

value of 3 . 0 , factor vii with a value of 4 . 0 , factor viii with a value of 10 . 0 , factor x with

a value of 4 . 0 , fibrinogen , functional with a value of 4 . 0 , glucose with a value of 7 . 0 , hematocrit

with a value of 1 . 0 , hemoglobin with a value of 2 . 0 , inr ( pt ) with a value of 10 . 0 [SEP]

Guideline:

…

Key indicators:

1. Alanine

Aminotransferase;

2. Aspartate

Aminotransferase;

…

[CLS] < report > : final report examination : chest single view indication : _ _ _ yof

with babesiosis and hcap - worsening effusions ? / / _ _ _ yof with

babesiosis and hcap - worsening effusions ? technique : portable ap

upright comparison : _ _ _ . findings : compared to the previous film there is increasing

bilateral pleural effusions . _ _ _ be atelectasis in the left base . no pulmonary vascular congestion pre

heart is normal in size impression : slight increase to the bilateral pleural effusions . < ehr > : the

patient is a 62 - year - old female . the values of lab results between 4 and 7 indicate normal

levels , values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high levels . the

lab results include absolute basophil count with a value of 3 . 0 , absolute eosinophil count with a

value of 3 . 0 , absolute lymphocyte count with a value of 4 . 0 , absolute monocyte count with a value

of 10 . 0 , absolute neutrophil count with a value of 10 . 0 , alanine aminotransferase

( alt ) with a value of 10 . 0 , alkaline phosphatase with a value of 9 . 0 , anion gap

with a value of 4 . 0 , asparate aminotransferase ( ast ) with a value of 10 . 0 , bands

with a value of 5 . 0 , basophils with a value of 4 . 0 , bicarbonate with a value of 4 . 0 , bilirubin ,

direct with a value of 10 . 0 , bilirubin , total with a value of 9 . 0 , calcium , total with a

value of 2 . 0 , chloride with a value of 4 . 0 , creatinine with a value of 5 . 0 , eosinophils with a value of

3 . 0 , factor vii with a value of 4 . 0 , factor viii with a value of 10 . 0 , factor x with a value

of 4 . 0 , fibrinogen , functional with a value of 4 . 0 , glucose with a value of 7 . 0 , hematocrit with a

value of 1 . 0 , hemoglobin with a value of 2 . 0 , inr ( pt ) with a value of 10 . 0 [SEP]

w/o RAD

RAD

[CLS] < report > : final report chest radiograph indication : worsening respiratory distress ,

evaluation . comparison : _ _ _ . findings : as compared to the previous radiograph , there is no

relevant change . elevation of the right hemidiaphragm with bilateral relatively extensive

parenchymal opacities . the appearance of these opacities and their distribution suggests

atelectasis rather than pneumonia . borderline size of the cardiac silhouette .

tortuosity of the thoracic aorta . no pleural effusions . no pneumothorax . <

ehr > : the patient is a 64 - year - old female . the values of lab results between 4 and 7

indicate normal levels , values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high

levels . the lab results include alanine aminotransferase ( alt ) with a value of 5 . 0 , alkaline

phosphatase with a value of 5 . 0 , anion gap with a value of 6 . 0 , asparate aminotransferase

( ast ) with a value of 6 . 0 , basophils with a value of 4 . 0 , bicarbonate with a value of 2 . 0 ,

bilirubin , total with a value of 5 . 0 , calcium , total with a value of 5 . 0 , calculated total co2

with a value of 4 . 0 , chloride with a value of 6 . 0 , creatinine with a value of 3 . 0 , eosinophils with a

value of 4 . 0 , glucose with a value of 3 . 0 , hematocrit with a value of 1 . 0 , hemoglobin with a value

of 1 . 0 , lactate with a value of 4 . 0 , lactate dehydrogenase ( ld ) with a value of 6 . 0 ,

lymphocytes with a value of 2 . 0 , mch with a value of 6 . 0 , mchc with a value of 3 . 0 , mcv with a

value of 7 . 0 , magnesium with a value of 4 . 0 , monocytes with a value of 5 . 0 , neutrophils with

a value of 9 . 0 , phosphate with a value of 7 . 0 , platelet count with a value of 4 . 0 , potassium with

a value of 5 . 0 , rbc with a value [SEP]

Guideline:

…

Key indicators:

1. Alanine

Aminotransferase;

2. Aspartate

Aminotransferase;

4. Platelet Count

…

[CLS] < report > : final report chest radiograph indication : worsening

respiratory distress , evaluation . comparison : _ _ _ . findings : as compared

to the previous radiograph , there is no relevant change . elevation of the right

hemidiaphragm with bilateral relatively extensive parenchymal opacities . the

appearance of these opacities and their distribution suggests atelectasis rather than

pneumonia . borderline size of the cardiac silhouette . tortuosity of the

thoracic aorta . no pleural effusions . no pneumothorax . < ehr > : the patient

is a 64 - year - old female . the values of lab results between 4 and 7 indicate normal

levels , values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high

levels . the lab results include alanine aminotransferase ( alt ) with a value of

5 . 0 , alkaline phosphatase with a value of 5 . 0 , anion gap with a value of 6 . 0 ,

4 . 0 , bicarbonate with a value of 2 . 0 , bilirubin , total with a value of 5 . 0 , calcium ,

total with a value of 5 . 0 , calculated total co2 with a value of 4 . 0 , chloride with a value of 6 . 0 ,

creatinine with a value of 3 . 0 , eosinophils with a value of 4 . 0 , glucose with a value of 3 . 0 ,

hematocrit with a value of 1 . 0 , hemoglobin with a value of 1 . 0 , lactate with a value of 4 . 0 ,

lactate dehydrogenase ( ld ) with a value of 6 . 0 , lymphocytes with a value of 2 . 0 , mch with a

value of 6 . 0 , mchc with a value of 3 . 0 , mcv with a value of 7 . 0 , magnesium with a value of 4 . 0 ,

monocytes with a value of 5 . 0 , neutrophils with a value of 9 . 0 , phosphate with a value of 7 . 0 , platelet

count with a value of 4 . 0 , potassium with a value of 5 . 0 , rbc with a value [SEP]

w/o RAD

RAD

[CLS] < report > : final report ap chest , 8 : 56 p . m . on _ _ _ . history : postop day 5

after whipple procedure . continued shortness of breath . impression : ap

chest compared to _ _ _ : lung volumes remain quite low , due to substantial bibasilar

atelectasis , and there is also at least a small amount of pleural fluid on both sides of the

chest . there is no pneumothorax . heart is mildly enlarged as before , but there

is no pulmonary edema . right ventricular transvenous pacer lead

follows the expected course . < ehr > : the patient is a 79 - year - old male . the

values of lab results between 4 and 7 indicate normal levels , values from 0 to 3 indicate low levels , and

values from 8 to 10 indicate high levels . the lab results include albumin with a value of 2 . 0 , anion

gap with a value of 5 . 0 , bicarbonate with a value of 5 . 0 , calcium , total with a value of 4 . 0 ,

chloride with a value of 6 . 0 , creatinine with a value of 6 . 0 , digoxin with a value of 3 . 0 , ferritin

with a value of 6 . 0 , glucose with a value of 10 . 0 , hematocrit with a value of 4 . 0 , hemoglobin with

a value of 3 . 0 , inr ( pt ) with a value of 5 . 0 , iron with a value of 3 . 0 , iron binding capacity ,

total with a value of 3 . 0 , mch with a value of 4 . 0 , mchc with a value of 2 . 0 , mcv with a

value of 6 . 0 , magnesium with a value of 6 . 0 , pt with a value of 5 . 0 , ptt with a value of 4 . 0 ,

phosphate with a value of 3 . 0 , platelet count with a value of 5 . 0 , potassium with a value of

6 . 0 , rdw with a value of 8 . 0 , red blood cells with a value of 3 . 0 , sodium with a value of

5 . 0 , transferrin with a value of 3 . 0. [SEP]

Guideline:

…

Key indicators:

1. Serum Iron and

Ferritin;

2. Serum Iron and

Iron-Binding

Capacity;

…

[CLS] < report > : final report ap chest , 8 : 56 p . m . on _ _ _ . history : postop day 5

after whipple procedure . continued shortness of breath . impression : ap chest

compared to _ _ _ : lung volumes remain quite low , due to substantial bibasilar atelectasis ,

and there is also at least a small amount of pleural fluid on both sides of the chest . there is no

pneumothorax . heart is mildly enlarged as before , but there is no pulmonary edema .

right ventricular transvenous pacer lead follows the expected course . < ehr > :

the patient is a 79 - year - old male . the values of lab results between 4 and 7 indicate normal

levels , values from 0 to 3 indicate low levels , and values from 8 to 10 indicate high levels . the lab

results include albumin with a value of 2 . 0 , anion gap with a value of 5 . 0 , bicarbonate

with a value of 5 . 0 , calcium , total with a value of 4 . 0 , chloride with a value of 6 . 0 , creatinine

with a value of 6 . 0 , digoxin with a value of 3 . 0 , ferritin with a value of

6 . 0 , glucose with a value of 10 . 0 , hematocrit with a value of 4 . 0 , hemoglobin

with a value of 3 . 0 , inr ( pt ) with a value of 5 . 0 , iron with a value of 3 . 0 , iron

binding capacity , total with a value of 3 . 0 , mch with a value of 4 . 0 , mchc with a

value of 2 . 0 , mcv with a value of 6 . 0 , magnesium with a value of 6 . 0 , pt with a value of

5 . 0 , ptt with a value of 4 . 0 , phosphate with a value of 3 . 0 , platelet count with a

value of 5 . 0 , potassium with a value of 6 . 0 , rdw with a value of 8 . 0 , red blood cells with a

value of 3 . 0 , sodium with a value of 5 . 0 , transferrin with a value of 3 . 0. [SEP]

w/o RAD

RAD

Aminotransferase, "IBC" is Iron-Binding Capacity, "WBC" is White Blood Cell Count, and "AST" is Aspartate Aminotransferase. Figure 11 presents longer and clearer cases of interpretability on the textual data. The third row is the complete content of Figure 1, and the other rows are other cases

with different indicators. It can be observed that RAD enables the model to dynamically focus on indicators valuable for the current diagnostic goal based on the retrieved guidelines.

C.6 Ablation Study

In this part, we conduct a comprehensive ablation study to systematically evaluate the impact of architectural backbones, key components, and hyperparameter configurations in RAD.

Ablation on different backbones. In Section 4.4, we demonstrated the impact of RAD on model performance when replacing different modality backbones, as reflected in the average metrics, AUC, F1, and mAP. To provide a more comprehensive evaluation, we have included additional metrics in Table 8, such as Precision, Recall, Accuracy, and Acc-S, which collectively illustrate the holistic enhancement of the model in diagnostic tasks.

Ablation on LLM refinement of the retrieved knowledge. To assess the necessity of LLM refinement in Section 3.2.1, we further conducted an ablation study by comparing RAD with and without this step. Specifically, we constructed baseline guidelines through direct concatenation of topk retrieved documents and evaluated the performance on MIMIC-ICD53. The results in Table 9 demonstrate that all metrics have decreased after removing the LLM filtering step, underscoring the importance of regularizing the retrieved text. The LLM refinement not only performs semantic filtering to eliminate irrelevant contexts but also standardizes heterogeneous medical knowledge into actionable diagnostic guidelines-a critical enabler for effective downstream knowledge infusion.

Ablation on Knowledge Sources. To investigate the effect of modifying the knowledge base on model performance, we compare each knowledge source's individual performance, as well as the performance of adding or removing sources based on our default setting. As presented in Table 10, clinical guidelines provide the most valuable knowledge, as they directly encode established

diagnostic criteria, key indicators, and decision pathways specifically designed for clinical practice. Research papers show the lowest contribution, as they often focus on novel discoveries, experimental treatments, or specialized cases rather than established diagnostic standards. For well-established diseases, diagnostic criteria have become a consensus, making cutting-edge research less useful. When applying multiple knowledge sources, the performance of RAD remains stable across different source counts (±0.2 Avg), demonstrating RAD's robustness to knowledge base modifications.

C.7 Cost Analysis.

To evaluate the practical feasibility of our framework, we analyze the additional cost of RAD brought by the guideline acquisition process. Since we only perform retrieval at the label level, which avoids the prohibitive cost of sample-wise retrieval. The retrieval process incurs negligible computational overhead. The additional cost primarily occurs during the LLM refinement phase, where the retrieved documents are processed by LLMs for each label of the dataset. When using Qwen2.5-72B model, the average processing time is 33.83s per label. The total preprocessing time for guideline retrieval and refinement on MIMIC-ICD53 is around 31 minutes. The cost can be further reduced using smaller LLMs. When expanding to new datasets, the linear growth of retrieval cost O ( N disease ) ensures efficient scalability, as it grows significantly slower than patient samples O ( N sample ) in real-world scenarios. Furthermore, the retrieval and refinement steps are executed once per dataset during preprocessing, eliminating runtime delays during clinical deployment. In general, RAD achieves knowledge infusion with minimal practical overhead.

C.8 Hyper-parameter analysis

To evaluate the impact of hyperparameters in RAD, we conduct experimental analysis on the three key hyperparameters α , β , and topk . The hyperparameter α determines the weight of the guidelineenhanced contrastive learning for visual and text features. And β determines the weight of binary cross-entropy loss and the guideline-enhanced contrastive loss. Topk controls the number of retrieved documents for each disease (label). Figure 12 presents the performance trends as these parameters vary. As β decreases, the model performance initially improves before declining. This pattern arises because an excessively high weight over-prioritizes the auxiliary loss, disrupting the optimization of the primary classification loss. On the contrary, a very low weight also leads to performance degradation, underscoring the utility of the guideline in refining multi-modal feature representations. α exhibits a similar pattern. The optimal values for α and β are 1 e -2 and 1 e -1 ,

respectively. Regarding the topk hyperparameter, the model achieves worst performance at k = 1 , with gradual improvement as k increases. However, performance plateaus after reaching a threshold ( k = 10 here). When retrieving too few documents, limited informative content leads to suboptimal results. Conversely, retaining excessive documents beyond the threshold primarily introduces noisy knowledge, as core disease-related information has already been captured within the top-ranked documents.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a model for clinical diagnosis based on images and medical records, guided by \"knowledge\" about a set of specific diseases. The overall model consists of visual and text encoders that feed corresponding embeddings into attention-based label and guideline decoders. The model is trained and evaluated on several datasets demonstrating improved performance versus other models across all datasets. The paper appears to be comprehensive, with relevant ablation studies, derivation of loss function, details of datasets, etc., however some model training details are omitted, such as the number of epochs.\n\nSTRENGTHS AND WEAKNESSES:\nOverall quality of the work seems good, with good justification, detailed enough description of the proposed model and datasets, ablation studies, etc. Some additional details and experiments would strengthen the work, see \"Questions\" section of this review. While the work demonstrates clear improvements compared to other models, these improvements are still not sufficient to make this work practical as a medical diagnostics tool. \n\nWhile not entirely novel, model guidance through augmentation of relevant knowledge retrieved from four different sources seems to be helpful to achieve the stated performance. This is definitely a strength of this work.\n\nNice to see the construction of another dataset, MIMIC-ICD53. However this dataset seems to be hugely disbalanced. Not surprising, Figure 8 indicates model's low performance for many diseases that do not have sufficient number of samples.\n\nQUESTIONS:\nIt would be helpful to understand which of the four sources used for knowledge retrieval are the most relevant to the performance of the model. For example, would just the clinical practice guidelines be sufficient?\n\nDo you think the results can be improved if a more balanced dataset is used? Or can some data augmentation be used to enrich the parts of the dataset where only a few samples are present?\n\nLIMITATIONS:\nThe only limitation listed by the authors is due to the static retrieval knowledge corpus. I would not consider this to be a limitation as new knowledge can always be added to the knowledge base and the model can be re-trained. I think a bigger limitation of this work is a limited number of classes of diseases the model is trained to recognize as this is a classifier model. New diseases cannot be recognized even if there is an adequate knowledge about them is present in the knowledge base.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a framework (RAD) that explicitly injects disease-specific clinical guidelines into multimodal diagnostic models. The pipeline (i) retrieves disease-centric documents from four curated medical corpora and refines them with a large-language-model summariser, (ii) aligns image and text representations to these guidelines via a Guideline-Enhanced Contrastive Loss (GECL) term, and (iii) fuses modalities with a dual‐decoder transformer whose queries are the guideline text and the disease labels. A dual-axis evaluation scheme is proposed to quantify interpretability. RAD is evaluated against other competitors across four anatomies (chest, fundus, dermatology, and brain).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Overall, the motivation behind this work is clear and aligns with current calls for trustworthy medical AI, where expert knowledge is critical to ensuring reliable decision-making. The work is well-written and the method is clearly explained. \n\n- The experimental setup is robust, and the RAD method's applicability is successfully demonstrated across diverse medical modalities (chest, eye, skin, and brain). The extensive evaluations on four datasets validate its generalizability and superiority against state-of-the-art baselines.\n\n- The proposed Guideline-enhanced Feature Constraint (GECL) is empirically proven to be an effective component, contributing to the improved performance and interpretability of RAD.\n\nWeaknesses\n- It remains unclear that the dual-decoder design—one of the paper’s headline contributions—provides value beyond a simpler alternative. \n\n- Because training protocols for baselines are under-specified, the performance gap may partly reflect unequal access to external knowledge rather than genuine architectural superiority, casting doubt on fair comparison. \n\n- The optimisation scope of GECL, which is an important component of the proposed method, is ambiguous, making it hard to judge whether the claimed guideline-driven reasoning is robust or merely correlational.\n\n(Details are in the questions)\n\nQUESTIONS:\n- Figure 2 depicting the Dual Diagnostic Network appears asymmetric. The two branches seem to contain different amounts of information, with the guidance branch being inclusive/indicative of the label. It is unclear if this dual-branch design is strictly necessary, and an ablation study on a single-branch design would strengthen the argument for its necessity (though current study swaps to an MLP, not one-branch).\n\n- The training processes for other comparison methods are not clearly elaborated, making it difficult to ascertain the fairness of the comparison with existing methods. If RAD's improvements are primarily due to learning more external knowledge, which other methods might not have been exposed to, then the observed gains might stem largely from this additional knowledge rather than the pipeline novelty itself. If so, can RAG's knowledge injection strategy/framework be applied to other models—for example, BiomedCLIP?\n\n- Equation (7) is an important loss of RAD and is the base of the method. But it's not clear to me if the the text encoder, which is used to obtain embeddings for input text and guidelines, is optimized?  If the encoders are frozen, the GECL term can only shift the lightweight projection heads or prototypes, so its ability to “pull” patient text T toward guideline G seems limited.  Conversely, if the encoders are updated, both T and G are re-encoded each step. In this case, how to ensure the guideline embedding G still faithfully represent the original guideline semantics? \n\n \n- Providing a couple of complete examples of the crafted condition guidelines in the main paper (or a prominent appendix section) would greatly enhance reader understanding of what these guidelines look like and could help extend the research. The appendix shows the prompt template, not concrete examples.\n\n- While Figure 1 and Table 3 show that RAD pays more attention to representative laboratory indicators/keywords. The current evidence—higher guideline-recall scores and attention heat-maps—mainly shows correlation: GECL already pulls representations toward guideline tokens, so a larger recall is almost guaranteed even if the model’s final decision does not truly hinge on those tokens. To demonstrate that RAD’s predictions are causally driven by guideline evidence, consider adding a counter-factual test (e.g., mask or perturb key indicators at inference) to see if performance drops.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RAD, a Retrieval-Augmented Diagnosis framework that enhances multimodal clinical diagnosis by injecting refined disease-specific external knowledge (e.g., clinical guidelines, medical textbooks, research papers) into the diagnostic process. RAD uses a three-pronged approach: (1) retrieving and refining guideline knowledge, (2) embedding this knowledge in multimodal representations via a contrastive loss, and (3) integrating it through a dual diagnostic decoder. The authors also propose a dual-axis interpretability evaluation framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- RAD introduces a systematic method to inject disease-specific guidelines into the entire diagnostic pipeline, including input augmentation, feature extraction, and decision fusion, rather than limiting knowledge use to pretraining stages.\n- Across four anatomically diverse datasets (MIMIC-ICD53, FairVLMed, SkinCAP, NACC), RAD consistently outperforms strong baselines like BiomedCLIP and KAD in terms of F1, AUC, and sample-wise accuracy.\n\n\nWeaknesses\n1. The main novelty lies in the explicit combination and systematic integration of prior methods (retrieval, LLM summarization, contrastive alignment, and dual decoder); however, the technical advances in each are modest individually. There is limited comparison or differentiation from conventional retrieval-augmented generation or discriminative frameworks. The distinction from RAG approaches is only briefly justified.\n2. The approach assumes that a suitable disease guideline exists, that LLM summaries are reliable, and that retrieved content truly covers all relevant diagnostic edge cases. The refinement of retrieved documents via Qwen2.5-72B is central to RAD, yet the quality and consistency of LLM-generated summaries are not critically evaluated.\n3. While improved averages are shown, the paper does not analyze or visualize errors where adherence to retrieved guidelines leads to mistakes or misses patient-specific context.\n4. While interpretability evaluation is a valuable addition, the reliance on attention weights (text and image) as evidence of adherence is controversial.\n5. The pipeline for retrieving, refining, and updating guidelines is resource-intensive (uses Qwen2.5-72B), potentially limiting reproducibility in resource-constrained or privacy-sensitive settings. \n6. The paper benchmarks against baselines in multimodal medical AI, but omits RAG-style methods tailored to clinical reasoning or explanation. Explanation for specific baseline choices or additional comparisons would strengthen the empirical claim.\n7. Occasional jargon (“dual diagnostic transformer decoder”, “feature prototypes”), ambiguous notation in equations (Section 3.2.2), and dense referencing to appendices can break the narrative flow. \n8. While the manuscript emphasizes clinical workflows, an analysis of clinician utility, end-user validation, or prospective trials in realistic deployment environments is missing.\n\nQUESTIONS:\n1. How does RAD handle errors or outdated content in the knowledge corpus or LLM-generated summaries? Have you analyzed the effect of injecting incorrect or incomplete guidelines on downstream accuracy or interpretability? \n\n2. Attention weights are used to proxy adherence to diagnostic indicators. Have you considered augmenting your interpretability evaluation with external expert review or ground-truth decision rationales?\n\n3. Could you expand on the differences between your method and retrieval-augmented generation or multimodal RAG approaches?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Retrieval-Augmented Diagnosis (RAD), a framework designed to enhance credible multimodal clinical diagnosis. Current AI-driven medical research often relies on implicitly encoded knowledge, ignoring the needs of specific tasks. RAD solves this problem by explicitly injecting external medical knowledge through the following three key mechanisms: retrieving and optimizing disease-centric guidelines, applying a guideline-enhanced contrastive loss function to align features, and using a dual diagnosis decoder to guide cross-modal fusion. The paper also constructs the MIMIC-ICD53 dataset and conducts experiments on 4 datasets to verify the effect. The paper has sufficient work, but may need to be strengthened in theory.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. This paper systematically integrates multi-source medical knowledge into the diagnostic pipeline, addressing the limitation of implicit knowledge encoding in prior methods.\n\n2. The MIMIC-ICD53 dataset is constructed to provide a reliable testing benchmark.\n\n3. Extensive experiments are conducted on four datasets to effectively verify the model.\n\nWeaknesses\n\n1. No contribution in search strategy. Search strategy is the most important part of RAD, but this article is not innovative enough in search method and directly uses MedCPT as a search engine.\n\n2. Retrieval mainly focuses on text modality and lacks the utilization of multimodal information.\n\n3. The LLM-based knowledge refinement and dual decoder structure may introduce higher computational costs, though the authors note retrieval is label-level and manageable.\n\n4. The framework relies on a fixed knowledge base and lacks experimental effects of modifying the knowledge base or discussions on knowledge base updating algorithms.\n\n5. There is a lack of discussion on the correctness of the retrieved results. If the wrong report is retrieved, it may lead to further error accumulation. \n\n6. The class labels of MIMIC-ICD53 are too unbalanced, which may bring serious obstacles to model training.\n\n7. The data refinement process directly uses the 72B offline model and lacks manual verification, which may lead to data deviation.\n\nQUESTIONS:\n1.Introduce the innovation of the retrieval process and give an explanation from a theoretical level.\n\n2.Analyze cases where retrieved reports are incorrect.\n\n3.72B Offline models may introduce bias and should be supplemented with manual verification instructions or ablations of different large model refinements.\n\n4.The comparative experiments lack comparison results with large language models, such as Qwen-vl, llava-med, and huatuogpt-vision.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}