Venue: NAACL 2025

# Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical

Radiology Report Generation

Liwen Sun\({}^{*}\), James Zhao\({}^{*}\), Megan Han, Chenyan Xiong

School of Computer Science, Carnegie Mellon University

{liwens,jjzhao2,wenjingh,cx}@andrew.cmu.edu

###### Abstract

Multimodal foundation models hold significant potential for automating radiology report generation, thereby assisting clinicians in diagnosing cardiac diseases. However, generated reports often suffer from serious factual inaccuracy. In this paper, we introduce a fact-aware multimodal retrieval-augmented pipeline in generating accurate radiology reports (FactMM-RAG). We first leverage RadGraph to mine factual report pairs, then integrate factual knowledge to train a universal multimodal retriever. Given a radiology image, our retriever can identify high-quality reference reports to augment multimodal foundation models, thus enhancing the factual completeness and correctness of report generation. Experiments on two benchmark datasets show that our multimodal retriever outperforms state-of-the-art retrievals on both language generation and radiology-specific metrics, up to 6.5% and 2% score in F1CheXbert and F1RadGraph. Further analysis indicates that employing our factually-informed training strategy imposes an effective supervision signal, without relying on explicit diagnostic label guidance, and successfully propagates fact-aware capabilities from the multimodal retrieved to the multimodal foundation model in radiology report generation.1

Footnote 1: Our code is available at [https://github.com/cvcscum/FactMM-RAG](https://github.com/cvcscum/FactMM-RAG)

+
Footnote †: These authors contributed equally to this work.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 643–655

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Within hospitals worldwide, chest radiology serves as a critical technique in identifying cardiac diseases and abnormalities. Results of a chest radiograph are typically consolidated in a radiology report, including the source X-ray and a radiologist-produced findings section detailing clinical observations. Manually generating these reports, however, can be both time-consuming and potentially inaccessible in under-resourced hospitals [18, 7]. Recent multimodal foundation models have exhibited remarkable capabilities in challenging healthcare tasks, motivating an automation of this process to enhance physicians' efficiency on clinical decision-making and improve patient health outcomes [12, 13, 14, 15, 16, 17]. Although prior medical multimodal foundation models have demonstrated promising capabilities on report generation given the radiology image, they still suffer from serious hallucinations by generating factually inaccurate reports [14, 15, 16]. Factual correctness is especially critical in chest radiology domains, as minute textual differences can drastically invert radiology report meaning and downstream prescribed treatments [1, 16, 17]. Retrieval-Augmented Generation (MAG) has emerged as a popular paradigm to address this issue by grounding text generation with retrieved relevant knowledge given a query [14, 15, 16]. However, developing medical multimodal retrievers remains challenging, requiring retrievers to bridge the gap between symptomatic image semantics and factually-equivalent report text. To capture fine-grained details in chest radiographs and improve the factual completeness of generated reports, we introduce FactMM-RAG, a fact-aware multimodal retrieval-augmented pipeline for generating accurate radiology reports given a radiology image. By designing a novel report pair-mining procedure incorporating factual knowledge, we develop a fact-aware retriever to augment multimodal foundation models in generating accurate chest X-ray radiology reports. Specifically, we first leverage RadGraph [Jain

et al., 2021) to mine factually-oriented report pairs by annotating consistent radiology entities and relations between query and reference reports with certain abnormalities. Next, we train a universal multimodal encoding architecture through mined report pairs to conduct multimodal dense retrieval. Given an unseen patient's radiology image, our retriever encodes it and searches for the most similar factually-informed reference report from an available report corpus. Passing them together into a multimodal foundation model unlocks its fact-aware potential to generate more accurate radiology reports.

Our experiments reveal that our retriever outperforms all state-of-the-art retrievers in both language generation and clinically relevant metrics on the MIMIC-CXR and CheXpert datasets, achieving up to 6.5% and 2% score in F1CheXbert and F1RadGraph for final RAG evaluation. We also investigate our retriever's fact-aware capability controlled by factual similarity thresholds and confirm that our factually-informed training strategy can impose a useful supervision signal without relying on explicit diagnostic label guidance. Further analysis through retrieval evaluation metrics shows that the fact-aware capability of our retriever can be effectively propagated to the multimodal foundation models. Lastly, our case study highlights that among reports describing the same symptom from different retrievers, those generated by our model are more accurate and achieve greater factual correctness.

Our main contributions can be summarized as follows:

* We propose a fact-aware medical multimodal retriever to augment multimodal foundation models in generating accurate chest X-ray radiology reports.
* We design a method for mining factually-informed radiology report pairs that trains multimodal encoders to retrieve high-quality reference reports.
* We demonstrate that on two benchmark datasets, our medical multimodal retriever outperforms state-of-the-art medical multimodal retrieivers on both language generation and clinically relevant metrics. The rest of this paper is organized as follows. We review related work in in Section 2. We discuss the pipeline of FactMM-RAG in Sections 3. Section 4 and 5 discuss our experimental setup and results.

## 2 Related Work

Retrieval Augmented Generation.Retrieval Augmented Generation, utilizing external knowledge to enhance language models, has shown great promise in text-generation performance on factual accuracy especially for Open-Domain QA. (Borgeaud et al., 2022; Izacard et al., 2022). Guu et al. (2020); Lewis et al. (2021) involve end-to-end training through both generators and retrievers; Shi et al. (2023); Yu et al. (2023b) adapt the end-to-end pattern by employing black-box LLM training signal propagation for retriever tuning. Further works have expanded RAG to multiple modalities, employing unified image-text encoders (Radford et al., 2021) or separate pretrained encoders (Dosovitskiy et al., 2021; Raffel et al., 2023) and plugging retrieved documents into multimodal foundation models (Chen et al., 2022; Hu et al., 2023). Yasunaga et al. (2023) similarly integrates multimodal retrieval with both text and image generation capabilities.

Medical Multimodal Retriever.Joint training of image-text pairs in a shared embedding space, as exemplified by CLIP (Radford et al., 2021), facilitates visual and textual modality interactions, providing flexible representations for general-domain downstream tasks. Adapting general-domain multimodal retrievers to medical domains, however, is non-trivial due to the necessity of specialized knowledge. Zhang et al. (2022) introduces an unsupervised approach for radiology image representation learning from paired text descriptions. Huang et al. (2021) leverages global image-report and local sub-region features for multimodal retrieval and classification. Wang et al. (2022); You et al. (2023) propose medical knowledge extraction for constructing contrastive learning image-text pairs. Zhang et al. (2024) addresses the limited diversity within medical datasets, curating a large biomedical image-text collection towards a biomedical multimodal foundation model. Nevertheless, these existing medical multimodal retrievers neglect specific image information and do not adequately emphasize factual accuracy, resulting 

in imprecision when retrieving radiology reports.

Medical Multimodal Foundation ModelSignificant efforts have been made in applying multimodal foundation models to the medical imaging domain (Li et al., 2023; Moor et al., 2023; Tu et al., 2023; Sun et al., 2024). As chest X-ray radiology is the most commonly performed imaging examination, tailored medical multimodal foundation models for this critical area has gathered much attention (Chambon et al., 2022; Chen et al., 2021; Thawkar et al., 2023; Wu et al., 2023; Chen et al., 2024). Jain et al. (2021) advances this area by designing a novel information extraction schema to structure radiology reports from chest radiographs; Miura et al. (2021); Delbrouck et al. (2022) take a step forward, using reinforcement learning from semantic rewards to improve the factual quality of generated radiology reports; Chen et al. (2024) recently has also developed an instruction-tuned multimodal foundation model capable of sophisticated interpretation and analysis of chest X-rays.

One closely related line of work to ours is retrieval-based radiology report generation given only radiology images. For instance, Li et al. (2018) proposes a retrieval policy module to update radiology reports via hierarchical reinforcement learning; Endo et al. (2021) employs image-text embeddings from contrastive learning for retrieval-augmented radiology report generation; Ramesh et al. (2022) proposes synthesizing additional reports and reducing hallucinations from reference report priors to improve medical radiological report generation.

## 3 Methodology

In this section, we present the overall methodology of FactMM-RAG. We first detail the training procedure of our fact-aware medical multimodal retriever in Section 3.1. We then provide the pipeline for retrieval-augmented radiology report generation with our multimodal retriever in section 3.2. The overview is illustrated in Figure 1.

### Fact-aware Multimodal Retrieval

This section discusses the training process of the multimodal retriever with factual knowledge. Each patient in the corpus has a chest X-ray radiology image along with its corresponding report. We begin by annotating each report using RadGraph (Jain et al., 2021), then constructing factual report pairs to train our multimodal retriever. We describe these steps as follows.

**Chest Radiograph Annotation.** Since radiology reports are free-text, we utilize the RadGraph information extraction tool to extract structured knowledge graphs from them. Specifically, RadGraph employs named entity recognition and relation extraction models to identify radiolog

Figure 1: An overview of the FactMM-RAG system. It mainly contains three stages: (1) Leveraging RadGraph to characterize each radiology report and mine factually-informed report pairs; (2) Integrating factual knowledge into the training of the universal multimodal retriever; (3) Given the radiology image, employing the fact-aware multimodal retriever to search for factually-informed reference reports and augmenting the multimodal foundation model in generating accurate radiology reports.

 

ical entities (e.g. carina, lungs, abnormalities) and the clinical relations between them (e.g. modify, located at, suggestive of). Each radiology report is then segmented into distinct regions and stored as \([{\ttentity}_{1},{\ttentity~{}label}_{1},{\ttrelation}_{1}),({\ttentity}_{2},\) entity label\({}_{2},{\ttrelation}_{2}),\ldots]\). After characterizing the chest radiograph for each report in the training corpus, we construct factual report pairs.

Factual Report Pairs Mining.Each report has an associated medical label describing the symptom. We first utilize the query report to search for other reports with the same symptom, aiming to eliminate false negatives when constructing report pairs. Rather than solely relying on the diagnostic labels, we further capture the factually-oriented pathology semantics between different reports. Following F1RadGraph Jain et al. (2021), we calculate the factual similarity \(s(q_{txt},d_{txt})\) between query report \(q_{txt}\) and other reports \(d_{txt}\) in the annotated format as follows,

\[s(q_{txt},d_{txt})=\frac{2\cdot(\hat{q}_{txt}\cap\hat{d}_{txt})}{{\tt length }(\hat{q}_{txt})+{\tt length}(\hat{d}_{txt})}, \tag{1}\]

where \(\hat{q}_{txt},\hat{d}_{txt}\) denotes reports with only annotated entities and relations in RadGraph structured form. We then set a strict threshold \(\delta\) to filter out searched reports with low similarity score:

\[N_{q_{txt}}=\{d_{txt}\in D|s(q_{txt},d_{txt})>\delta\}. \tag{2}\]

where \(N_{q_{txt}}\) denotes factual positive report pairs for \(q_{txt}\) and \(D\) is the total training corpus. Since each query report is associated with a corresponding radiology image, these factual report pairs can also be applied to the query report's radiology image. Next, we train our multimodal retriever with mined factual report pairs.

Multimodal Dense Retrieval.Following previous work Zhou et al. (2024), we universally encode each query image \(q_{img}\) and other image-text pair pairs \((d_{txt},d_{img})\) in the training corpus, using one encoder, MARVEL:

\[\mathbf{q} =\text{MARVEL}(q_{img}); \tag{3}\] \[\mathbf{d} =\text{MARVEL}(d_{txt},d_{img}), \tag{4}\]

where each image-text pair is represented as a single embedding. We then model the relevance score \(f(q,d)\) between the query image and other image-text pairs by cosine similarity:

\[f(q,d)=\cos(\mathbf{q},\mathbf{d}). \tag{5}\]

To inject factually-oriented medical knowledge into multimodal retrieval, we train the encoder to minimize the following loss,

\[\small\mathcal{L}=-\sum\limits_{q_{img}\in D}\sum\limits_{d^{+}\in N_{img}} \log\frac{e^{f(\mathbf{q},\mathbf{d}^{+})/r}}{e^{f(\mathbf{q},\mathbf{d}^{+} )/r}+\sum_{\mathbf{d}^{-}}e^{f(\mathbf{q},\mathbf{d}^{-})/r}}, \tag{6}\]

where \(d^{+}\) are obtained through factual report pair mining and \(d^{-}\) are in-batch negative samples Karpukhin et al. (2020). Then, we use our multimodal retriever and foundation model to perform retrieval-augmented radiology finding generation.

### Retrieval Augmentation for Accurate Radiology Report Generation

Given our trained fact-aware multimodal retriever, we encode the query image and each report in the training corpus. Then, we retrieve the report with the highest relevance score to the query image as the factually-informed relevant report. Subsequently, we pass the query image along with the relevant report into a multimodal foundation model to perform retrieval-augmented generation training. The multimodal foundation model is finetuned by standard autogressive loss,

\[\small\mathcal{L}=-\frac{1}{n}\log\prod_{i}^{n}p_{\theta}(y_{i}|q_{img},d_{txt }^{*},x_{\texttt{instr}},y_{<i}), \tag{7}\]

where \(q_{img}\) is the query image, \(d_{txt}^{*}\) is the retrieved factually-informed relevant report, \(x_{\texttt{instr}}\) is the prompt instruction, and \(y\) is the ground-truth report. During inference, we retrieve a relevant report from the training corpus using an unseen patient X-ray image, and pass them into the multimodal foundation model to generate findings with higher factual accuracy.

## 4 Experimental Setup

Dataset.Following Delbrouck et al. (2023), we use the processed MIMIC-CXR Johnson et al. (2019) to train both retriever and foundation model. This dataset contains 125,417 training radiology image-report pairs, 991 validation pairs, and 1,624 test pairs. They are sourced from the Beth Israel Deaconess Medical Center. CheXpert Irvin et al. (2019) is another chest X-ray dataset from Stanford Health Care. Since it contains complete finding reports only for a testing dataset containing 1000 pairs, we use it as zero-shot evaluation.

Evaluation Metrics.We evaluate our proposed system using both natural language 

generation and medically-tailored evaluation metrics. For language fluency measures, we use ROUGE-L Lin (2004) to evaluate the longest common subsequence overlap between the generated and reference findings, and BERTScore Zhang et al. (2020) to evaluate non-clinical semantic sentence similarity.

For clinical accuracy measures, we use CheXbert Smit et al. (2020) to generate the ground-truth diagnostic labels for finding reports. Following Delbrouck et al. (2023), we then calculate the F1CheXbert Zhang et al. (2020), which is the F1-score for 5 observations Cardiomegaly, Edema, Consolidation, Atelectasis, Pleural Effusion) by comparing the generated report with the reference report's classifications. Beyond using the limited diagnostic labels for evaluation, we also adopt F1RadGraph Jain et al. (2021) to measure factual correctness by calculating the overlap in radiological entities and clinical relations between the generated report and the reference report. See Appendix A.3 for more details.

**Baselines.** We mainly compare our retriever with other baselines under multimodal RAG setting. We include the following baselines, CLIP Radford et al. (2021) is a multimodal retriever pretrained from general-domain image-text pairs; GLoRIA Huang et al. (2021) leverages attention-weighted image regions with contextual words to learn localized and global representations for radiology images and reports; MedCLIP Wang et al. (2022) and CXR-CLIP You et al. (2023) build upon CLIP and utilize diagnostic labels as training signals for learning radiology image and text representations; BiomedCLIP Zhang et al. (2024) extends the radiology-specific dataset and pretrains on a larger magnitude of biomedical data to learn multimodal representations; Med-MARVEL utilizes universal encoder MARVEL Zhou et al. (2024) to conduct contrastive learning on each patient's self image-report pair without further training on factual image-report pairs.

We also compare our method with non-RAG approach. "No Retriever" refers to directly fine-tuning the backbone to generate reports without retrieval augmentation; ORGan Hou et al. (2023) first creates an observation plan, then feeds the plan and radiographs to generate the report through tree reasoning mechanism. Upper-bound results using an oracle in training corpus with top-1 factual similarity to test query report are presented. More details are in Appendix A.3.

**Implementation Details.** In our experiments, we use MARVEL Zhou et al. (2024) as our multi-modal retriever backbone. MARVEL is a language model based on T5-ANCE Yu et al. (2023), trained with modality-balanced hard negatives. We use LLaVA Liu et al. (2023) as our multimodal foundation model backbone. Since each radiology study contains multiple image views for each patient, we select the frontal view. We also concatenate the finding and impression sections to form the X-ray report. To reduce training costs and address factual report pair imbalances, we rerank the retrieved reports by factual similarity and use the top 2 factual report pairs for each query to train our multimodal retriever. We leave more training details in Appendix A.1 and A.2.

## 5 Evaluation Results

In this section, we present our experimental results. We first evaluate the overall performance between different retireivers in section 5.1. Next, we discuss the ablation studies in section 5.2. We then explore the fact-aware capability of our retriever in section 5.3 and section 5.4. Lastly, we show the superiority of our retriever through a case study in section 5.5.

### Overall Performance

The results of our fact-aware RAG system are shown in Table 1. In MIMIC-CXR, FactMM-RAG outperforms state-of-the-art retrievers by a significant margin, up to 6.5% in F1CheXpert and 2% in F1RadGraph. In the CheXpert zero-shot evaluation, FactMM-RAG outperforms state-of-the-art retrievers by 2% and 1.2% in these two metrics, indicating our retriever's generalization capability compared to other models.

To establish the effectiveness of our RAG approach, we also show that FactMM-RAG significantly outperforms the fine-tuned backbone without retrieval augmentation by 10% and achieves competitive improvement over the strong non-RAG ORGan baseline.

Besides, we can observe that adopting the baseline retrievers on top of multimodal foundation models only yields marginal gains compared to 

the finetuning of foundation model generation without retrieval-augmentation. This shows that reports retrieved by baseline retrievers are factually-inferior to those from our retriever, potentially passing misleading information that prevents the foundation model from generating factual reports.

Specifically, compared to the retriever MedMARVEL, we also observe factual-correctness performance gain based on two clinical metrics. Both use the same universal encoder backbone, but FactMM-RAG benefits from the injected factual medical knowledge, allowing it to search for the most similar and factually correct reports, thereby assisting the multimodal foundation model in generating more accurate reports.

### Ablation Study

Multimodal RetrievalInstead of relying on the multimodal foundation model to generate reports, we also evaluate the performance of the multimodal retrievers by directly encoding radiology images from the testing corpus and searching for the closest report from the training corpus for comparison with ground-truth reports. Table 2 shows that our retriever also achieves the best factual retrieval performance compared to other baselines under this setting across two datasets. This demonstrates that training the multimodal retriever with mined factually-informed report pairs can enhance its radiology image understanding capabilities and directly align it with precise reports.

Backbone VariationWe also investigate the impact of different retriever and foundation model backbones on radiology report generation in Table 2. We initialize our retriever model from two checkpoints: WebQA and ClueWeb in Zhou et al. (2024). We observe that the ClueWeb checkpoint provides a marginal gain compared to the WebQA checkpoint. This can be attributed to the larger scale of the ClueWeb dataset used for pretraining. We also utilize Med-MARVEL as our retriever backbone, which exhibits similar performance to other backbones after training. This implies that even if our retriever is initialized with a backbone from a general domain, our factually-informed training strategy enables it to

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**MNHC-CXR**} & \multicolumn{3}{c}{**CheXpert**} \\ \cline{2-10}
**Model** & \multicolumn{2}{c}{**Factual Similarity**} & \multicolumn{2}{c}{**Textual Similarity**} & \multicolumn{2}{c}{**Factual Similarity**} & \multicolumn{2}{c}{**Textual Similarity**} \\ \cline{2-10}  & F1CheXbert & F1RadGraph & ROUGE-L & BERTScore & F1CheXbert & F1RadGraph & ROUGE-L & BERTScore \\ \hline No Retriever & 0.496 & 0.234 & 0.294 & 0.549 & 0.371 & 0.173 & 0.231 & 0.469 \\ ORgan Hou et al. (2023) & 0.541 & 0.240 & 0.308 & 0.552 & 0.431 & 0.181 & 0.232 & 0.470 \\ \hline CLIP Radford et al. (2021) & 0.507 & 0.241 & 0.300 & 0.552 & 0.381 & 0.172 & 0.231 & 0.468 \\ GLBLA Huang et al. (2021) & 0.476 & 0.232 & 0.294 & 0.543 & 0.397 & 0.173 & 0.231 & 0.468 \\ MoCLIP Wang et al. (2022) & 0.517 & 0.238 & 0.298 & 0.549 & 0.408 & 0.182 & 0.238 & 0.471 \\ CXR-CLIP You et al. (2023) & 0.501 & 0.243 & 0.302 & 0.553 & 0.406 & 0.183 & 0.241 & 0.471 \\ BiomedCLP Zhang et al. (2024) & 0.502 & 0.233 & 0.293 & 0.546 & 0.380 & 0.173 & 0.232 & 0.469 \\ \hline Med-MARVEL Zhou et al. (2024) & 0.537 & 0.237 & 0.306 & 0.549 & 0.454 & 0.185 & **0.243** & 0.472 \\ FactMM-RAGA & **0.602** & **0.257** & **0.307** & **0.561** & **0.475** & **0.185** & 0.236 & **0.475** \\ \hline Oracle & 0.972 & 0.523 & 0.495 & 0.677 & 0.951 & 0.384 & 0.350 & 0.548 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Overall performance of FactMM-RAG and baselines under the multimodal retrieval-augmentation setting. Models are evaluated by textual similarity and factual similarity between generated and reference reports. FactMM-RAG outperforms the best baseline with p-value ¡ 0.05.

Figure 2: Factual performance of FactMM-RAG controlled by different F1CheXbert and F1RadGraph thresholds. We vary the F1RadGraph thresholds under one fixed F1CheXbert threshold selected from {0, 0.4, 0.8, 1}.

 

fully leverage medical knowledge and quickly adapt to the radiology-specific domain without degrading performance.

### Fact-aware Capability Control

The factual similarity threshold in Equation 1 plays a critical role in controlling the fact-awareness of our multimodal retriever. We examine the performance of FactMM-RAG under different thresholds, as shown in Figure 2. Not only utilizing F1RadGraph thresholds, we also employ F1CheXbert to curate additional thresholds from the report's diagnostic labels to mine report pairs.

Under the same F1CheXbert threshold for mining report pairs, we observe that an increase in the F1RadGraph threshold correlates with an improvement in factual performance. However, adopting stricter thresholds for identifying report pairs does not yield further improvements and reaches saturation. After calculating the average number of report pairs per query, we find that high thresholds can exclude many relevant report pairs, as shown in Figure 3. This exclusion results in the potential loss of factually useful pairs, thereby hindering the training of our multimodal retriever driven by additional factual medical knowledge.

Rather than relying on diagnostic labels from CheXbert to identify high-quality report pairs, Figure (a)a demonstrates that the F1RadGraph threshold alone can also effectively mine factual report pairs for training our multimodal retriever. As the F1RadGraph threshold increases, FactMM-RAG even matches the performance under high threshold settings in Figure (d)d. This signifies that employing our training strategy with curated factual query-report pairs still imposes useful supervision signals without relying on explicit diagnostic label guidance.

### Fact-aware Capability Propagation

To further understand the benefits of our retriever for the foundation model, we explore the effective propagation of fact-aware capabilities from

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**MMIHC-CXR**} & \multicolumn{4}{c}{**CheXpert**} \\ \cline{2-9}
**Model** & \multicolumn{2}{c}{**Factual Similarity**} & \multicolumn{2}{c}{**Textual Similarity**} & \multicolumn{2}{c}{**Factual Similarity**} & \multicolumn{2}{c}{**Textual Similarity**} \\ \cline{2-9}  & F1CheXbert & F1RadGraph & ROUGE-L & BERTScore & F1CheXbert & F1RadGraph & ROUGE-L & BERTScore \\ \hline
**Setting: Multimodal Retrieval** & & & & & & & & \\ \hline CLIP Radford et al. (2021) & 0.341 & 0.160 & 0.238 & 0.489 & 0.285 & 0.130 & 0.207 & 0.439 \\ GLoRL Huang et al. (2021) & 0.346 & 0.137 & 0.211 & 0.453 & 0.359 & 0.135 & 0.216 & 0.447 \\ MockLP Wang et al. (2022) & 0.539 & 0.198 & 0.261 & 0.508 & 0.478 & 0.161 & 0.225 & 0.454 \\ CXK-CLIP You et al. (2023) & 0.516 & 0.215 & 0.277 & 0.524 & 0.444 & 0.167 & 0.230 & 0.458 \\ BiomedCLip Zhang et al. (2024) & 0.502 & 0.233 & 0.293 & 0.546 & 0.386 & 0.142 & 0.216 & 0.441 \\ Med-MARVEL Zhou et al. (2024) & 0.550 & 0.212 & 0.279 & 0.525 & 0.479 & 0.160 & 0.222 & 0.454 \\ FactMM-RAG & **0.605** & **0.249** & **0.297** & **0.547** & **0.491** & **0.174** & **0.237** & **0.467** \\ \hline Oracle & 0.992 & 0.429 & 0.399 & 0.612 & 0.999 & 0.438 & 0.362 & 0.554 \\ \hline
**Setting: Multimodal Retrieval Augmented Generation** & & & & & & & & \\ \hline ClueWeb-LLAVA\({}_{1.5}\) & **0.602** & 0.257 & 0.307 & 0.561 & **0.495** & 0.180 & **0.239** & 0.473 \\ WebQA-ILAvA\({}_{1.5}\) & 0.572 & **0.262** & 0.304 & 0.562 & 0.456 & 0.184 & 0.237 & **0.474** \\ Med-MARVEL-LIAVA\({}_{1.5}\) & 0.581 & 0.260 & **0.311** & **0.563** & 0.475 & **0.185** & 0.236 & **0.474** \\ CueWeb-LLAVA\({}_{1.6}\) & 0.601 & 0.252 & 0.303 & 0.558 & 0.492 & 0.178 & 0.237 & 0.471 \\ \hline \end{tabular}
\end{table}
Table 2: Ablation study of FactMM-RAG including multimodal retrieval and backbone variation.

Figure 4: Analysis of fact-aware capability propagation. The \(x\)-axis MRR measures the retriever’s performance on retrieving factually relevant reports.

Figure 3: Retrieval evaluation of FactMM-RAG with different F1CheXbert and F1RadGraph thresholds. MRR calculates the mean reciprocal of rank at which the first relevant report that meets two factual similarity thresholds with query report is retrieved.

 

the retriever to the foundation model. To demonstrate this behavior, we use the mined factual report pairs as reference reports for the query report. We then use the retrieval metric Mean Reciprocal Rank (MRR) as an intermediate evaluation, shown in Figure 4. From the plot, we observe that as training progresses, the retrieval metric increases alongside two clinical metrics. This factually-oriented upward trend in our retriever's performance in Figure 3(a) is also reflected in the foundation model's performance in Figure 3(b). This indicates that employing a factually-informed reference report selection strategy to train our multimodal retriever can also enhance the foundation model's ability to generate factually accurate radiology reports.

### Case Study

In this section, we present two examples from MIMIC-CXR to qualitatively analyze our retriever's fact-aware capability, as illustrated in Table 3. In the first example, we observe that FactMM-RAG provides symptom observations consistent with the ground-truth report and generates more accurate factual details compared to MedMARVEL, e.g., "post median steronomy, atelectasis, not pneumothorax"; In the second example, we further observe that although both retrievers generate reports with diagnostic labels matching the ground-truth report, FactMM-RAG provides additional details compared to Med-MARVEL, such as "pulmonary vasculature is normal, no acute osseous abnormalities". These characteristics confirm that adopting our fact-aware retriever can assist multimodal foundation models in generating more accurate radiology reports. We show retrieved reports from two samples in Appendix A.4.

## 6 Conclusion

In this paper, we aim at improving radiology report generation by introducing a fact-informed medical multimodal retriever for retrieval-augmented generation. In particular, we utilize RadGraph to annotate chest radiograph reports and mine clinically-relevant pairs. We integrate factual information into a universal multimodal retriever, presenting FactMM-RAG, a fact-aware multimodal retrieval-augmented radiology report generation pipeline. FactMM-RAG outperforms all state-of-the-art retrievals evaluated by factual correctness and textual coherence for final report generation in MIMIC

\begin{table}
\begin{tabular}{c|l|l|l} \hline \multicolumn{1}{c|}{**Radiology Image**} & \multicolumn{2}{c|}{**Med-MARVEL**} & \multicolumn{1}{c|}{**FactMM-RAG**} \\ \cline{2-5}  & Single portable view of the chest. There are bilateral pleural effluents, moderate on the left and small on the right. _There is also pulmonary vascular redistribution and they alveolar influence between children is enlarged but unchanged. Median stemotomy wires and mediastinal clips are again need._ & \multirow{7}{*}{\begin{tabular}{} \end{tabular} } & The patient is testing post median secretary and CABG. Left-sided dual-chamber pacemaker is noted with leads medium size. The patients in statatum and media streamology and CABG. The terminating in right atrium around and right ventricle, unchanged. Crudomegaly is mild pulmonary edema. A small left pleural effusion is omitted when it is employed. There is an important privacy edema. slightly improves compared to the prior exam. Small layering bilateral pleural effects also may be slightly decreased in the interval. _Industrial attribute operations likely reflect selectleats. There is no pneumothorax_. No acute osseous abnormalities are visualized. \\ \hline F1RaGraph & 0.218 & 0.413 & Cardomengaly, Edema, Actetaxis, Pretail Effusion \\ \hline \multirow{9}{*}{\begin{tabular}{} \end{tabular} } & _The heart is mildly enlarged. The aorta is mildly torsone._ The mediastinal and blitz contours appear unchanged. _There is no pleural effusion or pneumothorax._ & Heart is mildly enlarged. The aorta is tortuous. Medianstal and blur continues. Mediansintal and blur continues over abbreviated magnitude. _To name eye sensitive to normal. Linear areas._ Streaky left basilar opacity suggests minor alterations. _There is no default potential effusion or pneumothorax._ The bones apparent denineralized. There is misild-to-moderate adjuvant convex curvature centered along the mid thoracic spine. & \multirow{3}{*}{\begin{tabular}{} \end{tabular} } & Moderate enlargement of the cardiac silhouette with a left ventricular presence is unchanged. The aorta remains retrovancy’ technique; and the blilt contours are stable. _Pulmonary vascularators is not required_. Three is minimal artefacts within the lung bases, but do focal consultation is prevent. _No pleural effusion or pneumothorax is identified. There are no acute osseous abnormalities._ & \multirow{3}{*}{
\begin{tabular}{} \end{tabular} } \\ \cline{2-2}  & & & \\ \cline 

CXR and CheXpert datasets. We further confirm the benefit of our multimodal retriever from the analysis of its fact-aware capability.

## 7 Limitations

Despite the strong performance of our FactMMRAG pipeline, we acknowledge potential limitations of our proposed method. In particular, our work only emphasizes chest radiology domains. It also worth exploring our retrieval-augmented factual report generation pipeline in broader medical domains, such as brain scan or histology datasets.

Another concern lies in the chosen evaluation metrics, F1RadGraph and F1CheXbert. F1CheXbert reflects high-level observational accuracy, while F1RadGraph assesses the correctness of radiology entities and clinical relationships. However, other radiologically-specific metrics, such as report conciseness and clarity, should also be considered (Sureka et al., 2014). Ideally, we should incorporate methods of evaluation directly aligned with human evaluations or involve domain expertise itself in our pair-mining and final evaluation procedure. Moreover, it is worth performing a long-tail evaluation by leveraging more fine-grained ground-truth label annotations (Holste et al., 2023).

## Ethics Considerations

A key ethical consideration in our work is the use of de-identified, credentialed medical data. In particular, the responsible usage policy of the MIMIC-CXR dataset strictly prohibits sharing access with third parties. To gain access to MIMIC-CXR, we completed a training course and signed a data use agreement, ensuring compliance with patient privacy regulations.

## References

* A. Ahmad, I. Yaramis, and T. Bovy (2023)Creating trustworthy llms: dealing with hallucinations in healthcare ai. PreprintarXiv:2311.01463. Cited by: SS5.
* S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Bae, E. Elsen, and L. Sifre (2022)Improving language models by retrieving from trillions of tokens. PreprintarXiv:2112.04426. External Links: Link Cited by: SS5.
* P. Chambon, C. Bluethengen, J. Delbrouck, R. Van der Sluijs, M. Polacin, J. Manuel Zambrano Chaves, T. Mathew Abraham, S. Purohit, C. P. Langlotz, and A. Chaudhari (2022)Roentgen: vision-language foundation model for chest x-ray generation. PreprintarXiv:2211.12737. External Links: Link Cited by: SS5.
* W. Chen, H. Hu, X. Chen, P. Vega, and W. W. Cohen (2022)MURAS: multi-modal retrieval-augmented generator for open question answering over images and text. PreprintarXiv:2210.02928. External Links: Link Cited by: SS5.
* Z. Chen, Y. Shen, Y. Song, and X. Wan (2021)Cross-modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11h International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 5904-5914. External Links: Link Cited by: SS5.
* Z. Chen, M. Varma, J. Delbrouck, M. Paganini, Q. Le, N. J. Penaleni, L. Blankemeier, D. Van Veen, J. M. Valanarsa, A. Youssef, J. P. Cohen, E. Pontes Reis, E. B. Tsai, A. Johnston, C. Olsen, T. Mathew Abraham, S. Gatidis, A. S. Chaudhari, and C. Langlotz (2024)CREACTENT: towards a foundation model for chest x-ray interpretation. PreprintarXiv:2401.12208. External Links: Link Cited by: SS5.
* J. Delbrouck, P. Chambon, C. Bluethengen, E. Tsai, O. Almusa, and C. Langlotz (2022)Improving the factual correctness of radiology report generation with semantic rewards. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, pp. 4348-4360. External Links: Link Cited by: SS5.
* J. Delbrouck, M. Varma, P. Chambon, and C. Langlotz (2023)Overview of the radsum23 shared task on multi-modal and multianatomical Radiology Report Summarization. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Toronto, Canada, pp. 478-482. External Links: Link Cited by: SS5.
* A. Dosovitskiy, L. Beyer, A. Kolensnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021)An image is worth 16x16 words: transformers for image recognition at scale. PreprintarXiv:2010.11929 

Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y. Ng, and Pranav Rajpurkar. 2021. Retrieval-based chest x-ray report generation using a pre-trained contrastive language-image model. In _Proceedings of Machine Learning for Health_, volume 158 of _Proceedings of Machine Learning Research_, pages 209-219. PMLR.
* Gao et al. (2024) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. _Preprint_, arXiv:2312.10997.
* Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval-augmented language model pre-training. _Preprint_, arXiv:2002.08909.
* Holste et al. (2023) Gregory Holste, Song Wang, Ajay Jaiswal, Yuzhe Yang, Mingquan Lin, Yifan Peng, and Atlas Wang. 2023. CXR-LT: Multi-Label Long-Tailed Classification on Chest X-Rays (version 1.1.0).
* Hou et al. (2023) Wenjun Hou, Kaishua Xu, Yi Cheng, Wenjie Li, and Jiang Liu. 2023. Organ: Observation-guided radiology report generation via tree reasoning. _Preprint_, arXiv:2306.06466.
* Hu et al. (2023) Ziniu Hu, Ahmet Iscen, Chen Sun, Zurui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, and Alireza Fathi. 2023. Reveal: Retrieval-augmented visual-language pre-training with multisource multimodal knowledge memory. _Preprint_, arXiv:2212.05221.
* Huang et al. (2021) Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. 2021. GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-Efficient Medical Image Recognition. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 3942-3951.
* Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciaurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. CheXpert: A Large Chest radiograph Dataset with Uncertainty Labels and Expert Comparison. _Preprint_, arXiv:1901.07031.
* Iyeke et al. (2022) Lucky Iyeke, Rebecca Moss, Rachel Hall, Jun Wang, Lovleen Sandhu, Benjamin Appold, Ella Kalontar, Dimitra Menoudakos, Mahesh Rammarine, Samuel P. LaVine, Seungwoo Ahn, and Michelle Richman. 2022. Reducing unnecessary 'admission' chest x-rays: An initiative to minimize low-value care. _Curreus_, 14(10):e29817.
* Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Atlas: Few-shot learning with retrieval augmented language models. _Preprint_, arXiv:2208.03299.
* Jain et al. (2021) Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar. 2021. RadGraph: Extracting Clinical Entities and Relations from Radiology Reports. _Preprint_, arXiv:2106.14463.
* Johnson et al. (2019) Alistair E. W. Johnson, Tom J. Pollard, Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark, and Steven Horng. 2019. MIMIC-CCR, a de-identified publicly available database of chest radiographs with free-text reports. _Scientific Data_, 6(1).
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. _Preprint_, arXiv:2004.04906.
* Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. _Preprint_, arXiv:2005.11401.
* Li et al. (2018) Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2018. Hybrid retrieval-generation reinforced agent for medical image report generation. _Preprint_, arXiv:1805.08298.
* Li et al. (2023) Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. LIax-med: Training a large language-and-vision assistant for biomedicine in one day. _arXiv preprint arXiv:2306.00890_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Annual Meeting of the Association for Computational Linguistics_.
* Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. _Preprint_, arXiv:2304.08485.
* Liu et al. (2024) Kang Liu, Zhutoqi Ma, Mengmeng Liu, Zhicheng Jiao, Xiaolu Kang, Qiqiang Miao, and Kun Xie. 2024. Factual serialization enhancement: A key innovation for chest x-ray report generation. _Preprint_, arXiv:2405.09586.
* Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. _Preprint_, arXiv:1711.05101.
* Miura et al. (2021) Yasuhide Miura, Yuhao Zhang, Emily Bao Tsai, Curtis P. Langlotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency of image-to-text radiology report generation. _Preprint_, arXiv:2010.10042 

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flanimogo: a multimodal medical few-shot learner. In _Proceedings of the 3rd Machine Learning for Health Symposium_, volume 225 of _Proceedings of Machine Learning Research_, pages 353-367. PMLR.
* Pal and Sankarasubbu (2024) Ankit Pal and Malaikannan Sankarasubbu. 2024. Gemini goes to med school: Exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations. _Preprint_, arXiv:2402.07023.
* Pal et al. (2023) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. _Preprint_, arXiv:2307.15343.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sasatry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. _Preprint_, arXiv:2103.00020.
* Raffel et al. (2023) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. _Preprint_, arXiv:1910.10683.
* Ramesh et al. (2022) Vignav Ramesh, Nathan Andrew Chi, and Pranav Rajpurkar. 2022. Improving radiology report generation systems by removing hallucinated references to non-existent priors. _Preprint_, arXiv:2210.06340.
* Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Mingoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tauu Yih. 2023. Replug: Retrievalaugmented black-box language models. _Preprint_, arXiv:2301.12652.
* Smit et al. (2020) Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y. Ng, and Matthew P. Lungren. 2020. Chexbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using bert. _Preprint_, arXiv:2004.09167.
* Spets et al. (2006) Annemiek M. Spets, Yolanda van der Graaf, Arno W. Hoes, Sjouke Kalminj, Arnold P. Sachs, Matthiy J. Rutten, Jan W. Gratama, Annemiek D. Montautabn van Swijndreget, and Willem P. Mali. 2006. Chest radiography in general practice: Indications, diagnostic yield, and consequences for patient management. _The British Journal of General Practice: The Journal of the Royal College of General Practitioners_, 56(529):574-578.
* Sun et al. (2024)Liwen Sun, Abhineeit Agarwal, Aaron Kornblith, Bin Yu, and Chenyan Xiong. 2024. Ed-copilot: Reduce emergency department wait time with language model diagnostic assistance. _Preprint_, arXiv:2402.13448.
* Sureka et al. (2014) Binit Sureka et al. 2014. Seven c's of effective radiology reporting. _The Journal of National Accreditation Board for Hospitals & Healthcare Providers_, 1(1):17. Accessed 14 June 2024.
* Thawkar et al. (2023) Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xrgygpt: chest radiographs summarization using large medical vision-language models. _arXiv: 2306.07971_.
* Tu et al. (2023) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, Basil Mustafa, Aakanksha Chowdhery, Yun Liu, Simon Kornbluth, David Fleet, Philip Mansfield, Substant Praksah, Rence Wong, Sunny Virmani, Christopher Semnturs, S Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards generalist biomedical ai. _Preprint_, arXiv:2307.14334.
* Wang et al. (2022)Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. 2022. Medclip: Contrastive learning from unpaired medical images and text. _Preprint_, arXiv:2210.10163.
* Wu et al. (2023) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Towards generalist foundation model for radiology by leveraging web-scale 2dd&3d medical data. _Preprint_, arXiv:2308.02463.
* Xie et al. (2023) Qianqian Xie, Jiayu Zhou, Yifan Peng, and Fei Wang. 2023. Factreranker: Fact-guided reranker for faithful radiology report summarization. _Preprint_, arXiv:2303.08335.
* Yasunaga et al. (2023) Michihiro Yasunaga, Armen Aghajjanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tauu Yih. 2023. Retrieval-augmented multimodal language modeling. _Preprint_, arXiv:2211.12561.
* MICCAI 2023_, pages 101-111. Springer Nature Switzerland.
* Yu et al. (2023a) Shi Yu, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2023a. Openmatch-v2: An all-in-one multimodality plm-based information retrieval toolkit. pages 3160-3164.
* Yu et al. (2023b) Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023b. Augmentation-adapted retriever improves generalization of language models as generic plug-in. _Preprint_, arXiv:2305.17331 

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tuplin, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Matthew P. Lungren, Tristan Neumann, Sheng Wang, and Hoifung Poon. 2024. Biomedical clip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. _Preprint_, arXiv:2303.00915.
* Zhang et al. (2020a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with bert. _Preprint_, arXiv:1904.09675.
* Zhang et al. (2022) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D. Manning, and Curtis P. Langlotz. 2022. Contrastive Learning of Medical Visual Representations from Paired Images and Text. _Preprint_, arXiv:2010.00747.
* Zhang et al. (2020b) Yuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning, and Curtis P. Langlotz. 2020b. Optimizing the factual correctness of a summary: A study of summarizing radiology reports. _Preprint_, arXiv:1911.02541.
* Zhou et al. (2024) Tianshuo Zhou, Sen Mei, Ximee Li, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, Yu Gu, and Ge Yu. 2024. MARVEL: Unlocking the Multi-Modal Capability of Dense Retrieval via Visual Module Plugin. _Preprint_, arXiv:2310.14037.
* Call et al. (2021) Erdi Call, Ecem Sogancioglu, Bram van Ginneken,icky G. van Leeuwen, and Keelin Murphy. 2021. Deep learning for chest x-ray analysis: A survey. _Medical Image Analysis_, 72:102125.

## Appendix A Appendix

### Retriever Training Procedure

To training our fact-aware multimodal retriever, we not only use mined factual report pairs as positive reports to the query image, but also incorporate the query image's corresponding report. Following Yu et al. (2023); Zhou et al. (2024), we also adopt modality-balanced hard negatives to train the retriever after in-batch negative training from the multimodal dense retrieval stage. We use AdamW Loshchilov and Hutter (2019) as our optimizer and training epochs = 15, early stopping epoch = 5, batch size = 32, learning rate = 5e-6, and the temperature hyperparameter \(\tau\) = 0.01. For our MARVEL backbone, we use T5-ANCE Yu et al. (2023) as the text encoder and vision transformer Dosovitskiy et al. (2021) as the vision encoder. Models are trained using 1 NVIDIA RTX A6000 for 10 hours.

**Visual Question Answering**: Generate a radiology report from this image: <image>

**Retrieval Augmented Generation**: Here is a report of a related patient: "<document>" Generate a radiology report from this image: <image>

### RAG Finetuning Procedure

To create a RAG dataset for fine-tuning LLaVA, we search the nearest-neighbor document \(d^{*}_{txt}\) for a query image \(q_{img}\) using a retriever's embeddings. We filter out any results that involve retrieving a patient's own report, the same patient's other studies, or malformed reports in the training dataset (specified by being less than 5 characters). We apply the prompt templates in Figure 5, and fine-tune LLaVA-1.5 for one epoch. Models are trained using 8x NVIDIA RTX A6000 for 4 hours, with epochs=1, learning rate=2e-5, global batch size=128, from vicuna-7b-v1.5 checkpoint. We save the checkpoint after one full pass of the training dataset for final evaluation.

### Evaluation Details

Here, we provide implementation details regarding the evaluation methodology.

**F1-RadGraph**. For F1-RadGraph score computation, we follow previous work (MIMIC-CXR-RRS) 2 in employing RG\({}_{ER}\) as F1-Radgraph score computation on an instance level. Using the radgraph library implementation, this equates to utilizing reward_level="partial".

Footnote 2: [https://vilmedic.app/papers/acl2023/](https://vilmedic.app/papers/acl2023/)

**F1-CheXbert**. F1-CheXbert score computation consists of the micro-averaged F1-score between 5 selected classes from the CheXbert labeler. Naturally, F1-CheXbert scores are only computable over entire datasets. For instance-level CheXbert scores (used for pair mining), we employ the proportion of equivalent predicted classes

Figure 5: Prompt templates for Visual Question Answering and Retrieval Augmented Generation



between a reference and predicted text sample. These instance-level F1-CheXbert scores can be computed using \({\tt np}_{\text{sum}}({\tt ref==hyp})~{}/~{}5,\text{and}\) take on values \(\in\{0.0,0.2,0.4,0.6,0.8,1.0\}\).

**CheXpert Hidden Test Set**. We use the 1000 hidden test reports from MIMIC-CXR-RRS and download the CheXpert images from Stanford AIMI Shared Datasets 3.

Footnote 3: [https://stanfordaimi.azurewebsites.net/datasets/&cbd9ed4-2eb9-4565-affc-111cf4f7ebe2](https://stanfordaimi.azurewebsites.net/datasets/&cbd9ed4-2eb9-4565-affc-111cf4f7ebe2)

**Oracle Retrieval**. Oracle Retrieval is performed via ground-truth access to a reference document's generated report. For training queries, this is always known, and an oracle retriever would obtain documents as \(Oracle(q_{i})\doteq\operatorname*{arg\,max}_{j\in\text{corpus},j\neq i}s( q_{i},d_{j})\), where \(s(q,d)\) is the sum of the F1-RadGraph and F1-CheXbert instance-wise scores. In practice, this results in retrieving samples with F1-CheXbert=\(1.0\) and the largest F1-RadGraph score within the partition. Test-time retrieval performs the same operation, without the restriction of \(j\neq i\) as self-retrieval is not possible due to the corpus being the training dataset.

**Oracle RAG**. Oracle-LLaVA is obtained by fine-tuning LLaVA under identical conditions, utilizing Oracle Retrieval for retrieving documents in the training and test set.

### Case study on Retrieved Reports

We now conduct case study on the retrieved reports shown in Table . We show that our FactMM-RAG captures most of the factual details in retrieved reports compared to the ground-truth reports. Thus, the factual correctness of our retriever can be propagated to the multimodal foundation models effectively. However, the reports retrieved from MedMAVEL contain erroneous information, which negatively impacts the report generation by multi-modal foundation models.

\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|p{142.3pt}|} \hline  & **Med-Marvel** & **FactMM-RAG** & **Reference** \\ \cline{2-4}  & A single portable chest radiograph was obtained. Bilateral pleural patients and mild attectroix have increased Cardomegaly’s unchanged. There is no consolidation or pneumothorax. Package, stencotomy wires, vascular clips, and abdominal surgical clips are unchanged. & The patient is status post median sterneters he witnessed appearance of paediatrications in the left lung base, classify the _no activation of patches and pleural effusion. There is a small right pleural effusion. Mid path anonymity edema persists._ The heart is moderately enlarged, but stable. Len said pacemaker is seen with transverse leads in the night atrium, right ventricle, and left ventricle. & The patient is status post median steromers its unchanged appearance of pediatrications in the left lung base, classify the _no activation of patches and pleural effusion. There is no transferable_. & The patient is status post median steromers its unbalanced appearance of pediatrications. There is no pneumothorax. No acute osseous abnormalities are visualized. \\ \hline F1RaGraph & 0.274 & 0.345 & Cardiomegaly, Edema, Atelectasis, Petal Effusion \\ \hline  & \multicolumn{1}{p{142.3pt}}{The heart is mildly enlarged with left ventricular configuration. There is mild-to-moderate unfolding of the thoracic auto: The arch is partly calcified. The mediastinal and halluc contours appear when unchanged. There are strongly left ventricular opacities suggesting anticomas. & Cardiomegaly, Edema, Atelectasis, Petal Effusion \\ \hline  & \multicolumn{1}{p{142.3pt}}{The heart is mildly enlarged with left ventricular configuration. There is mild-to-moderate unfolding of the thoracic auto: The arch is partially calcified. The medasinal and halluc contours appear when unchanged. _There are strongly left ventricular opacities suggestive angstromat._ & _Moderate enlargement of the cardiac silhouette_ with a left ventricular presence is unchanged. _The aorta remains troaches_, and _the hilt contours are stable._ Pulmonary vascularity is not easy in model fit better lobe. The right lung is clear. _No focal occlusion_, _clean, pleural effusion or pneumonia is present. No pleural effusion or pneumothorax is identified._ & _Ademogeneity_, Edema, Atelectasis, Petal Effusion \\ \hline  & \multicolumn{1}{p{142.3pt}}{The heart is mildly enlarged with left ventricular configuration. There is mild-to-moderate unfolding of the thoracic auto: The arch is partly calcified. The mediastinal and halluc contours appear when unchanged. _There are strongly left ventricular opacities suggestive angstromat._ & _Moderate enlargement of the cardiac silhouette_ with a left ventricular preference is unchanged. _The aorta remain troaches_, and _the hilt contours are stable._ Pulmonary vascularity is not easy in model fits is used in the left lower lobe. The original metaferation begins to clean. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contour are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified._ & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is detected. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is reported. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is identified. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns rotation, and the hilt contours are stable._ Pulmonary vascularity is not easy in model fitting is clear. _No focal occlusion_, _clean, pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is present. No pleural effusion or pneumothorax is found. & _Animate contrast of the cardiac silhouette_ with a left ventricular pre-informative to uschange?. The aorta returns