You are an expert in Deep Learning and Machine Learning. As an Academic Peer Review Assistant, your role is to provide a thorough and constructive review of submitted academic papers. When a user presents a paper for review, assess it across several key dimensions to ensure scholarly rigor and clarity. Your evaluation should cover:

•              Content Accuracy and Relevance: Analyze whether the information presented is accurate, up-to-date, and relevant to the stated research questions.

•              Argumentation and Analysis: Assess the strength and coherence of the arguments. Are the claims well-supported by evidence? Is the analysis thorough and critical?

•              Methodology: Evaluate the appropriateness and execution of the research methodology. Is it suitable for the research question? Are the methods clearly explained and justified?

•              Organization and Structure: Consider the logical flow of the paper. Is it well-organized, allowing readers to easily follow the progression of ideas?

•              Writing Quality and Clarity: Review the paper for clarity of language, style, and grammar. Is the writing clear, concise, and appropriately academic?

•              References and Citations: Check for the completeness and accuracy of citations and references. Are all sources credible and properly attributed?

Provide specific feedback for each category, citing examples from the paper to support your evaluations. Offer constructive criticism and suggestions for improvement to help the author enhance the paper. Summarize the overall strengths and weaknesses, and recommend whether the paper should be accepted, revised, or rejected based on your assessment.



## RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

## Pierre Chambon*

## Stanford AIMI

```
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
```
## Christian Bluethgen*

## Stanford AIMI

```
<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
```
## Jean-Benoit Delbrouck

## Stanford AIMI

## Rogier Van der Sluijs

## Stanford AIMI

## Małgorzata Połacin

## Stanford University

## Juan Manuel Zambrano Chaves

## Stanford AIMI

## Tanishq Mathew Abraham

## University of California, Davis

## Stability AI

## Shivanshu Purohit

## Stability AI

## Curtis P. Langlotz

## Stanford AIMI

## Akshay Chaudhari

## Stanford AIMI

## Abstract

```
Multimodal models trained on large natural image-text
pair datasets have exhibited astounding abilities in gener-
ating high-quality images. Medical imaging data is funda-
mentally different to natural images, and the language used
to succinctly capture relevant details in medical data uses a
different, narrow but semantically rich, domain-specific vo-
cabulary. Not surprisingly, multi-modal models trained on
natural image-text pairs do not tend to generalize well to
the medical domain. Developing generative imaging mod-
els faithfully representing medical concepts while providing
compositional diversity could mitigate the existing paucity
of high-quality, annotated medical imaging datasets. In this
work, we develop a strategy to overcome the large natural-
medical distributional shift by adapting a pre-trained la-
tent diffusion model on a corpus of publicly available chest
x-rays (CXR) and their corresponding radiology (text) re-
ports. We investigate the model’s ability to generate high-
fidelity, diverse synthetic CXR conditioned on text prompts.
We assess the model outputs quantitatively using image
quality metrics, and evaluate image quality and text-image
alignment by human domain experts. We present evidence
that the resulting model (RoentGen) is able to create visu-
ally convincing, diverse synthetic CXR images, and that the
output can be controlled to a new extent by using free-form
*equal contribution
```
```
text prompts including radiology-specific language. Fine-
tuning this model on a fixed training set and using it as a
data augmentation method, we measure a 5% improvement
of a classifier trained jointly on synthetic and real images,
and a 3% improvement when trained on a larger but purely
synthetic training set. Finally, we observe that this fine-
tuning distills in-domain knowledge in the text-encoder and
can improve its representation capabilities of certain dis-
eases like pneumothorax by 25%.
```
## 1. Introduction

```
Latent diffusion models (LDMs) are a type of denois-
ing diffusion models that have recently gained popularity
by enabling high-resolution, high-fidelity and yet diverse
image generation [37]. When coupled with a condition-
ing mechanism, these models allow fine-grained control of
the image generation process at inference time (e.g., by us-
ing text prompts) [36, 37, 39]. Such models have typically
been trained on large, multi-modal datasets like LAION-
5B which consist of billions of natural image-text pairs
[40]. LDMs can be repurposed for a large variety of down-
stream tasks and, given appropriate pre-training, may be re-
garded as foundation models (FM) [2]. Because the denois-
ing process of LDMs takes place in a comparatively low-
dimensional latent space [37], LDMs can be run on mod-
```
# arXiv:2211.12737v1 [cs.CV] 23 Nov 2022


Figure 1. Text-to-image synthesis of chest x-ray images usingRoentGen, a medical domain-adapted latent diffusion model based on
the Stable Diffusion pipeline. A fine-tuned or retrained conditional U-Net denoises a vector of random Gaussian noise, conditioned
by embeddings created from short medical free-text prompts by a fine-tuned or replaced text encoder. The decoder of the variational
autoencoder of the Stable Diffusion pipeline maps the denoised latent vector to pixel space, resulting in high-fidelity, diverse chest x-rays
showing corresponding imaging features.

erate hardware resources, facilitating their deployment to
end-users.
The impressive generative capabilities of such models
permit creation of high-fidelity synthetic datasets which
may be used to augment traditional supervised machine
learning pipelines in scenarios that lack training data. This
presents a possible remedy to the paucity of well-curated,
annotated high-quality medical imaging datasets. Annotat-
ing such datasets requires structured planning and extensive
efforts by trained medical experts capable of interpreting
subtle, but semantically meaningful, image features.
Notwithstanding the lack of large, curated, publicly
available medical imaging datasets, there is typically a
text-based radiology report that gives a detailed descrip-
tion of the relevant medical information contained in the
imaging studies. This ”byproduct” of medical decision
making can be leveraged to automatically extract labels
that can be used for downstream tasks [14], but still en-
forces a narrower problem formulation than could poten-
tially be expressed with human natural language. Lever-
aging the vision-language entanglement of pre-trained text-
conditional LDMs could provide an intuitive mechanism to
create synthetic medical imaging data by prompting with
relevant medical keywords or concepts of interest.
In this study, we explore the representational bounds
of a large vision-language LDM (Stable Diffusion, SD)

```
and evaluate how to adapt it to medical imaging con-
cepts, without explicitly training on these concepts. To
leverage the extensive image-text pre-training underlying
the components of the SD pipeline, we explore its use
for generating chest X-rays (CXR) conditioned on short
in-domain text prompts. As CXRs are fast to acquire,
inexpensive, and can give insight into a large variety of
important medical conditions, they are one of the most
widely used imaging modalities worldwide.
```
```
To the authors’ knowledge, this is the first paper that
systematically explores the domain-adaptation of an out-
of-domain pretrained LDM for the language-conditioned
generation of medical images beyond the few- or zero-shot
setting. To this end, the representational capacity of the
SD pipeline was evaluated, quantified, and ultimately
expanded, exploring different strategies for improving
this general-domain pretrained foundational model for
representing medical concepts specific to CXRs.
```
```
We presentRoentGen, a generative model for synthe-
sizing high-fidelity CXR, capable of inserting, combining
and modifying imaging appearances of various CXR find-
ings through free-form medical language text prompts, fea-
turing highly detailed image correlates of the corresponding
medical concepts. Additionally, the study establishes the
```

following advancements:

1. We present a comprehensive framework to evaluate
    medical domain-adapted text-to-image models using
    domain-specific tasks of i) classification using a pre-
    trained classifier, ii) radiology report generation, and
    iii) image-image- and text-image retrieval) to assess
    the factual correctness of these models.
2. We compare several approaches to adapt SD to a new
    CXR data distribution and demonstrate that fine-tuning
    both the U-Net and CLIP (Contrastive Language-
    Image Pre-Training [35]) text encoder yields the the
    highest image fidelity and conceptual correctness.
3. The original CLIP text encoder can be replaced with
    a domain-specific text encoder that leads to improved
    performance of the resulting stable diffusion model af-
    ter fine-tuning, in the setting where the text encoder is
    kept frozen and only the U-Net is trained.
4. The SD fine-tuning task can be used to distill in-
    domain knowledge to the text encoder, when trained
    along the U-Net, improving its representational capa-
    bilities of medical concepts such as rare abnormalities.

```
5.RoentGencan be fine-tuned on a small subset (1.1-
5.5k) of images and prompts for use as a data augmen-
tation tool for downstream image classification tasks.
Training with synthetic data only performed compara-
bly to training with real data, while training jointly on
real and synthetic data improved classification perfor-
mance by 5% in our setup.
```
## 2. Related Work

### 2.1. Generative Models for Chest X-Ray Generation

Several recent works have explored the feasibility and
potential benefits of synthetic CXR generation.
A series of models have been developed to generate
synthetic images for image classification of specific dis-
eases where few training examples are available for the dis-
ease class. Such is the case of Coronavirus disease 2019
(COVID19) [16, 25, 27, 32, 42, 51].
Other models have studied the value of synthetic CXR gen-
eration through Generative Adversarial Networks (GANs)
for detecting normal (vs. abnormal) images [23], tubercu-
losis [28] and pneumonia [46]. Other authors have studied
the value of GAN-based synthetic CXRs in distinguishing
multiple related chest pathologies in a multi-class classifica-
tion setting. Some studies evaluated multiple closely related
pathologies [22, 24, 29, 41]. GAN-based synthetic CXRs
have also been used for studies across multiple pathologies,
identifying benefits of including synthetic CXRs in minor-
ity class performance [9, 47].

```
Outside of image classification, other benefits of GAN-
based synthetic CXR models have been described. This in-
cludes GAN-based models for inpainting [1, 45], segmen-
tation of coarse anatomical structures (lungs, heart, clavi-
cles) [5], or both normal/abnormal classification and lung
opacity detection [49].
Thus, most approaches have been based on generative
adversarial networks (GANs), have been developed for spe-
cific pathologies and consist of single-modality (imaging
only) models. However, GANs
To date, two LDMs have been described for synthetic
CXR generation. The first approach demonstrated the fea-
sibility of fine-tuning SD in a few-shot setting to gener-
ate synthetic images of single classes by text prompting,
and that a CXR classifier trained on real radiographs was
able to distinguish the inserted pathology with 95% accu-
racy [3]. The second work showed the benefits of latent dif-
fusion models in generating models across multiple individ-
ual pathologies, comparing to GAN-based approaches [30].
The study focused on class-conditional image generation,
and compared the performance of a classifier pretrained on
real CXR data for a multi-label classification task on real
and synthetic data (in the latter case showing a reduced
classification performance with a mean AUROC of 72% (-
9.7%)). The authors did not report quantitative or qualita-
tive metrics to evaluate the CXR generation.
No study was found to evaluate the benefit of LDM-
based synthetic CXRs to improve downstream tasks like
image classification or segmentation, and no other study at-
tempted conditioning on text prompts.
```
### 2.2. LDM for other medical imaging modalities

```
Two studies have used LDMs to synthesize three-
dimensional imaging data for modalities like computed
tomography (CT) and magnetic resonance tomography
(MRI). The first study by Pinaya et al. [33] presented a
LDM for the generation of synthetic brain MRI scans. The
generative process can be conditioned on covariates like
age, sex and ventricle volumes. Another study explored the
feasibility of a modified LDM for the unconditional gen-
eration CT and MRI scans [17], leveraging latent space
representations learned by a VQ-GAN [11]. The authors
show that in a limited data setting, pretraining on generated
data can improve the downstream performance on a seg-
mentation task (breast MRI segmentation, Dice score 0.
(+0.04)). However, despite training these models with a
substantially lower resolutions than used in clinical prac-
tice, up to 32% of the 50 scans evaluated by radiologists
showed major anatomical inconsistencies.
```
## 3. Datasets

```
To develop a generative model capable of incorporating
a variety of medical concepts formulated in natural lan-
```

Figure 2. Text-conditioned synthesis of CXR. Each image was hand-picked out of four generated CXR per respective prompt. Here,
presence or absence of a finding (pleural effusions, dotted ROI added for visualization) and dimensions like size and laterality were
controlled via prompting. Note that the model correctly incorporated the radiological convention of displaying the right patient side on the
left side of the image, and vice versa.

guage for the domain of CXR, we leverage the publicly
available MIMIC-CXR dataset [15], under institutional
review board approval. The full dataset contains 377,
images and their associated radiology reports, from 227,
unique studies performed at the Beth Israel Deaconess
Medical Center in Boston, MA, USA.

For the purpose of this work, this dataset was filtered
using the following approach:

Each radiology report consists of two major sections: i)
Findings: A textual description of normal and abnormal
anatomical findings in the image, and ii) Impression: An
interpretative summary of the findings for supporting medi-
cal decisions. This work focused on impression sections as
input, dropping impressions shorter than 7 characters (e.g.,

```
”Slight”, ”Unchanged”, negligible occurrence) and those
exceeding the CLIP tokenizer’s limit of 77 tokens (14% of
all impression sections). On the other hand, findings tend
to be much longer (the last filter would drop 40% of them)
and would require text encoders with longer inputs.
Most CXR are acquired either in a posterior-anterior
(PA, detector plate in front of the patient), anterior-posterior
(AP, detector plate behind the patient) or lateral (LAT)
projection technique, depending on the indication, the pa-
tient’s condition, and affecting the radiographic appearance
of anatomical structures and pathologies.
MIMIC-CXR contains ten approximately equally dis-
tributed and sized subgroups (p10-19). Nine subgroups
(P10-P18, excluding the studies used in the official MIMIC-
CXR test set) were used for training and development, while
```

Dataset PA PA/AP/LAT MIMIC P
train train test test
n 38,009 175,622 2,225 5,

Abnormalities
Atelectasis 7795 40677 522 918
Cardiomegaly 5719 32744 520 698
Consolidation 1779 9814 193 182
Edema 3817 30546 686 453
Enlarged Card. 1107 6550 127 115
Fracture 1420 4752 54 142
Lung Lesion 2116 6627 96 203
Lung Opacity 9136 41852 641 1028
No Finding 10005 47846 486 1738
Pl. Effusion 7859 41204 694 1051
Pl. Other 652 1997 58 77
Pneumonia 8250 33139 512 921
Pneumothorax 1413 7304 62 162
Sup. Devices 3123 32732 487 368
Impression
Mean no. char. 128.0 142.0 199.1 118.
Std 90.1 96.4 174.0 89.
Mean no. tok. 29.0 32.1 44.3 27.
Std 18.5 20.0 37.19 18.

```
Table 1. Composition and details of each train and test set.
```
the last subgroup (p19) served as holdout set.
In each split, we cap the number of ”No finding”
reports, obtained through the CheXpert labeler [14], to
limit the imbalance of the dataset since ”No Findings” have
a higher representation than all other positive findings.
This enables a more balanced training and speed up the
learning of positive abnormalities, especially those with
rare occurrences.

This process yields two training sets, ”PA train” (con-
sisting exclusively of PA views) and ”PA/AP/LAT” train (all
views), and two test sets, ”P19 test” (PA views), and MIMIC
test, using the official MIMIC split. See Tab. 1 for details.

## 4. Stable Diffusion Fine-Tuning

### 4.1. Model Architecture

Stable Diffusion (SD) is a powerful pipeline of several
collaborative models [37], with three main components rel-
evant to the presented work (Fig. 1):

- The variational autoencoder (VAE) with an encoder
    capable of compressing high-dimensional inputs into
    lower-dimensional latent representations, and a de-
    coder mapping processed latent representations back
    to the pixel space.
       - A conditional denoising U-Net, used to iteratively de-
          noise an initially randomly generated latent vector.
       - A conditioning mechanism. In the case of the original
          SD pipeline, this is a CLIP text encoder mapping text
          inputs to a 768-dimensional embedding space, with a
          limit of 77 tokens (CLIP tokenizer limit) [35].

```
In the following, unless otherwise specified, this archi-
tecture was not modified except for disabling the built-in
”safety checker”, as it was found to have a high false-
positive rate for medical prompts.
```
### 4.2. Few-shot fine-tuning

```
Previous work investigated several approaches to fine-
tune the SD pipeline for CXR generation in a few-shot
setting [3].
```
```
Using a technique called ”Textual Inversion”, Gal et al.
explored the possibility of keeping the VAE, the U-Net
and almost all parts of the text encoder frozen [12]. In
order to avoid catastrophic forgetting, they choose to only
introduce a new token, e.g.,<chest-xray>to describe a
general image style or<pleural-effusion>to capture a
certain object/abnormality, and train only the embedding of
this token to correctly learn what it describes, on a set of
images and corresponding prompts where this new token
appears.
```
```
Another approach, called ”Textual Projection” [3]
leverages the text encoder part of the SD pipeline, to keep
the existing text encoder frozen and not introduce any
new token; instead replacing the original general-domain
CLIP text encoder with a domain-specific text encoder
followed by a projection head that maps the structure
of the domain-specific embedding space to the structure
of the general-domain embedding space of CLIP - used
to condition the denoising process of the frozen U-Net.
This approach did not produce satisfactory results in the
few-shot learning setting, so that further exploration in
terms of training effort and projection head is needed.
```
```
The ”DreamBooth” approach introduced by Ruiz et al.
focuses on the U-Net component, unfreezing it while keep-
ing the VAE and text encoder frozen, and further fine-tuning
on a few examples from the domain-specific images [38]. In
order to avoid catastrophic forgetting and/or incrementally
learn new concepts or styles, the authors suggest the use of
a prior-preserving loss, where pairs of images and prompts
belonging to the prior are randomly sampled from the newly
generated ones at training time to maintain the performance
of the model on these prior elements. Using this approach it
is possible to generate synthetic, high-fidelity CXR and in-
sert simple pathologies by text-conditioning. However, with
```

this approach it is still easy to overfit the model and the im-
age generation diversity is low [3]. The presented works use
the few-shot fine-tuned ”DreamBooth” model as a baseline.

### 4.3. Fine-tuning and training from scratch

In this work, we explore the potential of SD to be fine-
tuned or retrained on medical domain-specific images and
prompts, leveraging a large, radiology image-text dataset.
In particular, for a set of images and prompts, we leverage
the VAE, the text encoder and the U-Net, and compute an
MSE loss that can be used to backward propagate and train
the different components of the SD pipeline.
More specifically, for each text-image pair
(xtext, ypixel), random gaussian noiseNgets sampled in
the latent space of dimensions(h, w):

N∼N( (^0) h×w,I(h×w) 2 ) (1)
Using the text encoder and the VAE, both the prompt
xtextand the corresponding imageypixelare encoded, and
sampled noiseNis added to the latent representation of
the latter for a random number of timestepst. The U-Net
processes this noisy latent representationVAE(ypixel)⊕tN
along the encoded conditioning promptEnctext(xtext)to
predict the original sampled noiseNˆ:
Nˆ=Unet(Enctext(xtext),VAE(ypixel)⊕tN, t) (2)
An MSE loss computed between the true and predicted
noisesNandNˆenables to compute gradients and improve
the generation capabilities of the combined VAE, text en-
coder and U-Net:

#### L=

#### 1

```
h×w
```
```
∑h
```
```
i=
```
```
∑w
```
```
j=
```
```
(Nˆi,j−Ni,j)^2 (3)
```
Previous work indicated that the VAE component is suit-
able for CXR generation without modifications [3]. Thus,
for the following experiments, the VAE was kept frozen.
The experimental effort is then focused on exploring:

- the U-Net component. This component is kept un-
    frozen, as earlier studies showed limited performance
    when freezing the U-Net [3]. It can either be further
    fine-tuned from the original SD work [37] (default ap-
    proach), or reset randomly and trained from scratch on
    the in-domain dataset.
- the text encoder. This component can be kept frozen,
    therefore only training the U-Net, or it can be un-
    frozen and trained jointly with the U-Net (default ap-
    proach). Finally, provided that the tokenizer limits are
    preserved according to the original SD work using a
    CLIP text encoder, the text encoder can be replaced

```
with a domain-specific text encoder. In this latter case,
the U-Net is trained and learns to be conditioned on
this new text encoder.
```
```
In addition to these different fine-tuning approaches, the
size and distribution of the training dataset (Sec. 3), the
number of training steps and the learning rate can vary; and
finally the training can be done in full- or half-precision
(fp32 and fp16) or using the Brain Floating point format
(bf16).
```
### 4.4. Training details

```
Experiments were conducted on 64 A100 GPUs split
across two compute instances. Models were mostly trained
(unless otherwise specified) in bf16 precision, as this led
to 1/3 reduction of training time. Using bf16 precision,
no significant training time difference was noticed across
the various experiments, whether they fine-tuned one or
several SD components, except when changing the number
of training steps itself. At an image resolution of 512x
px, an A100 GPU fine-tuning SD can hold a batch size of 8
(bf16 precision). Splitting batches across the GPUs of each
compute instance, models were trained with a batch size
of 256. In this setting, fine-tuning a model for 1k training
steps took approximately 20 minutes; for 12.5k training
steps, around 5 hours; for 60k training steps, one day.
```
```
Model weights for the SD pipeline (version 1.4) were
obtained from the repository ”CompVis/stable-diffusion-
v1-4”(Hugging Face hub [53]), unless otherwise specified.
The code implementation was built on both the diffusers
library [50] and the ViLMedic library [8]. Two domain-
specific text encoders were used: RadBERT [4], download-
able on the HuggingFace repo StanfordAIMI/RadBERT,
and SapBERT [20] available at cambridgeltl/SapBERT-
from-PubMedBERT-fulltext. In the experiments that fol-
low, guidance scale 4 and 75 inference steps with a PNDM
noise scheduler [21] enabled the generation of synthetic im-
ages properly conditioned on the associated prompts.
```
## 5. Fidelity and diversity of generated images

```
Generative models need to show two major qualities:
the generated images should be close to the distribution
they are modeled after (fidelity), and the outputs should
ideally cover a large variability of the underlying real im-
ages (diversity). To assess fidelity, the Fr ́echet Inception
Distance (FID) was calculated as a metric for how simi-
lar the distributions of real and synthetic images are. A
smaller FID indicates that generated images are more sim-
ilar to original images, and vice versa. As FID is typ-
ically calculated using an Inception V3 model [48] pre-
trained on ImageNet, it might fail in a domain adaptation
setting by being unable to capture relevant features of the
```

#### FID↓ MS-SSIM↓

Experiments XRV IncepV3 CLIP

Baselines
Original SD 47.7 275.0 52.7 .09±.
DreamBooth SD 19.5 122.4 18.6 .28±.
LR, train steps
1e-4, 1k 6.1 64.6 2.1 .30±.
5e-5, 1k 6.0 65.0 2.1 .32±.
1e-4, 12.5k 6.4 66.9 2.7 .22±.
5e-5, 12.5k 8.2 67.4 3.3 .19±.
1e-4, 60k 7.4 85.5 8.1 .22±.
5e-5, 60k 3.6 54.9 2.6 .32±.
Components
Rnd U-Net, 1k 9.0 233.8 17.8 .20±.
Rnd U-Net, 60k 4.9 75.4 2.8 .29±.
Rnd U-Net only,
60k 16.5 114.2 7.0 .14±.
U-Net only, 60k 9.2 85.5 4.2 .19±.
Text Encoders
RadBERT, 1k 8.3 227.2 19.1 .21±.
RadBERT, 12.5k 4.6 68.5 6.0 .26±.
SapBERT, 60k 6.0 72.0 3.0 .22±.
RadBERT, 60k 6.7 88.3 5.9 .19±.
Multiple Views
60k 19.3 114.0 5.4 .12±.

Table 2. Quantitative assessment of image fidelity and diversity:
FID for features extracted from a CXR-pretrained DenseNet-
(XRV), ImageNet-pretrained InceptionV3, and natural image-text-
pretrained clip-ViT-B-32 (CLIP). Smaller FID indicates higher fi-
delity to the original images. Smaller MS-SSIM indicates higher
intra-prompt image generation diversity. SD: Stable Diffusion.
Image diversity: MS-SSIM mean and standard deviation. Frechet ́
Inception Distance (FID). MS-SSIM: Multi-scale structural simi-
larity index measure.

CXR modality [18]. For this reason, FID scores were cal-
culated from intermediate layers of three fundamentally dif-
ferent (in terms of architecture, domain, and pretraining)
models: InceptionV3 (2048-dimensional activation vector
from the pool3 layer) pretrained on ImageNet, CLIP-ViT-
B-32 (768-dimensional) trained on millions of image-text
pairs [35], and an in-domain classification model trained to
detect common pathologies in CXR (DenseNet-121, XRV,
1024-dimensional) [6].

Generation diversity was assessed by calculating the
pairwise multi-scale structural similarity index metric (MS-
SSIM, Gaussian kernel size 11; sigma, 1.5) [52] of four gen-
erated samples per prompt. A lower MS-SSIM indicates a
smaller structural similarity between images and can be in-
terpreted as higher diversity among the outputs.

```
The calculated metrics allow a comparison of the quality
```
```
(in terms of fidelity and diversity) of synthetic images gen-
erated from the impression sections of the p19 test set with
the original images from the same test set, as displayed in
Tab. 2. This works explores three main experimental di-
mensions: the effect of hyper-parameters like learning rate
and training steps; which SD component best to fine-tune
or retrain individually or in combination with other compo-
nents; and switching the CLIP text encoder with different
domain-specific text encoders.
```
```
Learning rate, train steps 1k training steps improved the
FID scores on the two baseline approaches, original SD and
DreamBooth SD, underlining the ability of the SD pipeline
to quickly learn domain specific content. As the number
of training steps grew to 12.5k, FID scores slightly deteri-
orated, which we hypothesize to be due to the model not
overfitting a few images anymore and diversifying its syn-
thetic generations, but losing in quality for each of these
generations. Finally, 60k steps provided the best quality of
results when using the learning rate 5e-5, with an FIDXRV
of 3.6 and an FIDIncepV3of 54.9. The results of the same
model evaluated with CLIP were not coherent: the FID
score was higher than training with only 1k steps, though
the domain-specific FIDXRVcoupled with a manual review
strongly suggested an improvement of performance. We be-
lieve this shows the limitations of using a CLIP-based met-
ric for this domain-specific task, once models become good
enough and spotting differences and improvements need a
more fine-grained evaluation. The gap in performance be-
tween 12.5k and 60k steps suggests that more training steps
could further improve the quality of the synthetic distribu-
tion of images. As results were worse for a learning rate
1e-4, other experiments relied on a learning rate of 5e-5.
```
```
Components All experiments on learning rates and train-
ing steps relied on jointly and continuously fine-tuning both
the U-Net and the text encoder, starting from the original
SD weights. This approach is compared with only fine-tune
the U-Net and/or initialize the U-Net to train from scratch.
Over 1k training steps, randomly initializing the U-Net and
training it along the text encoder led to a 50% deteriora-
tion compared to the continuous fine-tuning equivalent, but
was still able to beat the baselines by a large margin, from
FIDXRVvalues of 47.7 and 19.5 to an FIDXRVof 9.0. After
60k steps, the randomly-initialized U-Net variant achieves
an FIDXRVof 4.9, which does not beat an FIDXRVof 3.
achieved by our best model. Over the same number of train-
ing steps, training the U-Net only, from a random initializa-
tion, showed limitations and only achieved an FIDXRVof
16.5, whereas training the U-Net only from the original SD
approach yielded an FIDXRVof 9.2. This underlines the im-
portance of fine-tuning the text encoder along the U-Net, to
not only speed up the learning process but also to achieve
```

better asymptotic performance.

Text Encoders In our last main set of experiments, we ex-
plored how switching the CLIP text encoder with a domain-
specific text encoder, such as RadBERT [4] or SapBERT
[20], could improve performance, in the setting where we
keep the domain-specific text encoder frozen and only train
the U-Net initialized from scratch. After 60k training steps,
the model using SapBERT achieved an FIDXRVof 6.0, with
RadBERT an FIDXRV6.7, compared to the random U-Net
only model that only scored an FIDXRVof 16.5. This 60%
improvement in FIDXRV strongly suggests that domain-
specific knowledge already encoded in these text encoder
can be leveraged by the SD model to generate accurate im-
ages. As a next step experiment, we would like to explore
how fine-tuning these domain-specific text encoders along
the U-Net could help further improve performance, while
limiting catastrophic forgetting phenomenons.

Diversity All above-mentioned models were assessed
using the MS-SSIM metric, with a smaller values represent-
ing a lower structural similarity, which can be interpreted
as an indicator of higher diversity, and vice versa. This
metric, as is, can show limitations: it indicates higher
diversity for samples generated by a model trained for 1k
steps (lr, 5e-5) than the same model trained for 60k steps.
Manual inspection indicates that the 60k-steps variant
features a richer generation diversity, but a model trained
for only for 1k steps produces frequent errors and outputs
out-of-domain images, achieving poor fidelity but higher
MS-SSIM score. Therefore, MS-SSIM scores and FID
values need to be considered jointly. In these experiments,
we included a model trained on multiple views (PA, AP
and LATERAL) for 60k steps. Although it achieved lower
scores than most models in Tab. 2, the evaluation setting
involves the p19 test set that only includes PA images,
therefore partially invalidating the use of FID scores for
this model. As expected, the multi-view model showed the
lowest MS-SSIM among all the fine-tuned models, as it
generates images from all available views.

To evaluate a possible relationship between generation
diversity and the length of the conditioning text prompt,
we calculated the diversity (as expressed by MS-SSIM) for
pairwise comparisons between generated CXR images of
the same prompt Fig. 3. Compared to the original SD model
with a consistently high diversity (low MS-SSIM) across
CLIP token lengths, the fine-tuned models tend to exhibit
a smaller diversity with increasing token length. We hy-
pothesize this to be due to higher constraints imposed by a
more detailed and thus more specific prompt. This relation-
ship is not observed for the model trained for only 1k steps,

```
Figure 3. Intra-prompt image diversity by CLIP token length.
Mean MS-SSIM as a measure of intra-prompt generation diver-
sity for 5,000 prompts (with 4 generated images per prompt) for
selected models. Lower mean MS-SSIM indicates higher diver-
sity. For visualization, CLIP token lengths have been binned to
intervals of size 10. The light areas indicate 95% confidence in-
tervals. SD: Stable Diffusion. PA: Postero-anterior view. AP:
Antero-posterior view. Lat: Lateral view.
```
```
which could be explained by the above-mentioned limita-
tion of MS-SSIM as an indicator of diversity.
```
## 6. Factual correctness of generated images

```
While FID gives an insight of the feature discrepancy be-
tween ground-truth and generated images, it also has some
important shortcomings. Notably, models such as Inception
V3 or XRV DenseNet are vision-only and trained on a clas-
sification tasks. They may not capture every fine-grained
feature required to evaluate the semantic correctness of the
generated images. To further test our generative models, we
leveraged pre-trained multimodal models to benefit from an
evaluation at the intersection of vision and language. More
precisely, we used models that either generate text from im-
ages or encode medical text to report semantic, fine-grained
evaluations.
```
### 6.1. Multi-label classification

```
MIMIC-CXR provides class labels for findings com-
monly encountered in CXR, derived by natural language
processing, using the corresponding text reports similar to
the work by Irivin et al. [14]. Using the impression sections
from the p19 test set (filtered by the above-mentioned
criteria), different fine-tuned SD models were queried to
produce synthetic images that reflect the abnormalities
of the corresponding impression sections, as labeled by
CheXpert. Such synthetic images are displayed in Fig. 4.
```

Figure 4. Synthetic images created by prompting a fine-tuned model (60k training steps; learning rate 5e-5; PA-view) for typical CXR
abnormalities. The generated CXRs feature high levels of detail: When prompted for ”edema” (top right), perihilar haziness(white arrow-
heads) and peribronchial cuffing (black arrowhead), both features seen in pulmonary edema, can be observed. For ’pneumothorax’ (bottom
row, third image from the left), a fine line representing the visceral pleural lining of the partially collapsed lung can be delineated (dashed
line).

Without being further modification, a pre-trained classifi-
cation model (DenseNet-121, XRV) was used to classify
both the real images (the p19 test set baseline) as well
as 5,000 images (one image per prompt) per fine-tuned
SD model. NLP-derived labels of the respective reports
served as ground truth. All images were preprocessed
by downsizing to a maximum height or width of 512 px
(preserving aspect ratio), center cropping (512 px) and
downsizing to a resolution of 224 x 224 px (following the
original model’s specifications).

```
Putting all of this together, Tab. 3 displays the results
after having the DenseNet-121 classification model predict
classes from images generated by various SD models with
the original CheXpert weak labels. As a baseline check, the
original SD pipeline yields an AUROC of approximately
0.5, as the generated images are completely out of distri-
bution. Similarly, a few-shot trained model (”DreamBooth
SD”) only scores a (filtered average) AUROC of 0.61.
Although DreamBooth allows rapid and compute-efficient
fine-tuning, this approach does yield images with accurate
medical content outside of specific two-class-conditional
```

Source Atel. Cmgl. Cnsl. Edma Les. Opac. Eff. Other Pneum. PTX Filtered Avg

Baselines
p19 Test Set .75 .84 .70 .84 .65 .68 .87 .70 .60 .78.
Original SD .48 .49 .47 .49 .56 .51 .48 .56 .51 .55.
DreamBooth SD .62 .54 .58 .50 .66 .56 .72 .70 .51 .66.
LR, train steps
1e-4, 1k .76 .82 .69 .85 .60 .74 .90 .63 .61 .84.
5e-5, 1k .75 .79 .66 .84 .57 .68 .91 .61 .57 .80.
1e-4, 12.5k .77 .83 .70 .82 .62 .72 .91 .76 .56 .79.
5e-5, 12.5k .72 .77 .70 .77 .57 .69 .88 .72 .58 .76.
1e-4, 60k .72 .82 .61 .69 .51 .57 .86 .75 .45 .73.
5e-5, 60k .78 .84 .66 .78 .54 .66 .89 .71 .47 .76.
Components
Rnd U-Net, 1k .72 .69 .66 .70 .63 .68 .82 .59 .52 .76.
Rnd U-Net, 60k .79 .82 .70 .78 .63 .73 .90 .73 .52 .79.
Rnd U-Net only, 60k .72 .76 .70 .72 .60 .67 .87 .71 .58 .76.
U-Net only .77 .82 .67 .77 .55 .66 .90 .74 .48 .74.
Text Encoder
RadBERT, 1k .67 .59 .65 .65 .63 .59 .70 .43 .51 .78.
RadBERT, 12.5k .77 .78. 65 .80 .66 .69 .87 .72 .61 .81.
RadBERT, 60k .72 .73 .63 .67 .62 .68 .86 .66 .55 .78.
SapBERT, 60k .80 .84 .68 .77 .63 .72 .90 .71 .48 .77.
Multiple Views
60k .72 .73 .63 .67 .62 .68 .86 .66 .55 .78.

Table 3. Classification performance of DenseNet121 on the p19 test set and on 5000 synthetic CXR generated using the models listed in the
experiment column, measured by the area under the ROC curve. Per model, the impression sections corresponding to the 5000 images in the
p19 test set were used to generate the sample images. Atel.: Atelectasis, Cmgl.: Cardiomegaly, Cnsl.: Consolidation, Edma: Edema, Les.:
Lung lesion, Opac.: Lung Opacity, Eff.: Pleural effusion, Pneum.: Pneumonia, PTX: Pneumothorax. The scores corresponding to Enlarged
Cardiomediastinum and Fracture were not reported, as the baseline p19 test set had AUROC values below or equal to 0.5. The filtered
average scores is a macro-average of classes with baseline p19 test set AUROC values above 0.75, namely Atelectasis, Cardiomegaly,
Edema, Pleural Effusion and Pneumothorax. In each section, class-wise best values are underlined and emboldened, and second best
values are emboldened only.

training.
Different fine-tuned SD models can be compared using
these scores, though they lead to partially incoherent con-
clusions. Looking at the experiments on the learning rate
and the training steps, fine-tuning withlr=5e-5 for 1k
steps achieved a filtered average AUROC of 0.82 versus an
AUROC of 0.81 for 60k steps, the difference being even
larger withlr=1e-4. As a manual review suggests, train-
ing longer drastically increased performance. We identified
several hypothesis that can explain these results:

- Models trained for 1k steps systematically output im-
    ages where abnormalities are represented in an obvious
    manner (i.e. overfit), whereas models trained longer
    learn more subtle representations that are harder to
    classify.
- The CheXpert labels, used as the ground-truth, are
    noisy [44] and the confidence intervals exceed in width

```
the differences between the various scores observed in
Tab. 3.
```
- The CheXpert labels, in the way they were generated,
    can be incorrect. A model trained for a short amount of
    time on impression sections (used to generate CheX-
    pert labels) can make the same mistakes as the CheX-
    pert labeler in understanding the presence and absence
    of abnormalities, that a model trained longer will more
    accurately capture.
- The pretrained Densenet-121 model has limited per-
    formance on the test data, as underlined by the baseline
    p19 test set. In particular, its macro-averaged perfor-
    mance on this set for the categories present in Tab. 3 is
    only 0.74.

```
Taking these elements into account, we assess that such
metrics would benefit from better ground-truth CheXpert la-
bels [44], and also warrant further exploration of other clas-
```

```
Task and models Factual correctness metrics
```
```
BL4↑ ROUGEL↑ F 1 cXb↑ BERTScore↑ factENT↑ factENTNLI↑ RadGraph↑
Radiology Report Generation
Ground truth images 8.9 23.0 50.4 45.6 27.6 23.3 22.
```
```
DreamBooth SD 2.4 12.8 36.6 36.6 13.7 6.9 9.
5e-5, 1k 3.6 15.8 46.2 38.0 22.9 14.3 15.
5e-5, 60k 5.5 19.9 46.5 42.3 24.1 19.9 18.
RadBERT, 60k 4.7 18.0 42.6 40.3 20.6 17.4 16.
SapBERT, 60k 4.9 20.3 46.2 42.3 23.9 20.2 19.
U-Net only, 60k 4.1 15.6 36.3 38.7 17.4 13.8 13.
Multiple views, 60k 7.0 22.5 46.3 43.2 24.5 21.5 20.
```
```
Prec@5↑ Prec@10↑ Prec@50↑
Image-Image Retrieval
Ground truth images 55.2 47.8 40.
```
```
DreamBooth SD 27.8 23.9 20.
5e-5, 1k 39.6 35.5 26.
5e-5, 60k 46.1 44.4 34.
RadBERT, 60k 44.2 36.3 29.
SapBERT, 60k 42.3 38.8 32.
U-Net only, 60k 40.3 35.5 28.
Multiple views, 60k 49.5 47.7 39.
```
```
Prec@5↑ Prec@10↑ Prec@50↑ F 1 cXb↑ BERTScore↑ factENT↑ RadGraph↑
Image-Text Retrieval
Ground truth images 41.2 40.9 37.6 49.1 35.8 19.2 11.
```
```
DreamBooth SD 22.1 20.4 17.1 35.0 31.8 11.4 6.
5e-5, 1k 29.3 28.1 27.1 42.7 33.8 13.0 6.
5e-5, 60k 35.0 34.4 30.2 45.3 34.3 15.3 9.
RadBERT, 60k 29.3 28.1 25.6 41.6 33.2 13.0 7.
SapBERT, 60k 36.5 32.7 30.1 45.4 34.0 14.0 8.
U-Net only, 60k 28.3 26.5 23.7 39.0 32.6 13.1 8.
Multiple views, 60k 40.8 38.8 35.2 47.4 34.4 17.9 10.
```
Table 4. Factual correctness metrics for the tasks of radiology report generation, image-image retrieval and image-text retrieval. Best score
for each metric and each task are emboldened and underlined, second best score only emboldened.

sifiers. Despite this, the findings are sufficient to capture
differences between in- and out-of-domain models. The
following subsections investigate other downstream appli-
cations that can be used to assess the medical correctness of
synthetic images.

### 6.2. Radiology Report Generation

The task of Radiology Report Generation (RRG) con-
sists of building assistive systems that take X-ray images of
a patient and generate a textual report (called impression)
describing clinical observations in the images. In the scope
of our evaluation of the factual correctness of generated

```
images, we leveraged a model pre-trained on MIMIC-CXR,
which contains image-impression pairs [15]. Our evalua-
tion was carried out as follows: 1) we generated images
using our models for every ground-truth impression of the
MIMIC-CXR test set 2) we input these generated images
in the pre-trained model that outputs new impressions 3)
we compare these new impressions with the ground-truth
impression used to generate the images. We also report the
RRG model’s performances when using the ground-truth
images as an upper-bound baseline.
```
```
To evaluate the new impression against the ground-truth
```

impressions, we report the widely used natural language
generation metrics such as BLEU [31], ROUGE [19]
and BERTScore [54]. We also leverage the newly in-
troduced factual-oriented metrics, namely the factENT,
factENTNLI [26] and RadGraph score [7]. These three
metrics evaluate the factual correctness, completeness and
consistency of the generated impression using Named
Entity Recognition and Entity Relation systems trained
on radiologist-annotated data (more information in
Appendix A). Finally, we report the F 1 CheXbert [56]
metric computing the F1-score between the prediction of
CheXbert [43] run on the generated report and the ground
truth report.

The results in Tab. 4 underline the capabilities of such
metrics to capture the differences in generation capabilities
of our models, when a fixed RRG model is used on top
of the synthetic images to generate synthetic reports to be
compared to the ground-truth reports. As a sanity check,
we can observe the progress made from the DreamBooth
SD baseline, achieving 13.7 factENTand 9.0 RadGraph, to
our model trained for 1k steps, improving these scores to
22.9 and 15.0, and finally the scores of the 60k steps vari-
ant, respectively 24.1 and 18.9. Using these RRG metrics,
we notice the superiority of the approach trained on multi-
ple views for 60k train steps: a contrario to the evaluation
framework of Sec. 5, the RRG metrics are image-view ag-
nostic, and therefore capable of measuring the added-value
of training on a more diverse set of images from multiple
views. Our variants trained only on PA images, either by
continuously fine-tuning both the U-Net and the text en-
coder or by training the U-Net from-scratch with a frozen
in-domain text encoder, also achieve high scores that are
only around 2 to 5% lower than the best model, trained
on multiple views. Nevertheless, we suspect that training
longer would give a bigger advantage to the model using all
views, which would be capable of leveraging a larger and
more diverse set of training images and prompts.

### 6.3. Zero-shot Image-Image Retrieval

This evaluation is similar to the conventional content-
based image retrieval setting in which we search for images
of a particular category using a representative query image.
For evaluation, a group of query images and a larger
collection of candidate images, each with a class label,
are given to a pretrained CNN encoder. Each query and
candidate image is encoded with this encoder, and then
for each query, we rank all candidates by their cosine
similarities to the query in descending order. A point of
accuracy is given if a retrieved candidate image has the
same category (same abnormality label) than the query
image.

```
As pretrained visual encoder, we use conVIRT [55],
a multimodal model trained using contrastive learning
methods using radiology reports and chest x-rays. For
the images, we use the MIMIC-CXR test-set. We filter
this pool of images by only keeping images whose report
only contains one abnormality that is either atelectasis,
consolidation, cardiomegaly, edema, fracture, pleural
effusion, pneumonia or pneumothorax (we discard reports
with multiple labels). At the end of the filtering, we keep
200 query images and 400 candidate images. We focus our
evaluation on retrieval precision, and evaluate our models
with Precision@k metrics wherek = 5, 10 , 50. Note
that for our evaluations, the query images are the images
generated by our different models (given the selected
reports as described).
```
```
Compared to the previously described RRG task, the
image-image retrieval task allows us to better discriminate
our different trained variants, seeing the scores reported in
Tab. 4. The model trained on multiple views outperforms
the second best variant that generates PA images only, both
further fine-tuning the text encoder and the U-Net for 60k
steps, on all precision scores by a 7 to 13% improvement.
Though the models trained with domain-specific text en-
coders are not the best performing ones, we account for the
fact that they only fine-tune the U-Net and keep the text en-
coder frozen. We can notice that they outperform the variant
that uses the same fine-tuning approach but with the base
CLIP text encoder, further suggesting that correctly fine-
tuning these domain-specific text encoders, instead of keep-
ing them frozen, is a promising approach to improve the
generation capabilities of our models.
```
### 6.4. Zero-shot Image-Text Retrieval

```
This task is similar to the Image-Image scenario, with
the difference that a query image embedding is map into
a textual embedding space to retrieve the most likely
impression given the image. For this experiments, we
chose the CXR-RePaiR model [10]: a retrieval-based
radiology report generation approach using a pre-trained
contrastive language-image model. This model is trained
using a CLIP-style approach: the objectives maximizes
the similarity between the ground-truth text and image
pairs embeddings. We also use the MIMIC-CXR test-set
containing paired impressions-images.
```
```
We still focus our evaluation on retrieval precision using
Precision@kwithk= 5, 10 , 50. A point of accuracy is
given if a retrieved candidate impression has the same
category (same abnormality label) than the query image. In
this settings, the NLG metrics are not suitable. Nonethe-
less, we can use the factual correctness metrics presented
in Section 6.2. To do so, we evaluate the top-1 retrieved
```

Figure 5. Intra-prompt synthetic image diversity. Four generated
samples using the prompts ’Big right-sided pleural effusion with
adjacent atelectasis’ (top row) and ’Big left-sided pleural effusion
with adjacent atelectasis’ (bottom row). Note the diverse apper-
ance of the right-sided pleural effusion with atelectasis and vary-
ing amounts of interlobar fluid (top row, white arrowheads), and
the differences in contrast, with higher contrasts especially in the
three left images in the bottom row, similar to real CXR when us-
ing different X-ray tube voltage settings.

impression against the ground-truth impression of the
image.

Similar to the RRG and image-image retrieval metrics,
the image-text retrieval scores underline the superiority of
training on data from multiple views, achieving the best per-
formance for every metric of this task on top of the other
tasks. As a new method to benchmark language models,
training along a frozen SapBERT [20] leads to improved
performance compared to RadBERT [4], suggesting that
SapBERT has more and better structured in-domain knowl-
edge about radiology reports. Though for the image-only
image-image retrieval task, further fine-tuning the origi-
nal SD weights systematically outperformed the SapBERT-
based model, we notice that for both tasks that directly rely
on text, SapBERT-based training can outperform a model
that uses CLIP text encoder, even in the setting where the
CLIP text encoder is fine-tuned (while SapBERT is system-
atically frozen). CLIP training has a strong focus on learn-
ing features that are relevant to both text and images, pos-
sibly leading to better signal for an image-image retrieval
task. We hypothesize that an in-domain text encoder could
better capture some fine-grained textual details when en-
coding the impression prompts, improving performance on
tasks where text is then directly used to attribute a score.

### 6.5. Qualitative evaluation

In order to assess the visual quality of the generated im-
ages as well as the alignment of the radiological concepts
with the text prompts, two radiologists with 7 and 9 years of
experience in reading CXR were asked to review and rate:

- 104 blinded pairs of true images and synthetic images

```
(generated from the corresponding prompts, from a
balanced sample of p19 test set), rating them on a scale
from -2 (synthetic image is more realistic) to 2 (true
image is much more realistic).
```
- 107 pairs of synthetic images and original prompts,
    rating them on a scale from -2 (the synthetic image
    does not correspond at all to the conditioning prompt),
    -1 (the image does not show all of the findings men-
    tioned in the impression section), 0 (the image shows
    the most salient finding of the prompt but not all as-
    pects), 1 (the image shows most of the aspects men-
    tioned in the prompt) to 2 (the synthetic image aligns
    perfectly with the prompt).

```
In terms of diversity, and compared to previous work
fine-tuning the SD pipeline in a few-shot setting [3], the
generated outputs feature a broad variability of features per
prompt (Fig. 5).
For the first experimental setup, the average ratings given
by the two radiologists were on average 1. 67 ± 0. 63 and
1. 81 ± 0. 46. While the general image appearance was visu-
ally very similar to real CXR, some aspects like electrodes
or other device components almost always contained un-
realistic features (e.g. discontinuations or streaks), which
made this distinction simple. In the majority of cases ra-
diologists were thus able to confidently guess which image
was true and which was synthetic.
The second experiment yielded average ratings of 0. 41 ±
1. 41 and 0. 29 ± 1. 36. This underlines the ability of the
model, on average, to fulfill the conditional prompt, al-
though in most cases not all contents of the impression sec-
tion translated correctly to the imaging domain.
```
## 7. Data augmentation

```
To investigate the added value of creating synthetic
CXR, a DenseNet-121 classifier was trained from scratch
on varying splits of real training data (R), synthetic data
(S) generated from a model trained on the p10 subset
of MIMIC-CXR, and synthetic data generated on data
from p10-p18. The task was a multi-label classification
of six findings (cardiomegaly, edema, pleural effusion,
pneumonia, pneumothorax and ’no finding’). The fine-
tuned SD models were trained for 12.5k (p10) and 60k
(p10-p18) training steps and sampled using impression
sections from p10 or p10-p18. All classification models
were trained using an AdamW optimizer (learning rate,
1e-3; weight decay, 1e-5), a cyclic learning rate scheduler
and early stopping when the validation set AUROC did
not improve for 15 epochs. The models with the high-
est validation AUROC were used to classify the p19 test set.
```
```
Training exclusively on 1.1k synthetic images derived
from a model fine-tuned on the small dataset, a drop of 0.
```

Experiment Training Data AUROC Accuracy

```
Real Synth. (n = 5,000)
Small dataset
R (Baseline) 1.1k 0.73 0.
1.1k S 1.1k 0.69 (↓0.04) 0.
1.1k R/S 1.1k 1.1k 0.77 (↑0.04) 0.
5.5k S 5.5k 0.75 (↑0.02) 0.
R + 5.5k S 1.1k 5.5k 0.76 (↑0.03) 0.
Big dataset
30k R 30k 0.82 (↑0.09) 0.
30k S 30k 0.80 (↑0.07) 0.
30k R/S 30k 30k 0.84 (↑0.11) 0.
```
Table 5. Classification performance of a DenseNet-121 trained
from scratch on varying splits of real and synthetic training data.
1.1k and 5.5k synthetic data was sampled from a model trained
on p10 (small datasets), conditioned on prompts from p10. 30k
synthetic data was sampled from a model trained on p10-p18 (big
dataset), conditioned on prompts from p10-18. R: real data, S:
synthetic data, AUROC: area under the ROC curve, calculated on
a 5k sample of the p19 test set.

in AUROC was observed compared to the baseline. Train-
ing exclusively on 5×the initial amount of synthetic data
yielded a small improvement over the baseline (AUROC
+0.02). Augmenting real data from the small dataset with
the same amount of synthetic data (1.1k) led to a moderate
improvement (AUROC +0.04), but counter-intuitively, fur-
ther augmenting the small dataset with 5.5k synthetic sam-
ples led to a smaller increase. Adding more training data
(30k) improved the classification performance (AUROC
+0.09), as did training exclusively on 30k synthetic sam-
ples trained on the larger dataset (AUROC +0.07). Finally,
the highest improvement in classification performance was
reached by training on a combination of real and synthetic
data (AUROC +0.11 vs. AUROC 0.73 in the baseline
setup). See Sec. 7 for details.

## 8. Distilling in-domain knowledge and poten-

## tial catastrophic forgetting

By fine-tuning the U-Net alone or both the U-Net
and the text encoder on the Chest X-ray domain, we can
distil in-domain knowledge into the components of the
SD model. The quantitative and qualitative analysis of
the previous sections demonstrate the capabilities of the
model at learning visual representations that achieves high
fidelity, diversity and medical correctness. Nevertheless,
this does not measure the performance of the model on the
general domain it was initially trained on, as well as the the
performance of the isolated components of the fine-tuned
SD model. Looking at Fig. 6, we notice that the current

```
setting leads to a catastrophic forgetting of the knowledge
previously acquired. In particular, fine-tuning both the
text encoder and the U-Net accelerates the learning of
in-domain concepts but also the forgetting of the previous
domain knowledge.
```
```
As in our current setting we do not leverage any spe-
cific method to control catastrophic forgetting [13], we ob-
serve that as the model is introduced to new unseen im-
ages, its weights get rapidly altered leading to knowledge
collapse. Tab. 6 underlines the drop of performance for
theCheXpert@10score [3], that measure the in-domain
knowledge of the text encoder, for the first 1000 steps,
where the macro-averaged score drops between 20% and
40%.
Finally, looking at Tab. 6, we notice that the SD task
can improve the performance of the text encoder on an
in-domain task, in this case as measured by the macro-
averagedCheXpert@10score. Using learning rate 5e-5,
fine-tuning the model for 1k and 12.5k lead to a drop of
performance of 43% and 55%, that gets partially recov-
ered at 60k to only 25%. In the specific case of pneu-
mothorax, the knowledge of the model is even totally re-
covered or improved, if using a stronger learning rate of 1e-
```
4. Though the strong catastrophic forgetting of our current
training setting mitigates the amount of added knowledge
of the text encoder at the end of the SD fine-tuning, this
measure shows that such improvement of representation ca-
pabilities already happen for some abnormalities. Coupled
with methods that reduce the catastrophic forgetting phe-
nomenon, SD fine-tuning could be used to further adapt
text-encoders for a particular domain.

## 9. Limitations

```
RoentGenis capable of generating synthetic Chest X-
ray images that can be conditioned on prompts using medi-
cal language, however, limitations of the proposed approach
remain:
```
1. The CXR images generated byRoentGenare images
    and not actual radiographs, and come with a limited
    range of gray-scale values, preventing the use of op-
    erations like realistic windowing. They should not be
    regarded as a replacement of actual chest x-ray studies.
2. Only one dataset (MIMIC-CXR), from a single insti-
    tution, was used to fine-tune and evaluateRoentGen.
    Measuring the cross-institutional performance and ro-
    bustness of the model, as well as further training it on
    multi-institutional data, could make the model more
    generalizable, improve its understanding of condition-
    ing prompts and increase the diversity of the synthetic
    images.


Figure 6. Comparing baselines and various models, fine-tuning either the U-Net alone or both the U-Net and the text encoder, based on a
general-domain prompt and an in-domain one.

Model Frac. Ple.oth. Pneux Macro

Baseline
original SD 61.6 2.9 48.6 40.
1k training steps
lr 5e-5 55.4 2.1 24 23.
lr 1e-4 58 2.1 55.4 31.
12.5k training steps
lr 5e-5 37.4 2.9 17.2 18.
lr 1e-4 43.4 2.9 49.8 28.
60k training steps
lr 5e-5 38.2 5 48.4 30.
lr 1e-4 50 6.4 59.8 34.
lr 5e-5, Rnd U-Net init 15.4 0 11 16.
lr 5e-5, PA/AP/LAT 29.6 6.4 46.8 27.

Table 6. Class-wise (Fracture, Pleural other, Pneumothorax) and
total macro-averagedCheXpert@10scores [3] computed for the
text encoder of various fine-tuned SD models, using p19 test set.
Higher scores denote better capabilities of the model at clustering
reports per abnormality, in the latent space. All models except the
PA/AP/LATone were trained on the PA training set. All models
exceptRnd U-Net init, which has a random U-Net initialization,
were further fine-tuned from the CompVis 1.4 baseline with un-
frozen U-Net and text encoder.

3. Only the impression sections from the radiology re-
    ports associated with each image were used to train
    the model. These sections might not capture every
    aspect of the image, and training the model on other
    sections (especially findings) could enable more fine-
    grained conditioning capabilities.
4. We noticed that the model was prone to overfitting
    when trained on small datasets of a few hundreds of
    images. Follow-ups that study fine-tuning under lim-
    ited data constraints could benefit from detailed mea-

```
sures of overfitting.
```
5. Fine-tuning both the U-Net and the text encoder leads
    to catastrophic forgetting phenomenons. Developing
    methods that can limit these from happening could
    help speed up the training process as well as retain
    knowledge previously acquired.

## 10. Conclusion and future work

```
The latent diffusion model Stable Diffusion, pretrained
on billions of natural image-text pairs, can be domain-
adapted to generate high-fidelity yet diverse medical CXR
images. As established in this work, by exploring vari-
ous fine-tuning approaches and evaluation settings, the best-
performing model (RoentGen, a portmanteau of ”Roent-
gen”, as a hommage to one of the pioneers of radiology,
Wilhelm Conrad Rontgen, and ”Generator”) allows fine- ̈
grained control over the generated output by using free-
form, natural language text as input, including relevant
medical vocabulary.
The domain-adapted models can display a large vari-
ety of radiological findings (e.g., pleural effusions, atelec-
tases, pneumothoraces) beyond the scope of the usual class-
conditioned setting of previous generative approaches. Ad-
ditionally, the image appearance of these findings can be
modified using natural (e.g., by prompting for a specific side
or variations in size) and medical language (e.g. requestion
findings like ”consolidation” or ”pleural effusion”), without
being explicitly trained.
The best performance was observed after jointly fine-
tuning both pretrained U-Net and the text encoder. Replac-
ing the frozen CLIP text encoder with a domain-specific
text encoder improves performance when training the U-
Net from-scratch, underlining potential training speed-up
when aiming for domain adaptation.
We developed an evaluation framework that can assess
```

medical correctness of synthetic images with various down-
stream applications, such as radiology report generation or
image-image and image-text retrieval, allowing us to com-
pare different generative model in terms of clinical consis-
tency and completeness. In addition, we used fine-tuned
stable diffusion for downstream applications to either data
augment training datasets or replace real training data with
purely synthetic data, leading to improvements in both cases
and demonstrating the added-value of such models. Finally,
we measured the ability of stable diffusion fine-tuning to
distill in-domain knowledge into its components, in partic-
ular the text-encoder, improving its representation capabili-
ties on in-domain data.
Building upon these findings, future research will focus
on expanding the work to other study types and modalities,
furthering the medical information a fine-tuned stable dif-
fusion model could retain. In particular, we would like to
further investigate fine-tuning strategies that would allow
to limit catastrophic forgetting and therefore benefit from
previously acquired knowledge, through the use of domain-
specific text-encoders for instance.

## Acknowledgements

Research reported in this publication was made possible
in part by theNational Institute of Biomedical Imaging and
Bioengineering (NIBIB)of theNational Institutes of Health,
which funded PC under contracts 75N92020C00008 and
75N92020C00021. CB received support from the Swiss
Society of Radiology and the Kurt and Senta Herrmann-
Foundation, independent of this work. We acknowledge
support by Stability.AI in providing computational support
for this work. We acknowledge the support of this work by
the Wu Tsai Human Performance Alliance at Stanford Uni-
versity and the Joe and Clara Tsai Foundation. We would
also like to acknowledge the help of Arjun Desai in proof-
reading this manuscript.

## References

```
[1] Xingfang Ai, Min Cao, and Xuechen Li. A pseudo lesion
generation method for deep learning based chest x-ray lung
disease detection. In2021 IEEE 2nd International Con-
ference on Big Data, Artificial Intelligence and Internet of
Things Engineering (ICBAIE), pages 256–259. IEEE, 2021.
3
[2] Rishi et al Bommasani. On the opportunities and risks of
foundation models, 2021. 1
[3] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz,
and Akshay Chaudhari. Adapting pretrained vision-language
foundational models to medical imaging domains. arXiv
preprint arXiv:2210.04133, 2022. 3, 5, 6, 13, 14, 15
[4] Pierre Chambon, Tessa S. Cook, and Curtis P. Langlotz. Im-
proved fine-tuning of in-domain transformer model for in-
ferring covid-19 presence in multi-institutional radiology re-
ports.Journal of Digital Imaging, 2022. 6, 8, 13
```
```
[5] Giorgio Ciano, Paolo Andreini, Tommaso Mazzierli, Mon-
ica Bianchini, and Franco Scarselli. A multi-stage gan for
multi-organ chest x-ray image generation and segmentation.
Mathematics, 9(22):2896, 2021. 3
[6] Joseph Paul Cohen, Joseph D. Viviano, Paul Bertin, Paul
Morrison, Parsa Torabian, Matteo Guarrera, Matthew P
Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad
Hashir, and Hadrien Bertrand. TorchXRayVision: A library
of chest X-ray datasets and models. InMedical Imaging with
Deep Learning, 2022. 7
[7] Jean-Benoit Delbrouck, Pierre Chambon, Christian Blueth-
gen, Emily Tsai, Omar Almusa, and Curtis P Lan-
glotz. Improving the factual correctness of radiology re-
port generation with semantic rewards. arXiv preprint
arXiv:2210.12186, 2022. 12
[8] Jean-benoit Delbrouck, Khaled Saab, Maya Varma, Sabri
Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zam-
brano, Akshay Chaudhari, and Curtis Langlotz. ViLMedic:
a framework for research at the intersection of vision and
language in medical AI. InProceedings of the 60th Annual
Meeting of the Association for Computational Linguistics:
System Demonstrations, pages 23–34, Dublin, Ireland, May
```
2022. Association for Computational Linguistics. 6
[9] August DuMont Schutte, J ̈ urgen Hetzel, Sergios Gatidis, ̈
Tobias Hepp, Benedikt Dietz, Stefan Bauer, and Patrick
Schwab. Overcoming barriers to data sharing with medical
image generation: a comprehensive evaluation.NPJ digital
medicine, 4(1):1–14, 2021. 3
[10] Mark Endo, Rayan Krishnan, Viswesh Krishna, Andrew Y
Ng, and Pranav Rajpurkar. Retrieval-based chest x-ray report
generation using a pre-trained contrastive language-image
model. InMachine Learning for Health, pages 209–219.
PMLR, 2021. 12
[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming ̈
transformers for high-resolution image synthesis, 2020. 3
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik,
Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An
image is worth one word: Personalizing text-to-image gen-
eration using textual inversion, 2022. 5
[13] Jeremy Howard and Sebastian Ruder. Fine-tuned language
models for text classification.CoRR, abs/1801.06146, 2018.
14
[14] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Sil-
viana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad
Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins,
David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky
Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel,
Matthew P. Lungren, and Andrew Y. Ng. Chexpert: A large
chest radiograph dataset with uncertainty labels and expert
comparison, 2019. 2, 5, 8
[15] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz,
Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying
Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-
identified publicly available database of chest radiographs
with free-text reports. Scientific data, 6(1):1–8, 2019. 4,
11
[16] Yash Karbhari, Arpan Basu, Zong Woo Geem, Gi-Tae Han,
and Ram Sarkar. Generation of synthetic chest x-ray images


and detection of covid-19: A deep learning based approach.
Diagnostics, 11(5):895, 2021. 3
[17] Firas Khader, Gustav Mueller-Franzes, Soroosh Tayebi
Arasteh, Tianyu Han, Christoph Haarburger, Maximilian
Schulze-Hagen, Philipp Schad, Sandy Engelhardt, Bettina
Baessler, Sebastian Foersch, Johannes Stegmaier, Christiane
Kuhl, Sven Nebelung, Jakob Nikolas Kather, and Daniel
Truhn. Medical diffusion – denoising diffusion probabilis-
tic models for 3d medical image generation, 2022. 3
[18] Tuomas Kynk ̈a ̈anniemi, Tero Karras, Miika Aittala, Timo
Aila, and Jaakko Lehtinen. The role of imagenet classes in
frechet inception distance, 2022. 7 ́
[19] Chin-Yew Lin. Rouge: A package for automatic evaluation
of summaries. InText summarization branches out, pages
74–81, 2004. 12
[20] Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco
Basaldella, and Nigel Collier. Self-alignment pretraining
for biomedical entity representations. InProceedings of the
2021 Conference of the North American Chapter of the As-
sociation for Computational Linguistics: Human Language
Technologies, pages 4228–4238, Online, June 2021. Associ-
ation for Computational Linguistics. 6, 8, 13
[21] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo
numerical methods for diffusion models on manifolds, 2022.
6
[22] Mohamed Loey, Florentin Smarandache, and Nour Eldeen
M. Khalifa. Within the lack of chest covid-19 x-ray dataset: a
novel detection model based on gan and deep transfer learn-
ing.Symmetry, 12(4):651, 2020. 3
[23] Ali Madani, Mehdi Moradi, Alexandros Karargyris, and Tan-
veer Syeda-Mahmood. Chest x-ray generation and data aug-
mentation for cardiovascular abnormality classification. In
Medical imaging 2018: Image processing, volume 10574,
pages 415–420. SPIE, 2018. 3
[24] Tatiana Malygina, Elena Ericheva, and Ivan Drokin. Data
augmentation with gan: Improving chest x-ray pathologies
prediction on class-imbalanced cases. InInternational con-
ference on analysis of images, social networks and texts,
pages 321–334. Springer, 2019. 3
[25] Sumeet Menon, Joshua Galita, David Chapman, Aryya Gan-
gopadhyay, Jayalakshmi Mangalagiri, Phuong Nguyen, Yaa-
cov Yesha, Yelena Yesha, Babak Saboury, and Michael
Morris. Generating realistic covid-19 x-rays with a mean
teacher+ transfer learning gan. In2020 IEEE Interna-
tional Conference on Big Data (Big Data), pages 1216–

1225. IEEE, 2020. 3
[26] Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Langlotz,
and Dan Jurafsky. Improving factual completeness and con-
sistency of image-to-text radiology report generation. In
Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 5288–5304, 2021. 12,
18
[27] Daniel Iglesias Mor ́ıs, Joaquim de Moura, Jorge Novo, and
Marcos Ortega. Generation of novel synthetic portable chest
x-ray images for automatic covid-19 screening. InAI Ap-
plications for Disease Diagnosis and Treatment, pages 248–
281. IGI Global, 2022. 3

```
[28] Daniel I Mor ́ıs, Joaquim de Moura, Jorge Novo, and Marcos
Ortega. Unsupervised contrastive unpaired image generation
approach for improving tuberculosis screening using chest x-
ray images.Pattern Recognition Letters, 2022. 3
[29] Saman Motamed, Patrik Rogalla, and Farzad Khalvati. Data
augmentation using generative adversarial networks (gans)
for gan-based detection of pneumonia and covid-19 in chest
x-ray images.Informatics in Medicine Unlocked, 27:100779,
```
2021. 3
[30] Kai Packhauser, Lukas Folle, Florian Thamm, and Andreas ̈
Maier. Generation of anonymous chest radiographs using la-
tent diffusion models for training thoracic abnormality clas-
sification systems.arXiv preprint arXiv:2211.01323, 2022.
3
[31] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing
Zhu. Bleu: a method for automatic evaluation of machine
translation. InProceedings of the 40th annual meeting of the
Association for Computational Linguistics, pages 311–318,
2002. 12
[32] Sukonya Phukan, Jyoti Singh, Rajlakshmi Gogoi, Sandipan
Dhar, and Nanda Dulal Jana. Covid-19 chest x-ray image
generation using resnet-dcgan model. InAdvances in In-
telligent Computing and Communication, pages 227–234.
Springer, 2022. 3
[33] Walter H. L. Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon,
Pedro F da Costa, Virginia Fernandez, Parashkev Nachev,
Sebastien Ourselin, and M. Jorge Cardoso. Brain imaging
generation with latent diffusion models, 2022. 3
[34] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and
Christopher D Manning. Stanza: A python natural language
processing toolkit for many human languages. InProceed-
ings of the 58th Annual Meeting of the Association for Com-
putational Linguistics: System Demonstrations, pages 101–
108, 2020. 18
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
Krueger, and Ilya Sutskever. Learning transferable vi-
sual models from natural language supervision. CoRR,
abs/2103.00020, 2021. 3, 5, 7
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu,
and Mark Chen. Hierarchical text-conditional image gener-
ation with clip latents, 2022. 1
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Bjorn Ommer. High-resolution image
synthesis with latent diffusion models. In2022 IEEE/CVF
Conference on Computer Vision and Pattern Recognition
(CVPR). IEEE, June 2022. 1, 5, 6
[38] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch,
Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine
tuning text-to-image diffusion models for subject-driven
generation, 2022. 5
[39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala
Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed
Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi,
Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J
Fleet, and Mohammad Norouzi. Photorealistic text-to-image
diffusion models with deep language understanding, 2022. 1


[40] Christoph Schuhmann, Romain Beaumont, Cade W Gor-
don, Ross Wightman, mehdi cherti, Theo Coombes,
Aarush Katta, Clayton Mullis, Patrick Schramowski, Sri-
vatsa R Kundurthy, Katherine Crowson, Mitchell Worts-
man, Richard Vencu, Ludwig Schmidt, Robert Kaczmar-
czyk, and Jenia Jitsev. LAION-5b: An open large-scale
dataset for training next generation image-text models. In
Thirty-sixth Conference on Neural Information Processing
Systems Datasets and Benchmarks Track, 2022. 1
[41] Pir Masoom Shah, Hamid Ullah, Rahim Ullah, Dilawar
Shah, Yulin Wang, Saif ul Islam, Abdullah Gani, and
Joel JPC Rodrigues. Dc-gan-based synthetic x-ray images
augmentation for increasing the performance of efficientnet
for covid-19 detection.Expert Systems, 39(3):e12823, 2022.
3
[42] MY Shams, OM Elzeki, Mohamed Abd Elfattah, T Medhat,
and Aboul Ella Hassanien. Why are generative adversarial
networks vital for deep neural networks? a case study on
covid-19 chest x-ray images. InBig data analytics and arti-
ficial intelligence against COVID-19: innovation vision and
approach, pages 147–162. Springer, 2020. 3
[43] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek,
Andrew Y Ng, and Matthew Lungren. Combining auto-
matic labelers and expert annotations for accurate radiology
report labeling using bert. InProceedings of the 2020 Con-
ference on Empirical Methods in Natural Language Process-
ing (EMNLP), pages 1500–1519, 2020. 12, 18
[44] Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek,
Andrew Y. Ng, and Matthew P. Lungren. Chexbert: Com-
bining automatic labelers and expert annotations for accurate
radiology report labeling using bert, 2020. 10
[45] Ecem Sogancioglu, Shi Hu, <PRESIDIO_ANONYMIZED_PERSON>, and Bram van Gin-
neken. Chest x-ray inpainting with deep generative models.
arXiv preprint arXiv:1809.01471, 2018. 3
[46] Devansh Srivastav, Akansha Bajpai, and Prakash Srivastava.
Improved classification for pneumonia detection using trans-
fer learning with gan based synthetic image augmentation.
In2021 11th International Conference on Cloud Computing,
Data Science & Engineering (Confluence), pages 433–437.
IEEE, 2021. 3
[47] Shobhita Sundaram and Neha Hulkund. Gan-based data
augmentation for chest x-ray classification. arXiv preprint
arXiv:2107.02970, 2021. 3
[48] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,
Jonathon Shlens, and Zbigniew Wojna. Rethinking
the inception architecture for computer vision. CoRR,
abs/1512.00567, 2015. 6
[49] Youbao Tang, Yuxing Tang, Yingying Zhu, Jing Xiao, and
Ronald M Summers. A disentangled generative model for
disease decomposition in chest x-rays via normal image syn-
thesis.Medical Image Analysis, 67:101839, 2021. 3
[50] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro
Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj,
and Thomas Wolf. Diffusers: State-of-the-art diffusion
models. https://github.com/huggingface/
diffusers, 2022. 6
[51] Abdul Waheed, Muskan Goyal, Deepak Gupta, Ashish
Khanna, Fadi Al-Turjman, and Pl ́acido Rogerio Pinheiro.

```
Covidgan: data augmentation using auxiliary classifier gan
for improved covid-19 detection. Ieee Access, 8:91916–
91923, 2020. 3
[52] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multi-
scale structural similarity for image quality assessment. In
The Thrity-Seventh Asilomar Conference on Signals, Sys-
tems & Computers, 2003, volume 2, pages 1398–1402. Ieee,
```
2003. 7
[53] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau-
mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim
Rault, R ́emi Louf, Morgan Funtowicz, Joe Davison, Sam
Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama
Drame, Quentin Lhoest, and Alexander M. Rush. Hugging-
face’s transformers: State-of-the-art natural language pro-
cessing, 2019. 6
[54] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein-
berger, and Yoav Artzi. Bertscore: Evaluating text gener-
ation with bert. InInternational Conference on Learning
Representations, 2019. 12, 18
[55] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D
Manning, and Curtis P Langlotz. Contrastive learning of
medical visual representations from paired images and text.
arXiv preprint arXiv:2010.00747, 2020. 12
[56] Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D
Manning, and Curtis Langlotz. Optimizing the factual cor-
rectness of a summary: A study of summarizing radiology
reports. InProceedings of the 58th Annual Meeting of the As-
sociation for Computational Linguistics, pages 5108–5120,
2020. 12, 18

## A. Details of the factually-oriented metrics

```
factENT [26] A named entity recognizer (NER) is
applied to the generated reportyˆand the corresponding
reference y, giving respectively two sets of extracted
entitiesEˆyandEy. factENTis defined as the harmonic mean
of precision and recall between the two setsEyˆandEy.
The clinical model of Stanza [34] is used as NER.
```
```
factENTNLI[26] This score is an extension offactENT
with Natural Language Inference (NLI). Here, an entity
eofEˆyis not automatically considered correct if present
inEy. To be considered valid, the sentencesyˆcontaining
entityemust not present a contradiction with its counter-
part sentencesyin the reference report. The counterpart
sentencesy in the reference report is the sentence with
the highest BERTScore [54] againstsyˆ. The NLI model
outputs whether sentencesyis a contradiction ofsˆy. We
use the NLI model weights of [26], which relies on a
BERT-architecture.
```
```
F 1 CheXbert [56] This score uses CheXbert [43],
a Transformer-based model trained to output abnormali-
ties (fourteen classes) of chest X-rays given a radiology
report as input. F 1 CheXbert is the F1-score between the
```

prediction of CheXbert over the generated reportyˆand
the corresponding reference y. To be consistent with
previous works, the score is calculated over 5 observations:
atelectasis, cardiomegaly, consolidation, edema and pleural
effusion.