Medical diffusion on a budget: Textual Inversion for medical image generation

Authors: bram  de wilde
henkjan  huisman
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 13:13:59

Abstract: Diffusion models for text-to-image generation, known for their efficiency, accessibility, and quality, have gained popularity. While inference with these systems on consumer-grade GPUs is increasingly feasible, training from scratch requires large captioned datasets and significant computational resources. In medical image generation, limited availability of large, publicly accessible datasets with text reports poses challenges due to legal and ethical concerns. This work shows that adapting pre-trained Stable Diffusion models to medical imaging modalities is achievable by training text embeddings using Textual Inversion.
In this study, we experimented with small medical datasets (100 samples each from three modalities) and trained within hours to generate diagnostically accurate images, as judged by an expert radiologist. Experiments with Textual Inversion training and inference parameters reveal the necessity of larger embeddings and more examples in the medical domain. Classification experiments show an increase in diagnostic accuracy (AUC) for detecting prostate cancer on MRI, from 0.78 to 0.80. Further experiments demonstrate embedding flexibility through disease interpolation, combining pathologies, and inpainting for precise disease appearance control. Notably, the trained embeddings are compact (less than 1 MB), enabling easy data sharing with reduced privacy concerns.

Main paper: 
 introduction: 
 Image generation has increasingly captured the attention of many researchers, spurring an impressive progression in text-to-image generation. In particular, diffusion models have gained enormous popularity through their ability to generate high-quality and diverse images, conditioned on a text prompt (Ho et al., 2020; Dhariwal and Nichol, 2021; Ramesh et al., 2021 Ramesh et al., , 2022;; Saharia et al., 2022) . Among various text-to-image model implementations, Stable Diffusion has arguably generated the biggest impact in terms of users, owing to the fact that it is both released under a permissive license and operable using a single GPU (Rombach et al., 2022) .When applied to art or photorealistic images, generative models may exhibit some degree of error. On the other hand, the medical imaging field places a higher bar on generation quality (Yi et al., 2019; Skandarani et al., 2021) . Images need to be not only anatomically The Textual Inversion fine-tuning process for diffusion models trains a text conditioning embedding for a new token using a small set of images while keeping the rest of the architecture frozen. We show that this allows the adaption of latent diffusion models to a variety of medical imaging modalities, using only 100 examples and a single consumer-grade GPU.correct but diagnostically correct as well. Training a model like Stable Diffusion for medical imaging requires a large, varied, and ideally public dataset of images with captions, similar to those used for training on natural images (Schuhmann et al., 2022) . However, practical challenges, such as ethical and legal impediments to sharing medical data, particularly for unstructured radiology reports, complicate this endeavor (Scheibner et al., 2021; Bovenberg et al., 2020) . For one of the few public datasets of this caliber that exists, MIMIC-CXR, Chambon et al. have demonstrated that it is possible to train a latent diffusion model capable of generating chest X-ray images with high fidelity and diversity through free text prompts (Johnson et al., 2019; Chambon et al., 2022a) . They trained the system using up to 170,000 images on 64 A100 GPUs. On top of data sharing issues, some modalities and pathologies are inherently scarce: certain types of scans can be expensive or experimental and some diseases are rare or tied to specific demographics. For these reasons, especially in the medical domain, it is essential to have computationally feasible methods that can fine-tune existing models towards a smaller set of a specific modality or disease. In this paper, we pick one such method, Textual Inversion, and rigorously explore its capacities for adapting Stable Diffusion to medical imaging, with all experiments performed on a single RTX2070 GPU (Gal et al., 2022) . Code and trained embeddings are shared online. 1 
related work: 
 Several papers have applied diffusion to medical imaging, with a wide range of applications including anomaly detection, segmentation, registration, and modality transfer with image-to-image translation (Kazerouni et al., 2022) . Specifically for medical image generation, several recent works have trained diffusion models for image generation. Pre-trained models are often trained on 2D RGB datasets, but many medical imaging modalities are 3D. Recently, studies such as Khader et al. (2023) and Pinaya et al. (2022) have trained diffusion models from scratch on 3D data or even on 4D data (Kim and Ye, 2022) , and Han et al. (2023) use diffusion models conditioned on anatomical masks to generate labeled images for segmentation. Several other works studied text-to-image latent diffusion models for medical imaging (Chambon et al., 2022a; Akrout et al., 2023) . Closest to our work is (Chambon et al., 2022b) , where the authors explore various methods to adapt a pre-trained Stable Diffusion model to chest X-ray generation. They performed experiments with both Textual Inversion and fine-tuning the U-net component of Stable Diffusion, similar to (Ruiz et al., 2022) . They find that Textual Inversion works, but fine-tuning the U-net is more effective, especially with more complex prompts. They fine-tune using 5 examples per class.Our work builds on this by deeply exploring Textual Inversion by training with more examples and bigger embeddings. Additionally, we demonstrate the flexibility of the approach through example applications and by adapting to multiple and more complex modalities beyond chest X-ray. In contrast to other studies, we intentionally do not train from scratch and use small datasets to explore the feasibility of diffusion in low-data and low-compute environments. 
image generation: 
 All images are generated with Stable Diffusion v2.0, using an interactive open-source web interface (Rombach et al., 2022; AUTOMATIC1111, 2022 . Images are sampled using the ancestral Euler scheduler (Karras et al., 2022) . The main inference parameters influencing image generation quality are the number of steps for the sampling scheduler and the classifier-free guidance (CFG) scale (Ho and Salimans, 2022) . Using more steps for sampling typically leads to better image quality but increases the inference time. The CFG scale can be used to set the trade-off between sample quality and sample diversity. A high CFG scale makes the model follow the text prompt more closely at the expense of diversity. Conversely, a low CFG scale results in images that deviate more from the prompt and consequently have lower fidelity but higher diversity.To introduce a medical modality as a new concept to a pre-trained diffusion model, we use Textual Inversion (Gal et al., 2022) . This process finds a vector in the text embedding space which optimally represents the concept. Practically, this is done by freezing the entire architecture apart from the embedding vector and performing backpropagation with a similarity loss, as illustrated in Figure 1 . We train embeddings with a constant learning rate of 0.005 for 50,000 steps with a batch size of 1, which takes approximately 4 hours on an RTX2070 GPU. In the work of Gal et al., prompts are generated during training from a list of templates, for instance: "a photo of a <embedding>" or "a rendering of a <embedding>". Since this does not necessarily apply to a medical imaging context, we prompt the model only with "<embedding>" during training.We experiment with the number of sampling steps, the CFG scale, the number of images used to train embeddings, and the embedding vector size. To evaluate the impact of these parameters on generation quality, we compute the Fréchet Inception Distance using 1000 generated samples compared to 1000 real examples for each parameter setting (Szegedy et al., 2015) . FID scores are calculated with an ImageNet pre-trained networks (FID), and a domain-specific medically pretrained network, RadImageNet (MFID) (Mei et al., 2022) .To explore the potential benefits of a diffusion-based approach over a GAN-based approach, we include the state-of-the-art StyleGAN3 as a baseline (Karras et al., 2021) . To allow a fair comparison, we fine-tune a pre-trained StyleGAN3 on the same hardware for the same number of steps. A blind comparison between Stable Diffusion and StyleGAN3 was made by an expert prostate radiologist, who compared 50 pairs of images generated by the two methods, shown side-to-side and randomized. The radiologist indicated his preference for each of the 50 pairs and wrote down general impressions on the generation quality.To investigate the usability of the trained embeddings, we also experiment with combining multiple trained embeddings using composable diffusion (Liu et al., 2023) . This method allows prompting with a combination of embeddings using an AND operator in the prompt, e.g., "<cardiomegaly> AND <pleural effusion>" to generate an image with both cardiomegaly and pleural effusion present. Additionally, this method allows a weight to be given to each embedding to tune the strength of each embedding separately. In this study, we use this to experiment with interpolating between healthy and diseased states and to generate images with multiple diseases present.For classification experiments, we train ResNet-18 models, pre-trained on ImageNet (He et al., 2016; Deng et al., 2009) . Models are trained with a fixed learning rate of 10 -4 with the Adam optimizer for 6250 batches of 32 images on various combinations of real and synthetic data (Kingma and Ba, 2017) . AUC is evaluated on the validation set during training, and performance of the best validation checkpoint on the test set is reported. We apply random horizontal flipping, gaussian noise, intensity transformations, channel dropout, translation, scaling and rotation as data augmentation.The main dataset used in this work is a recently released public dataset of 1500 prostate MRI cases. This dataset was released as part of the PI-CAI (Prostate Imaging: Cancer AI) challenge, where the task is to detect clinically significant prostate cancer (Saha et al., 2022) . Each case is a 3D MRI scan featuring three modalities: T2-weighted imaging (T2W), apparent diffusion coefficient maps (ADC), and diffusion-weighted imaging (DWI). Since this work adapts a pre-trained 2D diffusion model, we extract one 2D axial slice per case. Each case is first resampled to a resolution of 3 × 0.5 × 0.5 mm and then center-cropped to a 90 × 150 × 150 mm (30 × 300 × 300 px) region. We select the median prostate slice for negative cases using the provided full prostate segmentations. We select the slice with the maximum tumor area for positive cases according to the provided tumor segmentation maps. Each slice is finally upsampled to 512 × 512 px. Each modality is encoded as one of the RGB channels when training multi-modal embeddings. The training, validation and test set of the classification experiments each consist of 100 randomly sampled negative slices and 100 randomly sampled positive slices. The embeddings are trained on the training set.CheXpert is a large public dataset of 224,316 chest radiographs, with corresponding labels for 14 different observations (Irvin et al., 2019) . Since we explicitly investigate compositional prompting with the learned embeddings, we only sample images with a single class present. Specifically, we sample 100 AP-view radiographs to train embeddings for the following four observations: No Finding (healthy), Cardiomegaly, Pleural Effusion and Pneumonia. Each radiograph is first cropped to non-zero borders. Then, the longest edge is resized to 512 px, while keeping the aspect ratio fixed. Finally, the image is zero-padded to a square resolution of 512 × 512 px. The training, validation, and test set for the classification experiments each consist of 100 healthy and 100 cardiomegaly samples.PatchCamelyon is a public dataset of 327,680 96×96 px patches extracted from histopathology whole-slide images of lymph node sections, originally released as part of the Camelyon16 challenge (Veeling et al., 2018; Ehteshami Bejnordi et al., 2017) . Each patch has a corresponding binary label indicating the presence of metastatic tissue. We randomly select 100 negative and 100 positive patches for the training set. We use the official validation and testing splits of 32,768 cases each. All images are upsampled to 512 × 512 px. 
adapting ti parameters to medical imaging: 
 All embeddings in this section were trained using 2D T2-weighted healthy prostate slices. T2-weighted images clearly show the anatomy and are, therefore, easiest to judge qualitatively. Table 1 shows the FID and MFID scores after varying the number of sampling steps, CFG scale, embedding size, and the number of training cases relative to our final configuration used in the remainder of the paper: embedding size of 64 vectors per token, 100 cases per class, 100 sampling steps and a CFG scale of 2.In general, we find that the FID and MFID scores identify general trends, but that they are not optimal metrics to judge generation quality and have sizable error margins (see Appendix A). For this reason, optimal parameters were chosen by inspecting generation results visually as well. Figure 5 in Appendix B shows the effect of the parameters studied in this section visually on a single random seed. For reference, Appendix C shows that directly generating images without applying textual inversion, by prompting the pre-trained model to generate prostate MRI scans, results in highly unrealistic images. Images generated by the fine-tuned StyleGAN3 model achieved an FID score of 53, and an MFID score of 0.12, substantially lower than those shown in Table 1 . However, in the blinded head-to-head comparison, the expert radiologist preferred the images generated by Stable Diffusion (36/50 images, 72%). There were more anatomically incorrect images generated by StyleGAN3, and often, the images had low contrast or were very dark. Similar to Section 4.1, this indicates that FID is not a particularly informative metric for comparing two architectures in a medical setting. Sets of 16 randomly generated images by both Stable Diffusion and StyleGAN3 are included in Appendix D and E, respectively.In this section, we experiment using synthetic data to train classification models on multimodal prostate MRI, chest X-ray and histopathology images. Embeddings are trained on two sets of 100 cases, with only negative or only positive cases. With these embeddings, up to 1000 cases for each class are generated, and combinations of real and synthetic data are used to train classification models. Similar to before, we perform more extensive experiments with multi-modal prostate MRI. Results are shown in Table 2 , showing that for prostate MRI augmenting the 200-case training set with 2000 synthesized cases leads to a 2% improvement in AUC, from 0.78 to 0.80. These 2000 synthesized cases are based on embeddings trained with the same 200-case set used to train the classification models. This shows that generated cases can add non-trivial variation to the data distribution and that the embedding does not simply reproduce training cases. Furthermore, models trained with only synthetic cases do not see a large drop in performance, indicating that the synthetic cases are diagnostically accurate. To confirm visual results from section 4.1, classification models trained with synthetic cases generated with embeddings trained on 10 cases instead of 100 show a dramatic drop in performance. This confirms that more cases are needed for Textual Inversion on medical data. For cardiomegaly, however, including extra synthesized cases during training is hardly an added benefit. For histopathology, adding synthesized cases results in a performance drop of about 1%, which may indicate that synthetic cases are less useful for improving models that already attain high performance.In this section, we give preliminary evidence that composable diffusion works for medical data in two examples. In Figure 2 , the disease state is gradually increased from healthy to diseased. The tumors in the prostate example become gradually more prominent (darker on ADC, brighter on DWI). Appendix F includes a more extensive figure. In Figure 3 , multiple conditions are progressively added to a single healthy example. From a healthy image, pleural effusion, pneumonia, and cardiomegaly are added to the prompt for a single random seed. For the image with all three diseases, we gave each embedding a strength of 0.5 and found that increasing the CFG scale to 3 works better.This section demonstrates the potential of inpainting to control disease location precisely. Starting from a healthy example, a portion of the image is masked. The diffusion model denoises the masked part of the image, conditioned on a specific disease embedding. In Figure 4 , the same healthy prostate example is masked in two locations with a different mask size. When inpainting conditioned on the positive embedding, this generates tumors at those locations of corresponding sizes. Similar to Section 4.4, this allows generating examples with specific disease appearance and could be useful for generating cases with rare tumor locations.  
conclusion: 
 In this paper, we use Textual Inversion to demonstrate the adaptability of pre-trained latent diffusion models across various medical modalities. High-quality images can be generated using embeddings trained on 100 examples on a single consumer-grade GPU. Our showcased applications include enhancing diagnostic models in low-data scenarios by incorporating synthetic cases during training, simulating disease progression, and generating images with specific disease appearances. While a dedicated diffusion model trained on a large captioned medical dataset would likely yield superior results, our findings are promising for institutions with limited computational resources. This approach is particularly relevant for rare diseases where collecting large datasets is impractical. It remains viable and compatible with medically pre-trained models, including 3D models. Finally, the small file size of the trained embeddings may facilitate the sharing of medical information with reduced privacy concerns.CFG scale of 1 results in the lowest FID score in Table 1 , visually, the results are much worse, featuring inaccurate general anatomy. A high CFG scale (e.g. 5 in Figure 5 ) also leads to bad results, showcased here by the simplified structure inside the prostate and a curious fractured pelvic bone. The difference between CFG scale 2 and 3 is not that large, but upon manual inspection, we find that a CFG scale of 2 gives better generations overall, as seems to be confirmed by the lower FID score in Table 1 . The embedding size is optimally chosen to be large, with sizes 8 and 16 showing inaccurate generation, particularly of the bladder. Although size 32 looks better, the structure of the prostate itself is not nearly as good as generated by the size 64 embedding. Finally, the impact of the amount of training cases seems to trump all other settings, where 5 and 10 cases produce very unrealistic images. The embedding trained with 100 cases generates images with the most realistic prostate structure.Figure 7 : Four generated images when prompting a pre-trained Stable Diffusion model with "a T2-weigthed MRI scan of a prostate" generated with a prompt like "0.25*<healthy> AND 0.75*<cardiomegaly>". This seems to work well across the modalities studied in this paper: the tumors in the prostate example become gradually more prominent (darker on ADC, brighter on DWI); the heart in the cardiomegaly example appears to grow from left to right; the tissue in the lymph node metastasis example becomes gradually more abnormal. 
 
 Appendix:  


Decision: The main strength of this paper lies in the extensive experiments, which demonstrates the efficacy of fine-tuning diffusion models on low computational budget. 

However, the methodological contributions are rather limited, since this paper takes an existing model and simply demonstrates it in the case of different fine-tuning scenarios. I also emphasise that this paper clearly lacks a section to describe the employed latent diffusion model. This greatly reduces interest for non-experts in latent diffusion models, makes ablations difficult to parse, and decreases insights in the comparison with StyleGAN3.  

Overall, I believe this work will lead to interesting discussions and I recommend acceptance. Finally, this paper is borderline for oral: despite the good scores, only 1 reviewer recommended oral, which is due to the paper being methodologically a bit light. I am personally leaning towards poster, but I'll leave the final decision to the program chairs.

Reviews: review: paper_summary: paper_summary: This paper explores using pre-trained latent diffusion models across various medical modalities through Textual Inversion; this process finds a vector in the text embedding space that represents the concept. Also, explored the usage of synthetic data to improve the performance of medical classification tasks, especially with limited computational sources. The authors have validated the models on MRI, ChestX-ray, and Histopathology datasets.
main_review: main_review: he paper by the authors presents an approach to enhance model performance with limited data. By utilizing diffusion models and training text embeddings using textual inversion, the authors have created a technique that results in more effective and accurate models, even with smaller datasets. I would suggest that you accept the paper without making any changes to my previous decision.
strength_weakness: strength_weakness: I would like to express my appreciation for the well-written paper that provided a clear and concise explanation of the concepts. I am pleased to inform you that, based on my thorough review, no revisions or modifications are required.
questions: questions: NA
paper_summary: paper_summary: The authors explore the application of textual inversion to a pre-trained diffusion model for natural images (stable diffusion) to enable the model to produce samples of different medical datasets. They explore using small numbers of samples for each dataset to perform textual inversion. The authors show these adapted models can be used to generate realistic-looking data, and samples generated from the model help to improve the accuracy of trained classifiers. The authors also show preliminary results with inpainting and composing embeddings to produce images with multiple pathologies.
main_review: main_review: The authors have done a good job responding to my comments. In particular, I appreciate the new results investigating stable diffusion's native ability to generate prostate MRI, and the willingness to better explore the FID results with bootstrapping and using medical backbones. I'm upgrading my rating.
strength_weakness: strength_weakness: The work is quite a light exploration of a few different applications, with no one area explored in any great detail. 

It seems the authors tested their classification models on 200 images per dataset, but there is much more data available in these datasets. Why not test on more data?
questions: questions: How does stable diffusion do without any application of textual inversion?

Improvement of FID scores, either with more samples / a medical specific network.
paper_summary: paper_summary: In the text-to-image generation problem, the performance of existing works always suffers from limited availability of large pre-trained dataset. The authors show that pre-trained Stable Diffusion models can assist in medical image using textual inversion. The training is very light weighted and fast.
main_review: main_review: I would like to extend my thanks to the authors for their comprehensive classification work. The incorporation of further experiments and the broadening of classifications have thoroughly addressed all of my concerns.

====update===== Overall, I would like to vote for acceptance.
strength_weakness: strength_weakness: - Is there any possibility to adopt the proposed method to a 3d model structure? Since the proposed method is tested on an MRI dataset which is originally a 3D dataset.
- The authors have implied in section 3.1 that the input sentence prompt shows like ‘a photo of  <embedding>’. Is there any possibility that a longer description of the image is passed, such as ‘a photo of a liver where an HCC is located at upper left, with size 3mm’. Would the model render a corresponding result? This could be more complicated than ”<cardiomegaly> AND <pleural effusion>”.
- What is the training time and training step of the diffusion model? Since the diffusion model always suffers from long training and inference time.
- Could you please show some other metric results, such as PSNR and SSIM?
- Does the sample size matter? Since the PatchCamelyon dataset is not originally 512\*512 and the authors upsampled them, while chest Xray is originally 512\*512.
- The authors seem to have missed some relevant literature. Specifically, they miss out on several relevant citations, e.g. “MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation”, and “Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective”. These relevant papers should be discussed.
questions: questions: - Is there any possibility to adopt the proposed method to a 3d model structure? Since the proposed method is tested on an MRI dataset which is originally a 3D dataset.
- The authors have implied in section 3.1 that the input sentence prompt shows like ‘a photo of  <embedding>’. Is there any possibility that a longer description of the image is passed, such as ‘a photo of a liver where an HCC is located at upper left, with size 3mm’. Would the model render a corresponding result? This could be more complicated than ”<cardiomegaly> AND <pleural effusion>”.
- What is the training time and training step of the diffusion model? Since the diffusion model always suffers from long training and inference time.
- Could you please show some other metric results, such as PSNR and SSIM?
- Does the sample size matter? Since the PatchCamelyon dataset is not originally 512\*512 and the authors upsampled them, while chest Xray is originally 512\*512.
- The authors seem to have missed some relevant literature. Specifically, they miss out on several relevant citations, e.g. “MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation”, and “Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective”. These relevant papers should be discussed.

Comments: comment: > Could you please show some other metric results, such as PSNR and SSIM? 

PSNR and SSIM are metrics typically used to compare pairs images which should look exactly the same. In our case, since we have a generative model, generated images will never be identical to training or validation examples. In fact, this is a desirable property, since it allows the model to enrich small dataset and potentially improve model training as shown in Section 4.3. 

The best one can do is to compare distributions of real and generated images to each other to see if they are similar, which is why we employed the Fréchet Inception Distance, which is the standard metric for image synthesis. However, we realize that this metric is suboptimal and have therefore, also based on suggestion by another reviewer, expanded our validation approach: 

- FID is now calculated by comparing 1000 generated images to 1000 real images, instead of 100. 
- We estimate 95% confidence intervals with bootstrapping in Appendix A to give an impression of the reliability of the scores. 
- We introduce an extra metric, MFID, which is the FID score when using RadImageNet as a feature extractor, instead of an ImageNet pre-trained network. 

We hope that the combination of these extended metrics and the classification experiments convinces the reviewer that the generated images are realistic. 

> Does the sample size matter? Since the PatchCamelyon dataset is not originally 512*512 and the authors upsampled them, while chest Xray is originally 512*512. 

The main reason we upsampled the PatchCamelyon patches to 512*512 is that this was the resolution on which the Stable Diffusion v1.5 checkpoint was originally trained. In principle it is also possible to generate images of different resolution with the same model, but to ensure maximum model performance we kept most settings, including resolution, as close as possible to the pre-training scenario.  

> The authors seem to have missed some relevant literature. Specifically, they miss out on several relevant citations, e.g. “MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation”, and “Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective”. These relevant papers should be discussed. 

This field is moving very rapidly, and we thank the reviewer for pointing us towards some interesting recent work. We have included “MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation” in Section 2 discussing related work on generative imaging, as it discusses mask-conditioned imaging, which relates well to the inpainting scenario in Figure 4. 

The scope of our paper is mostly generative AI, specifically with diffusion models. Given the strict 8-page limit, we have chosen to not discuss “Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective”. This paper, as we understand it, is more within the scope of semi-supervised/transfer learning, which relates to our classification experiment and mentioned application of learning with few images/labels present. However, there is no generative modelling in the paper, and discussing how our work relates to the state-of-the-art semi-supervised work in the medical field would require a sizable extra paragraph with many relevant citations, for which there is unfortunately no space. We hope the reviewer understands our choice.
We thank the reviewer for the thorough evaluation of our work, and the suggestions for improvement. Please see our detailed implementations and responses, point by point, below. 

> Is there any possibility to adopt the proposed method to a 3d model structure? Since the proposed method is tested on an MRI dataset which is originally a 3D dataset. 

In general, textual inversion can be applied to any text-to-image diffusion model, including 3D models. In this paper, we use Stable Diffusion, to show that even with a model pre-trained on natural images it is possible to fine-tune towards diagnostically relevant medical images. A requirement for this to work in 3D, would be a medically pre-trained 3D text-to-image latent diffusion model, which to the best of our knowledge is not yet openly available. Given such a model, we expect our approach to potentially learn faster and better embeddings, since the model has much more domain knowledge embedded. We have added a short line to the conclusion to attend the reader that our approach is compatible with 3D models. 

> The authors have implied in section 3.1 that the input sentence prompt shows like ‘a photo of <embedding>’. Is there any possibility that a longer description of the image is passed, such as ‘a photo of a liver where an HCC is located at upper left, with size 3mm’. Would the model render a corresponding result? This could be more complicated than ”<cardiomegaly> AND <pleural effusion>”. 

This is indeed possible and the authors of the Textual Inversion paper have experiments showcasing this possibility, but they noted that it gave worse results than simply prompting with the embedding name (Figure 10 in [1]). We did not do it in our paper, because it requires each sample in the training set to have an individual label describing the sample characteristics. For simplicity, we simply trained two embeddings per modality, one for the negative class and one for the positive class, prompting the model only with the embedding name, which we now clarified at the end of section 3.1, page 4. As a result, this approach can be applied to any dataset for which class labels are available, which is the most common scenario. 

Suppose for the PI-CAI data, we would have labels per scan describing location, appearance and size, the embedding could have been trained by prompting the model as you describe, e.g.: 

Scan 1: “a <prostate_mri_negative> with a benign lesion in the transitional zone” 

Scan 2: “a <prostate_mri_positive> of 5mm in the peripheral zone with irregular tumor boundary” 

Scan 3: ... 

Given our datasets and their lack of descriptive labels, we did not perform new experiments to test this approach. 

[1] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and 
Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation 
using Textual Inversion, August 2022 

> What is the training time and training step of the diffusion model? Since the diffusion model always suffers from long training and inference time. 

We train each embedding with a constant learning rate of 0.005 for 50,000 steps with a batch size of 1. This takes approximately 4 hours on an NVIDIA RTX2070. Since only the vector embedding is trained, few weight parameters are updated. Training the entire diffusion model from scratch indeed takes much longer, since all weights have to be optimized and the training datasets are typically orders of magnitude larger. Textual inversion leverages these big training efforts, by only optimizing a small set of weights on top of a pre-trained model. We have added this information to Section 3.1.
We thank the reviewer for the thorough evaluation of our work, and the suggestions for improvement. Please see our detailed implementations and responses, point by point, below. 

> The work is quite a light exploration of a few different applications, with no one area explored in any great detail. 

We have indeed chosen to focus on the wide applicability of textual inversion, and fine-tuning diffusion models in general, to showcase the potential applications in the medical field. We hope to provide enough evidence, for instance with the classification experiments, to convince readers of the validity and usefulness of our approach. 

> It seems the authors tested their classification models on 200 images per dataset, but there is much more data available in these datasets. Why not test on more data? 

We thank the reviewer for highlighting this potential improvement in the validation design. Our initial experiments were performed on the PI-CAI (prostate) dataset, which contains 1500 images of which 425 are cancer positive. For simplicity, and to avoid class imbalance, we sampled 50/50 negative/positive sets from this set. Given the total amount of 425 positive cases, we chose 100 samples for each class, for the train, validation and test sets. Later, we extended the classification experiments to CheXpert (cardiomegaly) and PatchCamelyon (histopathology) and kept the same sampling scheme. 

For CheXpert, out of the total 224,316 scans, there are only 671 which feature only cardiomegaly and no other disease, which is of the same order as the PI-CAI dataset. 

For PatchCamelyon, there are indeed much more patches available than we used in the paper. We have therefore re-run the experiments, using the same 100/100 negative/positive set for traing, but using the full official validation and test sets of 32,768 cases each (50/50 negative/positive). 

> I didn't get a sense of whether stable diffusion could be used to produce medical images before the application of textual inversion. I think it would be helpful to do some text prompting of the model to try to generate the classes and see how those images look - I suspect given the large dataset it was trained on there will have been some images present and it may do an OK job. 

This is indeed a good point to clarify in the paper. The first paper by Chambon we cite, does this for chest X-ray, and they found that Stable Diffusion models were able to produce images that resemble realistic chest X-ray scans when prompting with “a photo of a lung xray” [1]. We did the same and found that for prostate MRI this is not the case, with images generated by the diffusion model not being realistic. We have included example generations with prompts “a prostate MRI scan” and “a T2-weigthed MRI scan of a prostate” in Appendix C. 

[1] Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, and Akshay Chaudhari. Adapt- 
ing Pretrained Vision-Language Foundational Models to Medical Imaging Domains, Oc- 
tober 2022 

> I'm not sure it makes sense to calculate the FID with just 100 samples. It is typically recommended to calculate on 10,000 samples or more. This may be why the authors find the FID metric to not correlate well with observed sample quality. Perhaps the authors could estimate error bars, using bootstrapping, to understand if these measures are at all informative when calculated on small datasets. 

We have taken this suggestion to heart, and have extended the amount of samples for the FID calculation to 1000. We have not extended to the suggested 10,000, since the amount of real healthy prostate MRI scans in the PI-CAI is around 1000, and to avoid excessive generation time for the various training and inference settings in Table 1.  

Also, we have included a bootstrapping estimate in Appendix A to indicate the reliability of the FID scores. Indeed, we find that the error margins have substantial overlap with values in Table 1. Since we do not use only the FID metrics to confirm our parameter choice, but also visual inspection and classification experiments, we did not opt for a rigorous statistical comparison between parameter settings. 

> Related, it might be best to use a network trained on medical images, rather than the Inception network, as the backbone when calculating the FID. The extracted features will likely be more relevant. 

To explore the difference between an ImageNet pre-trained network and a medical pre-trained network, we have included FID scores based on RadImageNet [2]. Table 1 now features extra columns with the MFID scores, which indicate the FID scores based on RadImageNet. 

[2] Xueyan Mei, Zelong Liu, Philip M Robson, Brett Marinelli, Mingqian Huang, Amish Doshi, 
Adam Jacobi, Chendi Cao, Katherine E Link, Thomas Yang, et al. Radimagenet: an 
open radiologic deep learning research dataset for effective transfer learning. Radiology: 
Artificial Intelligence, 4(5):e210315, 2022
We are glad the reviewer appreciated the paper, and that there are no big points to address. Based on the comments of the other two reviewers we have made several additions and improvements: 

- The validation and test sets for PatchCamelyon are increased from 100 cases to the official splits consisting of 32,768 cases each. 
- The FID metrics were improved by increasing the number of cases for the generated and real sets from 100 to 1000. Additionally, we added Appendix A with a note on its reliability. 
- We added an extra FID metric, MFID, which is the FID score using RadImageNet as a feature extractor, instead of an ImageNet pre-trained network.
- We have included prostate MRI generations by Stable Diffusion, without applying textual inversion, in Appendix C.