Title: Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Synthetic clinical data

Machine learning for healthcare
Abstract: Generative models capable of precisely capturing nuanced clinical features in medical images hold great promise for facilitating clinical data sharing, enhancing rare disease datasets, and efficiently synthesizing (annotated) medical images at scale. Despite their potential, assessing the quality of synthetic medical images remains a challenge. While modern generative models can synthesize visually-realistic medical images, the clinical plausibility of these images may be called into question. Domain-agnostic scores, such as FID score, precision, and recall, cannot incorporate clinical knowledge and are, therefore, not suitable for assessing clinical sensibility. Additionally, there are numerous unpredictable ways in which generative models may fail to synthesize clinically plausible images, making it challenging to anticipate potential failures and design automated scores for their detection. To address these challenges, this paper introduces a pathologist-in-the-loop framework for generating clinically-plausible synthetic medical images. Our framework comprises three steps: (1) pretraining a conditional diffusion model to generate medical images conditioned on a clinical concept, (2) expert pathologist evaluation of the generated images to assess whether they satisfy clinical desiderata, and (3) training a reward model that predicts human feedback on new samples, which we use to incorporate expert knowledge into the finetuning objective of the diffusion model. Our results show that human feedback significantly improves the quality of synthetic images in terms of fidelity, diversity, utility in downstream applications, and plausibility as evaluated by experts. We also demonstrate that human feedback can teach the model new clinical concepts not annotated in the original training data. Our results demonstrate the value of incorporating human feedback in clinical applications where generative models may struggle to capture extensive domain knowledge from raw data alone.

Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedback

Abstract

Generative models capable of capturing nuanced clinical features in medical images hold great promise for facilitating clinical data sharing, enhancing rare disease datasets, and efficiently synthesizing annotated medical images at scale. Despite their potential, assessing the quality of synthetic medical images remains a challenge. While modern generative models can synthesize visually-realistic medical images, the clinical validity of these images may be called into question. Domainagnostic scores, such as FID score, precision, and recall, cannot incorporate clinical knowledge and are, therefore, not suitable for assessing clinical sensibility. Additionally, there are numerous unpredictable ways in which generative models may fail to synthesize clinically plausible images, making it challenging to anticipate potential failures and manually design scores for their detection. To address these challenges, this paper introduces a pathologist-in-the-loop framework for generating clinically-plausible synthetic medical images. Starting with a diffusion model pretrained using real images, our framework comprises three steps: (1) evaluating the generated images by expert pathologists to assess whether they satisfy clinical desiderata, (2) training a reward model that predicts the pathologist feedback on new samples, and (3) incorporating expert knowledge into the diffusion model by using the reward model to inform a finetuning objective. We show that human feedback significantly improves the quality of synthetic images in terms of fidelity, diversity, utility in downstream applications, and plausibility as evaluated by experts. We also show that human feedback can teach the model new clinical concepts not annotated in the original training data. Our results demonstrate the value of incorporating human feedback in clinical applications where generative models may struggle to capture extensive domain knowledge from raw data alone.

1 Introduction

Diffusion models have recently shown incredible success in the conditional generation of highfideltiy natural, stylized and artistic images [1-6]. The generative capabilities of these models can be leveraged to create synthetic data in application domains where obtaining large-scale annotated datasets is challenging. The medical imaging field is one such domain, where there is often a difficulty in obtaining high-quality labeled datasets [7]. This difficulty may stem from the regulatory hurdles that impede data sharing [8], the costs involved in getting experts to manually annotate images [9], or the natural scarcity of data in rare diseases [10]. Generative (diffusion) models may provide a partial solution to these problems by synthesizing high-fidelity medical images that can be easily shared among researchers to replace or augment real data in downstream modeling applications [11, 12].

UC Berkeley and UCSF

Implausible

Step 1.

Pathologist feedback collection

Step 2.

Clinical plausibility reward modeling

Step 3.

Clinically-informed finetuning

Cell size

Nucleus shape

Chromatin pattern

Cytoplasm color

Inclusions

Synthetic dataset

Clinical plausibility checklist

✅

✅

✅

✅

❌

❌

Pre-trained

diffusion model

Pathologist inspection

Synthetic images

.

Pathologist

annotations

. .

⤬

Plausible

Implausible

⤬

✓

Reward

model

Reward model training data

20%

Implausible

New synthetic

image

80%

Plausible

.

.

.

Synthetic dataset

Pre-trained

diffusion model

Finetuning objective

Clinical plausibility scores

0.2

0.7

0.1

Reward model

0.8

0.3

0.9

X

What sets medical image synthesis apart from image generation in other fields? Unlike mainstream generative modeling applications that prioritize visually realistic or artistically expressive images, synthetic medical images require a different approach. They must be grounded in objective clinical and biological knowledge, and as such, they leave no room for creative or unrestricted generation. Given that the ultimate goal of synthetic medical images is to be used in downstream modeling and analysis, they must faithfully reflect nuanced features that represent various clinical concepts, such as cell types [13], disease subtypes [7], and anatomies [14]. Off-the-shelf image generation models are not capable of recognizing or generating clinical concepts, rendering them unsuitable for generating plausible medical images without further adaptation [15, 16]. Therefore, our aim is to develop a framework for generating synthetic medical images that not only exhibit visual realism but also demonstrate biological plausibility and alignment with clinical expertise.

One way to generate synthetic medical images is to finetune a pretrained 'foundation' vision model, such as Stable Diffusion, that has been trained on billions of natural images (such as the LAION-5B dataset [17]), using real medical images. With a sufficiently large set of medical images, we can expect the finetuned model to capture the clinical knowledge encoded in medical images. However, the sample sizes of annotated medical images are typically limited to a few thousand. When a large vision model is finetuned on such a dataset using generic objective functions (such as the likelihood function), the model may capture only the generic features that make the medical images appear visually realistic, but it may miss nuanced features that make them biologically plausible and compliant with clinical domain knowledge (see examples in the next Section). Designing domain-specific objective functions for finetuning that ensure a generative model adheres to clinical knowledge is challenging. The difficulty arises from the numerous unpredictable ways in which these models can generate images that lack clinical plausibility. As a result, it is impractical to anticipate every possible failure scenario and manually construct a loss function that penalizes such instances.

Summary of contributions. In this paper, we develop a pathologist-in-the-loop framework for synthesizing medical images that align with clinical knowledge. Our framework is motivated by the success of reinforcement learning with human feedback (RLHF) in aligning the outputs of large language models (LLMs) with human preference [18, 19], and is directly inspired by [20], where human feedback was used to align the visual outputs of a generative model with input text prompts. To generate clinically-plausible medical images, our framework (outlined in Figure 1) comprises 3 steps:

Step 1: Wetrain a (conditional) diffusion model using real medical images. We then sample a synthetic dataset from the model to be evaluated by a pathologist. Each image is carefully examined, and the pathologist provides feedback on whether it meets the necessary criteria for clinical plausibility.

Step 2: We collate a dataset of synthetic images paired with pathologist feedback and train a reward model to predict the pathologist feedback, i.e., clinical plausibility, on new images.

Step 3: Finally, the reward model in Step 2 is utilized to incorporate expert knowledge into the generative model. This is achieved by finetuning the diffusion model using a reward-weighted loss function, which penalizes the generation of images that the pathologist considers clinically implausible.

Throughout this paper, we apply the steps above to the synthetic generation of bone marrow image patches, but the same conceptual framework can generalize to any medical imaging modality. We gathered pathologist feedback on thousands of synthetic images of various cell types generated by a conditional diffusion model. Then, we analyzed the impact of this feedback on the quality of the finetuned synthetic images. Our findings suggest that incorporating pathologist feedback significantly enhances the quality of synthetic images in terms of all existing quality metrics such as fidelity, accuracy of downstream predictive models, and clinical plausibility as evaluated by experts. Additionally, it also improves qualities that are not directly addressed in the pathologist evaluation, such as the diversity of synthetic samples. Furthermore, we show that human feedback can teach the generative model new clinical concepts, such as more refined identification of cell types, that are not annotated in the original training data. These results demonstrate the value of incorporating human feedback in clinical applications where generative models may not be readily suited to capture intricate and extensive clinical domain knowledge from raw data alone.

Mast cell (B1)

Neutrophil (M5)

Red blood cells (ER6)

lobes

or multilobed-shaped

Not red enough to be

recognized as ER6

Darker than

expected

2 Pathologist-in-the-Loop Generation of Synthetic Medical Images

In this Section, we provide a detailed description of our synthetic medical image generation framework. We will use a running example pertaining to single-cell images extracted from bone marrow aspirate whole slide images. Details of this setup and the dataset used in our study are provided in Section 4.

2.1 Step 1: Pathologist feedback collection

Generative model pretraining. The first step in our framework starts with training a generative model to synthesize medical images through the standard training procedure. As we discuss in more detail in Section 4, we utilized a dataset of 2,048 bone marrow image patches to train a conditional diffusion model [6]. The model was trained to generate class-conditional images, where an image class corresponds to a cell type. We conducted an exploratory analysis where we found that neither latent diffusion nor Stable Diffusion models yielded superior results compared to a customized diffusion model that we employed in this study (See Appendix A). We opted for using a class-conditional model rather than a text-conditional model as we found that existing pretrained vision-language models were not fit for capturing the scientific jargon related to bone marrow cell types.

Given a dataset of real images D r = { ( x i , c i ) } n r i =1 , where x is a medical image and c ∈ C is an image class, we train a diffusion model to generate class-conditional images through the forward process:

where ϵ t ∼ N (0 , ρ 2 ) is the noise term at time-step t , x t is the data point at time-step t , α t is the step size at t , θ is the model parameters and ∇ x log( p ( x t | x, c )) is the gradient of the log probability distribution with respect to x , conditioned on the original data x and class c . Once the model is pretrained, we sample a synthetic dataset D s = { ( ˜ x j , ˜ c j ) } n s j =1 by first sampling a class ˜ c from C , and then sampling a medical image ˜ x conditioned on the class ˜ c through the reverse diffusion process.

Pathologist evaluation. Each image in the synthetic dataset D s = { ( ˜ x j , ˜ c j ) } n s j =1 generated by the pretrained model is inspected by an expert pathologist to assess its clinical plausibility. The objective of this evaluation is to identify the specific inaccuracies in the synthetic data that can only be identified by an expert, and provide feedback for the model to refine its synthesized images in the finetuning step.

When a model is trained with only a modest number of real image samples, it may generate bone marrow image patches that look visually appealing but are not biologically plausible. In Figure 2, we present 8 synthetic images sampled from the conditional diffusion model, which correspond to four different cell types. Each of these images achieves high precision and fidelity scores individually, but they also have biological implausibilities such as inaccurate cell coloring or nucleus shapes. Therefore, models that prioritize visual features without considering biological knowledge may miss important clinical features required for synthetic images to be useful for downstream analysis. Generic evaluation scores (e.g., [21, 22]) cannot diagnose these failures because they also lack biological domain knowledge. By incorporating feedback from pathologists, we can refine the generative model by identifying biological information that is missed by the pretrained model.

The expert pathologist examined each synthetic image and provided a feedback score on its biological plausibility. The evaluation typically involved inspecting the image and checking 7 aspects that contribute to its perceived plausibility (Table 1). These aspects pertain to the consistency of the shapes, sizes, patterns and colors of the contents of a synthetic bone marrow image ˜ x with the cell type ˜ c .

Clinical plausibility criteria

(1) Cell size

(5) Chromatin pattern

(2) Nucleus shape and size

(6) Inclusions

(3) Nucleus-to-cytoplasm ratio

(4) Cytoplasm color and consistency

Among the determinants of plausibility is the cell size-different cells have different sizes, e.g., Lymphocytes are generally smaller than Monocytes or Neutrophils. Nucleus shape and size also depend on the cell type, e.g., Band Neutrophils have a horseshoe-shaped nucleus, whereas Segmented Neutrophils have a multi-lobed nucleus. Chromatin patterns within the nucleus are dense and clumped in Lymphocytes, while in Myeloid cells they are diffuse and fine. The number, size and color of granules also contribute to plausibility. Detailed explanation of all criteria is provided in the Appendix.

Note that we have full control over the number of synthetic images n s , i.e., we can sample an arbitrary number of synthetic images from the conditional diffusion model. The key limiting factor on n s is the time-consuming nature of the feedback collection process. To enable scalable feedback collection, we limited our study to binary feedback, i.e., the pathologist flagged a synthetic image as 'implausible' if they found a violation of any of the criteria in Table 1 upon visual inspection. We collected these binary signals and did not pursue a full checklist on all plausibility criteria for each synthetic image.

The output of Step 1 is an annotated dataset D s = { ( ˜ x j , ˜ c j , ˜ y j ) } n s j =1 , where ˜ y j ∈ { 0 , 1 } is the pathologist feedback on the j -th synthetic image, where ˜ y j = 1 means that the image is implausible.

2.2 Step 2: Clinical plausibility reward modeling

We conceptualize the pathologist as a 'labelling function' Γ : X × C → { 0 , 1 } that maps the observed synthetic image ˜ x and declared cell type (class) ˜ c to a binary plausibility score. In Step 2, we model the 'pathologist' by learning their labelling function Γ on the basis of their feedback annotations.

To train a model Γ that estimates the pathologist labelling function, we construct a training dataset that comprises a mixture of real and synthetic images as follows:

Synthetic images

We construct a dataset D Γ s = { ( ˜ x j , ˜ c j , ˜ y j ) } n s j =1 comprising the synthetic images and corresponding pathologist feedback collected in Step 1.

Real images

We build a dataset D Γ r = { ( x i , ˜ c i , y i ) } n r i =1 comprising the real images and pseudo-labels defined as y i = 1 { c i = ˜ c i } , where ˜ c i ∼ Uniform (1 , 2 , . . . , |C| ) .

̸

We combine both datasets D Γ = D Γ r ∪ D Γ s to construct a training dataset for the model Γ . The real dataset D Γ r is built by randomly permuting the image class and assigning an implausibility label of 1 if the permuted class does not coincide with the true class. We use the real dataset to augment the synthetic dataset with the annotated pathologist feedback. By augmenting the datasets D Γ r and D Γ s , we teach the model Γ to recognize two forms of implausibility in image generation: (i) instances where the synthetic image looks clinically plausible but belongs to a wrong cell type (i.e., training examples

in D Γ r ), and (ii) instances where the synthetic image is visually consistent with the correct cell type but fails to meet some of the plausibility criteria in Table 1 (i.e., subset of the training examples in D Γ s ). We call the resulting model Γ a clinical plausibility reward model. Using the augmented feedback dataset D Γ , we train the reward model by minimizing the mean square error as follows:

where λ r is a hyper-parameter that controls the contribution of real images in training the reward functions, and ϕ is the parameter of the reward model.

2.3 Step 3: Clinically-informed finetuning

In the final step, we refine the diffusion model by leveraging the pathologist feedback. Specifically, we incorporate domain knowledge into the model by utilizing the reward model Γ in the finetuning objective. Following [20], we use a reward-weighted negative log-likelihood (NLL) objective, i.e.,

to finetune the conditional diffusion model, where β r is a hyper-parameter and ̂ ϕ is the reward model parameters obtained by minimizing (2) in Step 2. The finetuning objective in (3) incorporates the pathologist knowledge through the reward model, which predicts the pathologist evaluation of the synthetic images that the model generates as it updates its parameters θ . The reward-weighted objective penalizes the generation of images that do not align with the pathologist preferences, hence we expect that the finetuned model will be less likely to generate clinically implausible synthetic images.

2.4 Bonus step: Feedback-driven generation of new clinical concepts

Besides refining generative models for clinical plausibility, pathologist feedback can be used to incorporate novel clinical concepts into the generative process that were not initially labeled in the real dataset. This could allow generative models to continuously adapt in changing clinical environments. For instance, in our bone marrow image generation setup, pathologist feedback can refine image generation by introducing new sub-types of the original cell types in C , as illustrated in Figure 3.

Instead of collecting pathologist feedback that is limited to clinical plausibility, we also collect their annotation of new cell sub-types (e.g., segmented and band variants of Neutrophil cell types). Next, we train an auxiliary model Γ ρ ( x ) with parameter ρ to classify the new sub-types based on the pathologist annotations. Finally, we finetune the conditional diffusion model through a combined loss function, i.e., L ( θ, ̂ ϕ ) + L ( θ, ̂ ρ ) , that incorporates the two forms of pathologist feedback, i.e., annotations of new cell types and clinical plausibility.

Band Neutrophil

Segmented Neutrophil

Pathologist

feedback

Synthetic Neutrophils

2.5 Pathologist feedback vs. Automated feedback

To asses the added value of human feedback, we consider a baseline where the generative model is supplemented with automatically generated feedback on clinical plausibility. To this end, we implement a baseline based on classifier-guided diffusion [23], where a classifier serves as an automatic feedback signal that deems a synthetic image implausible if it does not match the corresponding cell type. For this baseline, we train an auxiliary classifier to predict the cell type c of an image x , and then we incorporate the gradient of the log-likelihood of this classifier in the training objective as described in [23] (See the Supplementary material for implementation details).

3 Related Work

Before delving into the experimental results, we discuss three strands of literature that are related to our synthetic data generation framework. These include: generative modeling for synthetic clinical data, evaluation of generative models and learning from human feedback.

Generative modeling of synthetic medical images. The dominant approach for synthesizing medical images is to train or finetune a generative model, such as a Variational Autoencoder (V AE) [24], a Generative Adversarial Network (GAN) [25], or a diffusion model [6, 23], using a sufficiently large sample of images from the desired modality. Owing to their recent success in achieving state-of-the-art results in high-fidelity image synthesis [23], diffusion probabilistic models have become the model of choice for medical image synthesis applications [12, 26-29]. In [12], the Stable Diffusion model-an open-source pretrained diffusion model-was used to generate synthetic X-ray images, and in [28] it was shown that diffusion models can synthesize high-quality Magnetic Resonance Images (MRI) and Computed Tomography (CT) images. [27] used latent diffusion models to generate synthetic images from high-resolution 3D brain images. All of these models are trained with the standard likelihood objective and the synthetic images are typically evaluated through downstream classification tasks or generic, domain-agnostic metrics for image fidelity. To the best of our knowledge, none of the previous studies have explored a human-AI collaboration approach to synthetic image generation or incorporated clinical knowledge into generative models of medical images.

Evaluation of synthetic images in the medical domain and beyond. Unlike discriminative modeling (i.e., predictive modeling) where model accuracy can be straightforwardly evaluated by comparing the model predictions with ground-truth labels in a testing set, evaluating the quality of generative models can be quite challenging since we do not have a 'ground-truth' for defining what makes a synthetic sample is of high or low quality. Devising a generic score to evaluate a generative model can be tricky since there are many potential modes of failure [22]. Consequently, it is essential to design robust multidimensional scores that capture the most relevant failure modes for a given application.

Recently, there have been various attempts at defining domain-specific scores [30-32] as well as generic scores for evaluating the quality of synthetic images. Examples include the FID score which is based on a distributional distance between real and synthetic images [33]. Other examples for samplelevel evaluation metrics include the precision and recall metrics [21] which check if synthetic data resides in the support of the real data distribution. However, these scores do not encode clinical domain knowledge, which is critical for identifying failures in generating clinically meaningful images. Traditional scores of medical image quality include signal- and contrast-to-noise ratio [34-36], mean structural similarity [37]. These scores are typically applied to real images and cannot be repurposed to judge the generative capacity of a synthetic data model in a meaningful way. The lack of an automated score for detecting clinically implausible synthetic medical images is a key motivation for our work. We believe that the most reliable way to assess the quality of a synthetic image is to have it evaluated by an expert pathologist. From this perspective, the reward model Γ in Section 2.2 can be thought of as a data-driven score for image quality trained using pathologist evaluations.

Learning from human feedback. The success of many modern generative models can be attributed in part to finetuning using feedback solicited from human annotators. The utilization of human feedback in model finetuning is very common in natural language processing applications, particularly in finetuning of large language models (LLMs). Examples for applications were human feedback was applied include translation [38], web question-answering [39] and instruction tuning [40-42]. The key idea in these applications is that by asking a human annotator to rate different responses from the same model, one can use such annotations to finetune the model to align with human preference. Similar ideas have been applied to align computer vision models with human preferences [20, 43-46]. In the context of our application, the goal is to align the outputs of a generative model with the preferences of pathologists, which are naturally aligned with clinical domain knowledge. Our finetuning objective builds on the recent work in [20] and [45], which use human feedback to align text-prompts with generated images using a reward-weighted likelihood score.

4 Experiments

In this Section, we conduct a series of experiments to evaluate the utility of pathologist feedback in improving the quality of synthetic medical images. In the next Subsection, we start by providing a detailed description of the single-cell bone marrow image dataset used in our experiments.

4.1 Bone marrow cells dataset

In all experiments, we used a dataset of hematopathologist consensus-annotated single-cell images extracted from bone marrow aspirate (BMA) whole slide images. The images were obtained from the

clinical archives of an academic medical center. The dataset comprised 2,048 images, with the images evenly distributed across 16 morphological classes (cell types), with 160 images per class (Table 2). These classes encompass varied cell types found in a standard bone marrow differential. The dataset covers the complete maturation spectrum of Erythroid and Neutrophil cells, from Proerythroblast to mature Erythrocyte and from Myeloid blast to mature Neutrophils, respectively. The dataset also differentiates mature Eosinophils with segmented nuclei from immature Eosinophils and features Monocytes, Basophils, and Mast cells. Bone marrow cell counting and differentiating between various cell types is a complex task that poses challenges even for experienced hematologists. Hence, we expect pathologist feedback to significantly improve the quality of bone marrow image synthesis.

4.2 Synthetic data generation and pathologist feedback collection

We used a conditional diffusion model trained on real images (64 × 64 pixels in size) to generate synthetic image patches. Training was conducted using 128 images per cell type, with 32 images per cell type held out for testing and evaluating all performance metrics. For the reward model Γ( x, c ) , we used a ResNeXt-50 model [47] pre-trained on a cell type classification task to obtain embeddings for individual images, and then concatenated the embeddings with one-hot encoded identifiers of image class (cell type) as inputs to a feed-forward neural network that predicts clinical plausibility. Further details on model architectures and selected hyper-parameters are provided in the Appendix.

We collected feedback from an expert pathologist on 3,936 synthetic images generated from the diffusion model. The pathologist identified most of these images as implausible-the rate of implausible images was as high as 85% for some cell types (e.g., Basophil cells, see Table 2). After training the reward model using pathologist feedback, we finetune the diffusion model as described in Section 2.

4.3 Results

Expert evaluation of synthetic data quality. To evaluate the impact of pathologist feedback on the generated synthetic data, we created two synthetic datasets: a sample from the diffusion model (before finetuning with pathologist feedback) and a sample from the finetuned version of the model after incorporating the pathologist feedback. Each dataset comprised 400 images (25 images per cell type). An expert pathologist was asked to evaluate the two samples and label each image as plausible or implausible (in a manner similar to the feedback collection process).

Table 3 lists the fraction of clinically plausible images per cell type for the two synthetic datasets (before and after finetuning using the pathologist feedback) as evaluated by an expert hematopathologist. As we can see, the pathologist feedback leads to a significant boost in the quality of synthetic images across all cell types, increasing the average rate of clinical plausibility from 0.21 to 0.75. Note that in this experiment, the human evaluator emulates the reward function Γ( x, c ) . Hence, the improved performance of the finetuned model indicates success in learning the pathologist preferences.

Evaluating synthetic data using fidelity & diversity scores. In addition to expert evaluation, we also evaluated the two synthetic datasets generated in the previous experiment using standard metrics for evaluating generative models. We considered the Precision, Recall and Coverage metrics [21, 22].

Precision measures the fraction of synthetic samples that resides in the support of the real data distribution and is used to measure fidelity. Recall and Coverage measure the 'diversity' of synthetic samples, i.e., the fraction of real images that are represented in the output of a generative model. In addition to the two synthetic datasets (with and without feedback) generated in the previous experiment, we also evaluated a third synthetic dataset finetuned using the automatic feedback (classifier-guidance) approach described in Section 2.5. The results in Table 4 show that pathologist feedback improves the quality of synthetic data compared to the two baselines across all metrics under consideration. Interestingly, we see that feedback not improves fidelity of synthetic images, but it also improves the diversity of samples, which was not one the criteria considered in the pathologist feedback.

Downstream modeling with synthetic data. Morphology-based classification of cells is a key step in the diagnosis of hematologic malignancies. We evaluated the utility of synthetic medical images in training a cell-type classification model. In this experiment, we train a ResNext-50 model to classify the 16 cell types using real data, synthetic data from the pretrained model with no feedback, and synthetic data from the model finetuned with pathologist feedback. To ensure a fair comparison, we created synthetic datasets consisting of 128 images per cell type, matching the size of the real data. The classification accuracy of all models was then tested on the held-out real dataset, containing 32 images per cell type. Results are shown in Table 5. Unsurprisingly, the model trained on real data demonstrated superior performance across all accuracy metrics, exhibiting a significant gap compared to the model trained on synthetic data without human feedback. However, incorporating pathologist feedback helped narrow this gap and improved the quality of synthetic data to the point where the resulting classifiers only slightly underperformed compared to the one trained on real data.

To evaluate the marginal value of human feedback, we also considered two ablated versions of our synthetic data generation process. These included a synthetic dataset generated using automatic feedback (Section 2.5) as well as a reward model trained using the pathologist feedback on synthetic data only (i.e., dataset D Γ s ) without real data augmentation (Section 2.2). The results in Table 5 show that automatic feedback only marginally improves performance, which aligns with the results in Table 4. Real data augmentation (i.e., finetuning on D Γ s + D Γ r ) slightly improves classification accuracy, but most of the performance gains are achieved by finetuning on the pathologist-labeled dataset D Γ s .

Impact of the number of feedback points. How much human feedback is necessary to align the generative model with the preferences of pathologists? In Figure 4, we analyze the effect of different amounts of pathologist-labeled synthetic images on training the reward model. We explore four scenarios: 0%, 10%, 50%, and 100% of the 3,936 synthetic images labeled by the pathologist. For each scenario, we fine-tune the pretrained model using a reward model trained with the corresponding fraction of synthetic images. We then repeat the cell classification experiment to evaluate the accuracy of the classifiers trained using synthetic data generated in these four scenarios. Figure 4(a) shows that as the number of pathologist-labeled synthetic images increases, all accuracy metrics increase to improve over the pretrained model performance and become closer to the accuracy of training on real data. Additionally, we see that even a modest amount of feedback (e.g., 10% of pathologist-labeled synthetic images) can have a significant impact on performance. Figure 4(b) also show the qualitative improvement in the quality of synthetic images as the amount of human feedback increases.

Incorporating new clinical concepts using pathologist feedback. Finally, we evaluate the feedbackdriven generation approach outlined in Section 2.4. Here, our goal is not only to leverage pathologist

(b) Qualitative improvement in synthetic data quality with human feedback

0%

10%

50%

100%

Amount of

pathologist

feedback

Real data

(a)

Accuracy improvement with feedback

Synthetic data

% Pathologist feedback

Precision

Recall

F1-score

Real data

Pretrained

Finetuned

Segmented Neutrophil samples

Band Neutrophil samples

(c) Feedback-driven

generation of

cell sub-types

Pretrained:

Generation of

Mature Neutrophil (M5) samples

Finetuned:

Conditional generation of Neutrophil sub-types

0.8

0.7

I1

0.6

0.8

0.7

0.6

0.8

0.7

0.6

0

25

50

75

100

feedback for rating the plausibility of synthetic images, but also to harness their expertise in providing additional annotations that can enable the conditional diffusion model to generate more refined categories of bone marrow image patches, e.g., abnormal cell types that develop from preexisting normal cell types. Hence, refining the generative model to synthesize new cell subtypes can help continuously update the model to capture new pathological cells and build downstream diagnostic models.

In this experiment, we focus on finetuning the model to distinguish between band Neutrophils and segmented Neutrophils (Figure 5(c)). These two sub-categories are often lumped together in bone marrow cell typing, as was the case in our dataset (see Table 2). However, in many clinical settings, differentiating between the two subtypes is essential. For instance, a high percentage of band Neutrophils can indicate an acute infection, inflammation, or other pathological conditions. We collected pathologist annotations of band and segmented Neutrophils, and trained a subtype classifier to augment the plausibility reward as described in Section 2.4. The finetuned model was able to condition on the new classes and generate plausible samples of the two Neutrophil subtypes (Figure 5(c)), while retaining the classification accuracy with respect to the new classes (See Appendix for detailed results).

Conclusions

Synthetic data generation holds great potential for facilitating the sharing of clinical data and enriching rare diseases datasets. However, existing generative models and evaluation metrics lack the ability to incorporate clinical knowledge. Consequently, they often fall short in producing clinically plausible and valuable images. This paper introduces a pathologist-in-the-loop framework for generating clinically plausible synthetic medical images. Our framework involves finetuning a pretrained generative model through feedback provided by pathologists, thereby aligning the synthetic data generation process with clinical expertise. Through the evaluation of synthetic bone marrow patches by expert hematopathologists, leveraging thousands of feedback points, we demonstrate that human input significantly enhances the quality of synthetic images. These results underscore the importance of incorporating human feedback in clinical applications, particularly when generative models encounter challenges in capturing nuanced domain knowledge solely from raw data.

References

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Richard J Chen, Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering , 5(6):493-497, 2021.

Debbie Rankin, Michaela Black, Raymond Bond, Jonathan Wallace, Maurice Mulvenna, Gorka Epelde, et al. Reliability of supervised machine learning using synthetic data in health care: Model to preserve privacy for data sharing. JMIR medical informatics , 8(7):e18910, 2020.

Samuel Budd, Emma C Robinson, and Bernhard Kainz. A survey on active learning and humanin-the-loop deep learning for medical image analysis. Medical Image Analysis , 71:102062, 2021.

Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image processing: Overview, challenges and the future. Classification in BioApps: Automation of Decision Making , pages 323-350, 2018.

Kai Packhäuser, Lukas Folle, Florian Thamm, and Andreas Maier. Generation of anonymous chest radiographs using latent diffusion models for training thoracic abnormality classification systems. arXiv preprint arXiv:2211.01323 , 2022.

Hazrat Ali, Shafaq Murad, and Zubair Shah. Spot the fake lungs: Generating synthetic medical images using neural diffusion models. In Artificial Intelligence and Cognitive Science: 30th Irish Conference, AICS 2022, Munster, Ireland, December 8-9, 2022, Revised Selected Papers , pages 32-39. Springer, 2023.

Faisal Mahmood, Daniel Borders, Richard J Chen, Gregory N McKay, Kevan J Salimian, Alexander Baras, and Nicholas J Durr. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE transactions on medical imaging , 39(11):3257-3267, 2019.

Hajar Emami, Ming Dong, and Carri K Glide-Hurst. Attention-guided generative adversarial network to address atypical anatomy in synthetic ct generation. In 2020 IEEE 21st international conference on information reuse and integration for data science (IRI) , pages 188-193. IEEE, 2020.

Lisa C Adams, Felix Busch, Daniel Truhn, Marcus R Makowski, Hugo JWL Aerts, and Keno K Bressem. What does dall-e 2 know about radiology? Journal of Medical Internet Research , 25:e43110, 2023.

Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari. Adapting pretrained vision-language foundational models to medical imaging domains. arXiv preprint arXiv:2210.04133 , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems , 31, 2018.

Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning , pages 290-306. PMLR, 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Gustav Müller-Franzes, Jan Moritz Niehues, Firas Khader, Soroosh Tayebi Arasteh, Christoph Haarburger, Christiane Kuhl, Tianci Wang, Tianyu Han, Sven Nebelung, Jakob Nikolas Kather, et al. Diffusion probabilistic models beat gans on medical images. arXiv preprint arXiv:2212.07501 , 2022.

Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with latent diffusion models. In Deep Generative Models: Second MICCAI Workshop, DGM4MICCAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings , pages 117-126. Springer, 2022.

Firas Khader, Gustav Mueller-Franzes, Soroosh Tayebi Arasteh, Tianyu Han, Christoph Haarburger, Maximilian Schulze-Hagen, Philipp Schad, Sandy Engelhardt, Bettina Baessler, Sebastian Foersch, et al. Medical diffusion-denoising diffusion probabilistic models for 3d medical image generation. arXiv preprint arXiv:2211.03364 , 2022.

Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, Ilker Hacihaliloglu, and Dorit Merhof. Diffusion models for medical image analysis: A comprehensive survey. arXiv preprint arXiv:2211.07804 , 2022.

Tonghe Wang, Yang Lei, Zhen Tian, Xue Dong, Yingzi Liu, Xiaojun Jiang, Walter J Curran, Tian Liu, Hui-Kuo Shu, and Xiaofeng Yang. Deep learning-based image quality improvement for low-dose computed tomography simulation in radiation therapy. Journal of Medical Imaging , 6(4):043504-043504, 2019.

Ilona Urbaniak and Marcin Wolter. Quality assessment of compressed and resized medical images based on pattern recognition using a convolutional neural network. Communications in Nonlinear Science and Numerical Simulation , 95:105582, 2021.

Daniel Lévy and Arzav Jain. Breast mass classification from mammograms using deep convolutional neural networks. arXiv preprint arXiv:1612.00542 , 2016.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Harry Nyquist. Certain factors affecting telegraph speed. Transactions of the American Institute of Electrical Engineers , 43:412-422, 1924.

Stefan Winkler and Praveen Mohandas. The evolution of video quality measurement: From psnr to hybrid metrics. IEEE transactions on Broadcasting , 54(3):660-668, 2008.

WAEdelstein, PA Bottomley, HR Hart, and LS Smith. Signal, noise, and contrast in nuclear magnetic resonance (nmr) imaging. J Comput Assist Tomogr , 7(3):391-401, 1983.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 , 2023.

Cheng-Kang Ted Chao and Yotam Gingold. Text-guided image-and-shape editing and generation: A short survey. arXiv preprint arXiv:2304.09244 , 2023.

Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-toimage models with human preference. arXiv preprint arXiv:2303.14420 , 2023.

Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 , 2023.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1492-1500, 2017.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on a large image data set. Blood, The Journal of the American Society of Hematology , 138(20):1917-1927, 2021.

A Experimental Setup

A.1 Exploratory analysis of generative models

To select the generative model underlying our pathologist-in-the-loop framework, we conducted an exploratory analysis for the performance of various models, including conditional GANs, a conditional diffusion model, and a latent diffusion model [1]. Following an evaluation of these models, we selected the conditional diffusion model as our baseline for incorporating human feedback. The decision to choose the conditional diffusion model as our baseline was based on the fact that it outperformed the other baselines when pretrained on the bone marrow image patches. The rationale for selecting the best performing model in the pre-training phase is that the marginal value of pathologist feedback is best assesed when incorporated into a model that already performs well without feedback.

We found that the conditional GAN model did not converge to a reliable set of parameters and was therefore deemed unfit as a base model in our framework. The latent diffusion model demonstrated the ability to generate visually appealing cell images, but a majority of these images did not correspond to the intended class labels. (See Table A.1 and Figure A.1 for a quantitative and qualitative comparison of all baseline generative models.) In addition to these generative baselines, we also finetuned the Stable Diffusion model on our dataset, but the resulting images displayed an artistic style that could not be identified by expert as realistic bone marrow slides.

Conditional GAN

Conditional Latent

Diffusion

Conditional Diffusion

A.2 Dataset details, image acquisition and annotation

Our dataset was created from pathology slides for bone marrow aspirate. All slides were randomly selected from the clinical service based on normal morphology and adequate specimen. The slides were stained using a version of the Fisherbrand modified Wright-Giemsa Stain Pack. The quality of images varied across slides, reflecting variations in stain intensity, slide preparation, and slide age.

Cell types included in our analysis. The 16 image classes listed in Table 2 cover most cell types included in a standard bone marrow differential. A very broad spectrum of Erythroid and Neutrophil maturation was included, from Proerythroblast to mature Erythrocyte, and from Myeloid blast to Neutrophil. Eosinophils were separated into mature Eosinophils with segmented nuclei and immature Eosinophils. In addition, the set included Monocytes, Basophils and Mast cells. We extracted 128 images per class for training and 32 image per class for evaluating performance metrics.

Pathologist annotation and criteria for clinical plausibility. Clinicians consider various aspects when annotating the (real) bone marrow cell images, and these aspects also underlie the criteria for evaluating the plausibility of synthetic images (Table 1). These criteria include (1) Cell size: Cells can be classified based on their size, ranging from small to large. (2) Nucleus shape and size: The shape and size of a cell nucleus can provide valuable information about the cell type. For instance, band

Neutrophils have a horseshoe-shaped nucleus, while segmented Neutrophils have a multi-lobed nucleus. (3) Nucleus-to-cytoplasm ratio: The proportion of the cell occupied by the nucleus compared to the cytoplasm vary across cell types. Immature cells usually have a larger nucleus-to-cytoplasm ratio, while mature cells have a smaller ratio. (4) Chromatin pattern: The appearance of chromatin (DNA and proteins) within the nucleus can offer clues to the cell type. For example, Lymphocytes typically have dense, clumped chromatin, while Myeloid cells have a more diffuse, fine chromatin pattern. (5) Cytoplasm color and consistency: The color and texture of a cell cytoplasm can be indicative of its type. Erythroid cells have a deeply Basophilic (blue) cytoplasm, while Myeloid cells have a lighter, Eosinophilic (pink) cytoplasm. (6) Granules: The presence, size, number, and color of granules within the cytoplasm can help identify cell types. Neutrophils have small, pale granules, Eosinophils have large, orange-pink granules, and Basophils have large, dark purple granules.

A.3 Model architecture

Diffusion model. We used conditional diffusion [48] as our baseline generative model. Our finetuning pipeline is based on public repository (https://github.com/openai/improved-diffusion.git). The model was fine-tuned using the Adam optimizer [49]. The model is trained in half-precision on 2 × 24 GB NVIDIA GPUs, with a per-GPU batch size of 16, resulting in a toal batch size of 32. We used a learning rate of 104, and an exponential moving average over parameters with a rate of 0.9999.

Classification model. To evaluate the utility of synthetic data in downstream tasks, we trained a classification model to classify cell types in synthetic data and tested its performance in real data. We implemented this classifier using the ResNeXt-50 architecture, which have been shown to display good performance in classifying bone marrow cells in [50]. (ResNeXt is a convolutional neural network architecture, which combines elements of of VGG, ResNet and Inception models.) The network was initialized using weights from ImageNet dataset and then trained using our dataset.

Reward model. The reward model Γ( x, c ) takes the image and its cell type as input, and issues a plausibility scores that predicts whether a pathologist would label the image as clinically plausible. The reward model was constructed as a feed-forward neural network that operates on the cell type as well as the embedding feature of images extracted from the ResNext network structure.

B Performance Metrics for Generative Models

We implemented the evaluation scores for generative models in Section 4.3 based on publicly released repository (https://github.com/clovaai/generative-evaluation-prdc). The precision and recall scores, originally introduced in [21], are formally defined as:

where the manifold is the defined as Manifold ( X 1 , · · · , X N ) := ⋃ N i =1 B ( X i , NND k ( X i )) , with B ( x, r ) being a Euclidean ball around the point x with radius r and NND k ( X i ) is the distance to the k -th nearest neighbour. The coverage metric is defined as follows:

In the definitions above, { X i } i denotes the real sample and { Y j } j denotes the synthetic sample. The precision metric evaluates the fraction of synthetic samples that reside in the real data manifold, whereas the recall and coverage metrics evaluate the fraction of real samples that are represented in the synthetic sample. Precision can be though of as an automated score for clinical plausibility.

C Incorporating New Clinical Concepts using Pathologist Feedback

In Section 4.3, we discussed the use of human feedback to provide additional annotations that can enable the conditional diffusion model to generate more refined categories of bone marrow image patches. In this experiment, we finetuned the model to distinguish between band Neutrophils and

segmented Neutrophils (Figure 5(c)). These two sub-categories are often lumped together in bone marrow cell typing, as was the case in our dataset (see Table 2). However, in many clinical settings, differentiating between the two subtypes is essential. For instance, a high percentage of band Neutrophils can indicate an acute infection, inflammation, or other pathological conditions. We collected pathologist annotations of band and segmented Neutrophils, and trained a subtype classifier to augment the plausibility reward as described in Section 2.4. The finetuned model was able to condition on the new classes and generate plausible samples of the two Neutrophil subtypes (Figure 5(c)), while retaining the classification accuracy with respect to the new classes as shown in Table C.2.

D Pseudocodes

Algorithm 2 Pretraining the conditional diffusion model for generating synthetic images

D , a dataset of real images N , the number of training iterations B , the batch size α , learning rate A trained conditional diffusion model Initialize the conditional diffusion model M i = 1 to N Sample a mini-batch of images X ⊂ D with size B Compute the noise schedule σ ( t ) for the diffusion process Initialize the set of generated images Y with denoised images from X t = T downto 1 Compute the conditional diffusion probabilities p ( y t | y t +1 ) for Y using model M Update Y by sampling y t from p ( y t | y t +1 ) Compute the reconstruction loss L between the generated images Y and the real images X Update the model parameters θ using gradient descent: θ ← θ -α ∇ θ L

Algorithm 3 Finetuning the conditional diffusion model using pathologist feedback

D r , a dataset of real images D a , a dataset of artificial images generated from diffusion models trained on the real images N , the number of training iterations B , the batch size α , learning rate γ , penalty parameter Model Reward , Reward Model trained with clinician feedback Fine tuning Conditional Diffusion Model Load the trained conditional diffusion model M i = 1 to N Sample a mini-batch of images X ⊂ D with size B Compute the noise schedule σ ( t ) for the diffusion process Initialize the set of generated images Y with denoised images from X t = T downto 1 Compute the conditional diffusion probabilities p ( y t | y t +1 , c ) for Y and classC using model M Update Y by sampling y t from p ( y t | y t +1 ) Compute the reconstruction loss L between the generated images Y and the original images X X ∈ D r loss = loss ∗ γ X ∈ D a

gradient descent:

loss = loss ∗ Model Reward ( X ) RASIEERROR Update the model parameters θ using θ ← θ -α ∇ θ L

Algorithm 4 Incorporating new clinical concepts into the model

D pre , a collection of artificial images, a mixture of sub types

1: 2 3 function NEW_CONCEPT_FUNCTION (Training from clinician feedback)

2: img embedding = NN.encode i mage ( preprocess ( Image ))

3: class embedding = NN.encode l abel ( label )

4: return pred f eedback ( img embedding , class embedding )

5: end function

6: for (Image, label, feedback) in Dataloader do

7: scores = Reward_function(Image, label)

8: loss = MSELoss(scores, feedback) optimizer.zero_grad()

loss.backward() optimizer.step()

9: end for

E Illustration of Synthetic Samples

In this Section, we present illustrative samples of synthetic bone marrow image patches generated by our model.

Figure E.1. Representative samples from the conditional diffusion model before (left) and after (right) incorporating pathologist feedback.

Figure E.2. Representative samples from the conditional diffusion model.

Figure E.3. Representative samples from the finetuned model with 10% of the pathologist feedback points.

Figure E.4. Representative samples from the finetuned model with 50% of the pathologist feedback points.

Figure E.5. Representative samples from the finetuned model with 100% of the pathologist feedback points.

Mast Cell

Basophil

Immature Eosinophil

Mature Eosinophil

Pronormoblast

Myeloid Blast

Promyelocyte

Metamyelocyte

Neutrophil

Monocyte

Basophilic Normoblast

Polychromatophilic Normoblast

Orthochromic Normoblast

w.o. clinician

feedback

feedback

Mast Cell

Basophil

Immature Eosinophil

Mature Eosinophil

Pronormoblast

Myeloid Blast

Promyelocyte

Metamyelocyte

Neutrophil

Monocyte

Basophilic Normoblast

Polychromatophilic Normoblast

Orthochromic Normoblast

Mast Cell

Basophil

Immature Eosinophil

Mature Eosinophil

Pronormoblast

Myeloid Blast

Promyelocyte

Metamyelocyte

Neutrophil

Monocyte

Basophilic Normoblast

Polychromatophilic Normoblast

Orthochromic Normoblast

Mast Cell

Basophil

Immature Eosinophil

Mature Eosinophil

Pronormoblast

Myeloid Blast

Promyelocyte

Metamyelocyte

Neutrophil

Monocyte

Basophilic Normoblast

Polychromatophilic Normoblast

Orthochromic Normoblast

Mast Cell

Basophil

Immature Eosinophil

Mature Eosinophil

Pronormoblast

Myeloid Blast

Promyelocyte

Metamyelocyte

Neutrophil

Monocyte

Basophilic Normoblast

Polychromatophilic Normoblast

Orthochromic Normoblast

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to use the diffusion model with expert feedback to improve the quality of synthetic medical images. The key motivation is the embedding of RLHF for medical image generation. To achieve this, there are three steps: (1) pre-trained a generation model and collected expert feedback based on several manual rules; (2) trained a model to predict the feedback scores; (3) incorporated the scores to fine-tune generative models. Extensive experiments demonstrate the effectiveness of the proposed model. Overall, this is an interesting and sound paper.\n\nSTRENGTHS:\n1. RLHF-based generation model for medical knowledge incorporation. Interesting and useful.\n2. Reasonable framework. The proposed model is based on the conditional diffusion models and the whole process is sound.\n3. Extensive evaluation. Besides the commonly-used quality-related metrics and downstream performance, the new knowledge discovery and visual results are also impressive.\n\nWEAKNESSES:\n1. Reproductivity. It's better to make the codes public and improve the reproductivity.\n2. Manual rules. The expert feedback is highly related to the manual design rules. It's better to see the performance with different amounts of rules or with more rules. Also, please elaborate more on how to define these rules. In other words, it is not clear why the designed rules can indicate the complex image quality and small image differences.\n3. Downstream performance. There is only one downstream task to denote effectiveness. More tasks should be considered. Meanwhile, why not show the performance of downstream models trained by both synthetic and real images?\n\nQUESTIONS:\nSee the above weaknesses.\n\nLIMITATIONS:\nBesides the comments on the paper's weaknesses, please also consider:\n\n1. Even though it is a kind of application paper, please add some discussions to state the technical contributions.\n2. The authors claimed such a synthetic way could be useful for rare diseases. Please provide the evidence.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper develops a framework that generates synthetic medical images aligned with the clinical knowledge of doctors through training a reward model based on pathologists' image plausibility annotations. The intuition for their very simple but powerful approach that depends on incorporating clinical expert knowledge into the diffusion models by using the reward model to inform a finetuning objective arises from their observation: "designing domain-specific objective functions for finetuning foundational models that ensure a generative model adheres to clinical knowledge is challenging". Their presentation is very clear, and empirical evaluations & ablations are convincing. I recommend this paper for acceptance beyond my concerns around how well this paper fits into NeurIPS venue.

STRENGTHS:
- Very clear presentation
- Simple but powerful idea that brings in an idea from a different domain (LLMs) to medical imaging
- Convincing empirical results and ablations

WEAKNESSES:
My main worry about this paper is that NeurIPS might not be the best venue, given the algorithmic novelty is limited here. However, I do think this is an interesting application of integrating domain-specific knowledge to models through RLHF-style training.

QUESTIONS:
- How do you think your findings might change as the diffusion models get stronger in creating more plausible images (given the rate of implausible images was higher than 80% for some cell types)?
- How do you think you might generalize your method to other modalities such as interpreting MRI images where clinical plausibility criteria might be more limited? (might be harder to define the criteria beyond aliasing artifacts in certain anatomies?)

LIMITATIONS:
Authors have not explicitly described limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors proposed to include pathologist in the loop for medical data synthesis. The pathologist can be replaced by training a reward model, which is used to fine-tune the generation model.

STRENGTHS:
- I admire that the authors broke down the barriers between disciplines, and I believe their proposed pathologist-in-the-loop synthetic data generation framework is a promising strategy for improving the generation of biomedical samples. It is the main reason why I give a positive score. As "pathologist-in-the-loop" is the key contribution, I personally encourage that the raw scores from doctors and the corresponding images should be published in pair after acceptance -- then we can check the quality of the manual annotations.

- The paper is well-organized and the experiments are comprehensive.

WEAKNESSES:
- The authors only gives the overall performances. I wonder if there is a direct verification for the reward model.
- It would be better if this idea can be evaluated on other datasets. Of course, I know it is hard.
- The technical innovation seems to be limited.

QUESTIONS:
A direct evaluation of the reward model is helpful. I am curious about the relationship of the reward model quality and the synthesis performances.

LIMITATIONS:
The authors claimed that they have discussed the limitations, but it is not clear for me. Maybe I missed something.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a pathologist-in-the-loop framework for generating clinically plausible synthetic medical images. The training process is similar to generative adversarial nets, with two major modifications. First, the discriminator was trained by human input instead of real/fake labels. Second, the generator was replaced by Diffusion Models. Evaluation of synthetic bone marrow patches by expert hematopathologists, leveraging thousands of feedback points, demonstrates the significant quality enhancement achieved through human input.\n\nSTRENGTHS:\n1. I found the paper to be well-written and informative, and I thoroughly enjoyed reading it.\n\n2. The generation of synthetic data holds significant importance.\n\n3. The extensive visualization of the synthetic images, along with the notable improvement resulting from the pathologist's involvement, is highly convincing.\n\nWEAKNESSES:\n1. Missing an important baseline method that incorporates feedback from a real/fake binary classifier. This classifier can distinguish between plausible (real) and implausible (fake) images without relying on pathologists' feedback or cell-type labels (automated feedback). It is imperative to investigate whether utilizing this information can enhance the ability of Diffusion Models to generate realistic images.\n\n2. The clinical plausibility criteria presented in Table 1 pose challenges for implementation in various other tasks. Firstly, developing the checklist items necessitates domain expertise. Secondly, disagreements among pathologists may arise regarding these criteria.\n\nQUESTIONS:\n1. Despite incorporating 100% pathologist feedback, there appears to be a discernible disparity between synthetic and real data. Could you elaborate on the underlying reasons for this gap and propose potential solutions to mitigate it?\n\n2. What if pathologists are unable to differentiate between real and synthetic images? For example, in some cases [1], the experts may not be able to distinguish the real/synthetic images.\n\n3. What is the rate of implausible images after fine-tuning the diffusion model?\n\n4. Consider integrating both real and synthetic data for training purposes. To what extent can this integration enhance the overall performance? This is particularly important as the authors have not yet achieved comparable performance of AI models trained solely on synthetic data when compared to those trained on real data.\n\n**Reference**\n\n[1] Hu, Qixin, Yixiong Chen, Junfei Xiao, Shuwen Sun, Jieneng Chen, Alan L. Yuille, and Zongwei Zhou. \"Label-free liver tumor segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7422-7432. 2023.\n\nLIMITATIONS:\nNo potential negative societal impact was detected.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pathologist-in-the-loop framework for generating clinically plausible synthetic medical images using diffusion models. The process involves pretraining a conditional diffusion model on real medical images, then using synthetic images labeled by expert pathologists to train a \"clinical plausibility reward model\" to predict pathologist feedback on new images. Finally, the diffusion model is fine-tuned using a reward-weighted likelihood loss that incorporates the reward model, to align synthetic outputs with clinical knowledge.\nThe method is evaluated on a bone marrow cell image dataset. Results show the proposed method incorporating pathologist feedback significantly improves the clinical plausibility, fidelity, diversity, and downstream utility of synthetic images.\n\nSTRENGTHS:\n1) Clinically plausible medical image synthesis is an interesting and important problem. This paper provides a promising human-in-the-loop framework to incorporate clinical knowledge.\n2) The method is evaluated extensively both qualitatively and quantitatively. The results demonstrate clear improvements from incorporating expert feedback.\n3) The paper is well-written and easy to follow. The method is described in sufficient detail.\n\nWEAKNESSES:\n1) The criteria used by the pathologist for judging clinical plausibility could be described more precisely. Are there any quantitative metrics for each criterion?\n2) Only binary feedback is collected from the pathologist. More fine-grained ratings could provide richer supervision signal.\n3) Evaluations could be conducted on multiple datasets and with higher resolution images.\n4) The comparison to \"automatic feedback\" using one classifier is somewhat weak. Better baselines or more comparisons could be evaluated.\n\nQUESTIONS:\n1) The proposed method is only evaluated on a single dataset. Experiments on more datasets may be necessary to validate the generalizability of the method. Experimenting on higher resolution image synthesis could also be important for potential clinical usage.\n2) It might be helpful to include more visual or qualitative comparisons between synthetic images marked as \"implausible“ or \"plausible\".\n3) Ye et al. also proposed a relevant idea using automatic feedback as the reward in [1]. It might be worth discussing this paper.\n[1] Ye et al. Synthetic Sample Selection via Reinforcement Learning. MICCAI 2020.\n\nLIMITATIONS:\nAuthors could consider discussing the limitations and potential negative societal impact of this work:\n\nLimitations:\n1) The method has only been evaluated on one medical imaging modality/dataset (bone marrow). Applicability to other datasets or modalities with different visual and clinical characteristics is unclear.\n2) The framework requires extensive pathologist time for labeling synthetic images. Scalability to large diverse datasets may be problematic.\n3) Binary plausibility labels may not capture nuanced aspects of clinical knowledge. More granular ratings could be beneficial.\n4) The synthetic images are with low resolution (64x64). Quality and utility for diagnosis may degrade for higher resolution synthesis.\n\nExample negative societal impacts:\n1) If not carefully validated, inaccuracies in synthetic medical images could mislead or harm practitioners relying on them for diagnosis/treatment.\n2) Widespread availability of synthetic medical data could lead to privacy risks if reconstructed from patient data without consent.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}