Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedback
Authors: Shenghuan Sun
Gregory Goldgof
Atul Butte
Ahmed Alaa
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Synthetic clinical data
Machine learning for healthcare

Abstract: Generative models capable of precisely capturing nuanced clinical features in medical images hold great promise for facilitating clinical data sharing, enhancing rare disease datasets, and efficiently synthesizing (annotated) medical images at scale. Despite their potential, assessing the quality of synthetic medical images remains a challenge. While modern generative models can synthesize visually-realistic medical images, the clinical plausibility of these images may be called into question. Domain-agnostic scores, such as FID score, precision, and recall, cannot incorporate clinical knowledge and are, therefore, not suitable for assessing clinical sensibility. Additionally, there are numerous unpredictable ways in which generative models may fail to synthesize clinically plausible images, making it challenging to anticipate potential failures and design automated scores for their detection. To address these challenges, this paper introduces a pathologist-in-the-loop framework for generating clinically-plausible synthetic medical images. Our framework comprises three steps: (1) pretraining a conditional diffusion model to generate medical images conditioned on a clinical concept, (2) expert pathologist evaluation of the generated images to assess whether they satisfy clinical desiderata, and (3) training a reward model that predicts human feedback on new samples, which we use to incorporate expert knowledge into the finetuning objective of the diffusion model. Our results show that human feedback significantly improves the quality of synthetic images in terms of fidelity, diversity, utility in downstream applications, and plausibility as evaluated by experts. We also demonstrate that human feedback can teach the model new clinical concepts not annotated in the original training data. Our results demonstrate the value of incorporating human feedback in clinical applications where generative models may struggle to capture extensive domain knowledge from raw data alone.

Aligning Synthetic Medical Images with Clinical Knowledge using Human Feedback
Shenghuan Sun *
Gregory M. Goldgof*
University of California, San Francisco shenghuan. sun@ucsf. edu
Memorial Sloan Kettering Cancer Center goldgofgemskcc.oo org
Atul Butte
Ahmed M. Alaa UC Berkeley and UCSF
University of California, San Francisco stul.butte@ucsf.com
amalaa@berkeley. eealllyyyddu
Abstract
Generative models capable of capturing nuanced clinical features in medical im- ages hold great promise for facilitating clinical data sharing, enhancing rare disease datasets, and efficiently synthesizing annotated medical images scale. Despite their potential, assessing the quality of synthetic medical images remains a chal- lenge. While modern generative models can synthesize visually-realistic medical images, the clinical validity these images may be called into question. Domain- agnostic scores, such FIID score, precision, and recall, cannot incorporate clinical knowledge and are, therefore, not suitable for assessing clinical sensibility. Addi- tionally, there numerous unpredictable ways in which generative models may fail to synthesize clinically plausible images, making it challenging to anticipate potential failures and manually design scores for their detection. To address these challenges, this paper introduces wuahollgist-in-th--looo framework for generat- ing Aumically-plausibee synthetic medical images. Starting with a diffusion model pretrained using real images, our framework comprises three steps: (1) evaluating the generated images by expert pathologists to assess whether they satisfy clinical desiderata, (2) training reward model that predicts the pathologist feedback on new samples, and (3) incorporating expert knowledge into the diffusion model by using the reward model to inform a finetuning objective. We show that hu- man feedback significantly improves the quality synthetic images in terms of fidelity, diversity, utility in downstream applications, and plausibility evaluated by experts. We also show that human feedback can teach the model new clinical concepts not annotated in the original training data. Our results demonstrate the value of incorporating human feedback in clinical applications where generative models may struggle to capture extensive domain knowledge from raw data alone.
Introduction
Diffusion models have recently shown incredible success in the conditional generation of high- fideltiy natural, stylized and artistic images [1-6]. The generative capabilities of these models can be leveraged to create synthetic data in application domains where obtaining large-scale annotated datasets is challenging. The medical imaging field is one such domain, where there is often difficulty in obtaining high-quality labeled datasets [7]. This difficulty may stem from the regulatory hurdles that impede data sharing [8], the costs involved in getting experts to manually annotate images [9], or the natural scarcity of data in rare diseases [10]. Generative (diffusion) models may provide partial solution to these problems by synthesizing sigh-iideityy medical images that can be easily shared among researchers replace or augment real data in downstream modeling applications [11, 12].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Step Pathologist feedback collection
Step 2. Clinical plausibility reward modeling
Step 3. Clinically-infrrmed finetuning
Pre-trained diffusion model
Reward model training data
Pre-trained diffusion model
syntherir image
Synthetic dataset
Synthetic images
Pottfologist annclucions
Synthetic dataset
Implousible
Implausible
Reward model
Ciriccl plausibiity checklts
Reward model
Cell size
0.1
X
Nucleus shape
0.5
0.
Chromatin
Cytoplasm color Granules
plousibility scores
Plausible
Impluusbble
Pethologal impection
Inclusions
Finetuning objective
Figure I: Overview of our pathologist-in-the- synthetic data generation framework, (1) In Step 1, a syn- thetic dataset is sampled from a generative model pretrained using a dataset real medical images. The dataset then inspected by a pathologist who examines each image to determine its plausibility based on set of criteria. each image, the pathologist provides binary feedback, labeling a synthetic image as "I" if fails to meet all of the plausibility criteria. (2) In Step 2. the synthetic images and pathologist feedback obtained in Step I are used to train a reward model that predicts human feodback on new images. (3) Finally, the generative model is finetuned via an objective function that uses the reward model to incentivize the generation of clinically plausible images.
What sets medical image synthesis apart from image generation in other fields? Unlike main- stream generative modeling applications that prioritize visually realistic or artistically expressive images, synthetic medical images require different approach. They must be grounded in objective clinical and biological knowledge, and as such, they leave no room for creative or unrestricted gener- ation. Given that the ultimate goal of synthetic medical images is to be used in downstream modeling and analysis, they must faithfully reflect nuanced features that represent various clinical concepts, such cell types [13], disease subtypes [7], and anatomies [14]. Off-the-shelf image generation models are not capable of recognizing or generating clinical concepts, rendering them unsuitable for generating plausible medical images without further adaptation [15, 16). Therefore, aim is to de- velop a framework for generating synthetic medical images that not only exhibit visual realism but also demonstrate biological plausibility and alignment with clinical expertise.
One way to generate synthetic medical images is to finetune a pretrained "foundation" vision model, such as Stable Diffusion, that has been trained on billions of natural images (such as the LAION-5B dataset [17]), using real medical images. With sufficiently large set of medical images, we can expect the finetuned model to capture the clinical knowledge encoded in medical images. However, the sample sizes of annotated medical images are typically limited to a few thousand. When a large vision model is finetuned on such a dataset using generic objective functions (such the likelihood function), the model may capture only the generic features that make the medical images appear visually realistic, but it may miss nuanced features that make them biologically plausible and compliant with clinical domain knowledge (see examples in the next Section). Designing domain-spectt objective functions for finetuning that ensure a generative model adheres to clinical knowledge is challenging. The difficulty arises from the numerous unpredictable ways in which these models can generate images that lack clinical plausibility. As result, it is impractical to anticipate every possible failure scenario and manually construct a loss function that penalizes such instances.
Summary of contributions. In this paper, we develop pathologist-in -hop framework for synthe- sizing medical images that align with clinical knowledge. Our framework is motivated by the success of reinforcement learning with human feedback (RLHF) in aligning the outputs of large language models (LLMs) with human preference [18, 19], and is directly inspired by [20], where human feed- back was used to align the visual outputs of generative model with input text prompts. To generate clinically- plaussble medical images, our framework (outlined in Figure 1) comprises steps:
Step 1: We train a (conditional) diffusion model using real medical images. We then sample synthetic dataset from the model to be evaluated by a pathologist. Each image carefully examined, and the pathologist provides feedback whether it meets the necessary criteria for clinical plausibility.

---

Step 2: We collate dataset of synthetic images paired with pathologist feedback and train a reward model predict the pathologist feedback, i.e., clinical plausibilityy on new images.
Step 3:
Finally, the reward model in Step is utilized to incorporate expert knowledge into the genera- tive model. This is achieved by finetuning the diffusion model using reward-weightn loss function, which penalizes the generation of images that the pathologist considers clinically implausible.
Throughout this paper, apply the steps above to the synthetic generation of bone marrow image patches, but the same conceptual framework can generalize to any medical imaging modality, We gathered pathologist feedback on thousands of synthetic images of various cell types generated by a conditional diffusion model. Then, we analyzed the impact of this feedback on the quality of the finetuned synthetic images. Our findings suggest that incorporating pathologist feedback significantly enhances the quality synthetic images in terms of all existing quality metrics such as fidelity, accu- racy of downstream predictive models, and clinical plausibility evaluated by experts. Additionally, also improves qualities that are not directly addressed in the pathologist evaluation, such as the diversity of synthetic samples. Furthermore, we show that human feedback can teach the generative model new clinical concepts, such as more refined identification of cell types, that are not annotated in the original training data. These results demonstrate the value of incorporating human feedback in clinical applications where generative models may not be readily suited to capture intricate and extensive clinical domain knowledge from raw data alone.
Mast (B1)
Mature Easinophil (E4)
Neutrophil (M5)
Red blood cells (ER6)
Darker than expected
Nucleus should have two lobes
Nucleus should band muttilobed-shaped
enough to be recognized as
Figure 2: Samples for biolagicallyimmplunibbbe synthetic bone marrow image patches across four different cell types. On the bottom panels, show pathologist evaluations detailing the reasons for their implausibility.
2 Pathologist-in-the-LLooo Generation of Synthetic Medical Images
In this Section, we provide a detailed description of our synthetic medical image generation framework. We will use a running example pertaining single-cell images extracted from bone marrow aspirate whole slide images. Details of this setup and the dataset used in our study are provided in Section 4.
2.1 Step 1: Pathologist feedback collection
Generative model pretraining. The first step in our framework starts with training generative model to synthesize medical images through the standard training procedure. As we discuss in more detail in Section 4, we utilized dataset of 2,048 bone marrow image patches to train a conditional diffusion model [6]. The model was trained to generate cladscconiiiional images, where an image class cor- responds to cell type. We conducted diffusion nor Stable Diffusion models yielded superior results compared to customized diffusion model that employed in this study (See Appendix A). We opted for using class-conditional model rather than a text-conditional model as we found that existing pretrained vision- laniuage models were not fit for capturing the scientific jargon related bone marrow cell types.
exploratory analysis where we found that neither latent
Given a dataset of real images D, {(x', c²) T where x is a medical image and c C is an image class, we train diffusion model to generate class-conditional images through the forward process:
Tz+1
V log(po(zz/zz c)) + 6
(1)
where (O, ) is the noise term at time-step 24 is the data point at time-step f, a is the step size at 0 the model parameters and V. log(p( (æc c)) is the gradient of the log probability distribution with respect to conditioned on the original data and class c. Once the model is pretrained, we sample a synthetic dataset D. = {(7), "))) by first sampling a class 7 from C, and then sampling a medical image ž conditioned the class through the reverse diffusion process.

---

Pathologist evaluation. Each image in the synthetic dataset D. = ((,,2)), generated by the pre- trained model is inspected by an expert pathologist to assess its clinical plausibilityy The objective of this evaluation is to identify the specific inaccuracies in the synthetic data that can only be identified by an expert, and provide feedback for the model to refine its synthesized images in the finetuning step.
When a model is trained with only modest number of real image samples, it may generate bone marrow image patches that look visually appealing but are not biologically plausible. In Figure we present 8 synthetic images sampled from the conditional diffusion model, which correspond to four different cell types. Each of these images achieves high precision and fidelity scores individually, but they also have biological implausibilities such as inaccurate cell coloring or nucleus shapes. Therefore, models that prioritize visual features without considering biological knowledge may miss important clinical features required for synthetic images to be useful for downstream analysis. Generic evaluation scores (e-g-. [21, 22I) cannot diagnose these failures because they also lack biological domain knowledge. By incorporating feedhack from pathologists, we can refine the generative model by identifying biological information that is missed by the pretrained model.
The expert pathologist examined each syn- thetic image and provided feedback score
Clinical plausibility criteria
its biological plausibility. The evaluation typi-
(1) Cell size
(5) Chromatin pattern
cally involved inspecting the image and check-
(2) Nucleus shape and size
(6) Inclusions
ing 7 aspects that contribute to its perceived
plausibility (Table 1). These aspects pertain
(3) tucleus-to-cytoplasm ratio
(7) Granules
to the consistency of the shapes, sizes, pat- (4) Cytoplasm color and consistency
terns and colors of the contents of a synthetic bone marrow image I with the cell type č.
Table 1:
Pathologist evaluation criteria.
Among the determinants of plausibility is the cell size- different cells have different sizes, e.g., Lym- phocytes are generally smaller than Monocytes or Neutrophils. Nucleus shape and size also depend on the cell type, e.g., Band Neutrophils have horseshoe-shaped nucleus, whereas Segmented Neu- trophils have a multi-lobed nucleus. Chromatin patterns within the nucleus are dense and clumped in Lymphocytes, while in Myeloid cells they are diffuse and fine. The number, size and color of granules also contribute to plausibility. Detailed explanation of all criteria is provided in the Appendix.
Note that we have full control over the number of synthetic images n,, i..., we can sample an arbitrary number of synthetic images from the conditional diffusion model. The key limiting factor on n, is the time-consuming nature of the feedback collection process. To enable scalable feedback collection, we limited our study to binary feedback, i.e., the pathologist flagged a synthetic image as "implausible" they found violation of any of the criteria in Table upon visual inspection. We collected these binary signals and did not pursue a full checklist on all plausibility criteria for each synthetic image.
The output of Step 1 is an annotated dataset D. [(T' "2"""""")) )) where y {0. 1} is the pathologist feedback on the j-th synthetic image, where g² = means that the image is implausible.
2.2 Step 2: Clinical plausibility reward modeling
We conceptualize the pathologist as a "labelling function" r': synthetic image ãे and declared cell type (class) č to a binary plausibility score. In Step 2, we model the "pathologist" by learning their labelling function on the basis of their feedback annotations.
(0. that maps the observed
To train a model I' that estimates the pathologist labelling function, we construct a training dataset that comprises a mixture of real and synthetic images as follows:
Synthetic images
Real images
We construct a dataset DJ = ((3) ((",,))))
We build a dataset DJ {(x', ,. ) compris- ing the real images and pseudo-labels defined as 3'=1fe= + Z*}, where ट² Uniform(1. 2,.,,... (C).
comprising the synthetic images and correspond- ing pathologist feedback collected in Step I.
We combine both datasets D" D. UD! to construct a training dataset for the model r. The real dataset D. is built by randomly permuting the image class and assigning an implausibility label of I the permuted class does not coincide with the true class. We use the real dataset to augment the synthetic dataset with the annotated pathologist feedback. By augmenting the datasets D. and D., we teach the model to recognize two forms of implausibility in image generation: (i) instances where the synthetic image looks clinically plausible but belongs to wrong cell type (i.c., training examples

---

in D.), and (ii) instances where the synthetic image is visually consistent with the correct cell type but fails to meet some of the plausibilitt criteria Table (i.e., subset of the training examples in D"). We call the resulting model T clinical plausibility reward model. Using the augmented feedback dataset DD,, we train the reward model by minimizing the mean square error as follows:
Lr = Ejeøy (7' - ( 2) + Ze (v/ (a 2))²,
(2)
where Ar is a Sapper-prrameter that controls the contribution of real images in training the reward functions, and @ is the parameter of the reward model.
2.3 Step 3: Clinicaly-informmedd finetuning
In the final step, we refine the diffusion model by leveraging the pathologist feedback. Specifically, we incorporate domain knowledge into the model by utilizing the reward model in the finetuning ob- jective. Following [20], we use a reward- negative log- log-likelihood (NLL) objective, i.e.,
L(e, Ø) 1- log(px(T(a)) + B, ((,,)) [- (x)(()))))))
(3)
to finetune the conditional diffusion model, where 8, is hyper-parameter and Ø is the reward model parameters obtained by minimizing (2) in Step 2. The finetuning objective (3) incorporates the pathologist knowledge through the reward model, which predicts the pathologist evaluation of the syn- thetic images that the model generates as updates its parameters 8. The werarddwiighted objective penalizes the generation of images that do not align with the pathologist preferences, hence expect that the finetuned model will be less likely to generate clinically implausible synthetic images.
2.4 Bonus step: Feedback-driver generation of new clinical concepts
Besides refining generative models for clinical plausibility, pathologist feedback can be used to incor- porate novel climical concepts into the generative process that were not initially labeled the real dataset. This could allow generative models to continuously adapt in changing clinical environments. For instance. in our bone marrow image generation setup, pathologist feedback can refine image gen- eration by introducing new sub-types of the original cell types in C, as illustrated in Figure 3.
Instead of collecting pathologist feedback that is limited to clinical plausibility, we also collect their annotation of new cell sub-types (e.g., segmented and band variants of Neutrophil cell types). Next, we train an auxiliary model (z)) with parameter o to classify the new sub-types based on the pathol- ogist annotations. Finally, we finetune the condi- tional diffusion model through combined loss function, i.e., T(f, Φ) + T.O0,7 p). that incorporates the two forms of pathologist feedback, i.e., annota- tions of new cell types and clinical plausibility.
Synthetic Neutrophils
Pathologist feedback
Segmented Neutrophil
Band Neutrophil
Figure 3: Generation of refined cell sub-types.
2.5 Pathologist feedback vs. Automated feedback
To asses the added value of human feedback, we consider baseline where the generative model is supplemented with automatically generated feedback on clinical plausibility. To this end, we imple- ment baseline based on classifier-guided diffusion [23], where a classifier serves as an automatic feedback signal that deems synthetic image implausible if it does not match the corresponding cell type. For this baseline, we train an auxiliary classifier to predict the cell type of an image x, and then we incorporate the gradient of the log- likelihood of this classifier in the training objective as described in [23] (See the Supplementary material for implementation details).
Related Work
Before delving into the experimental results, we discuss three strands of literature that are related to our synthetic data generation framework. These include: generative modeling for synthetic clinical data, evaluation of generative models and learning from human feedback.
5

---

Generative modeling of synthetic medical images. The dominant approach for synthesizing medical images to train or finetune generative model, such as a Variational Autoencoder (VAE) [24], a Generative Adversarial Network (GAN) [25], or a diffusion model [6. 23], using sufficiently large sample of images from the desired modality. Owing to their recent success in achieving state-of-the-a results in high-fidelity image synthesis [23], diffusion probabilistic models have become the model of choice for medical image synthesis applications [12, 26-29J. 29I. In [12], the Stable Diffusion model- an open-source pretrained diffusion model-wass used to generate synthetic X-ray images, and in [28] it was shown that diffusion models can synthesize high-quality Magnetic Resonance Images (MRI) and Computed Tomography (CT) images. [27] used latent diffusion models to generate synthetic images from high-resote high-resolution 3D brain images. All of these models are trained with the standard likelihood objective and the synthetie images are typically evaluated through downstream classification tasks or generic, domain-agnostic metrics for image fidelity. To the best of our knowledge, none of the previous studies have explored human-AI collaboration approach synthetic image generation or incorporated clinical knowledge into generative models of medical images.
Evaluation of synthetic images in the medical domain and beyond. Unlike discriminative modeling (i.e., predictive modeling) where model accuracy can be straightforwardly evaluated by comparing the model predictions with ground-truth labels in a testing set, evaluating the quality of generative models can be quite challenging since we do not have a "ground-truth for defining what makes a synthetic sample is of high or low quality. Devising generic score to evaluate generative model can be tricky since there are many potential modes of failure [22]. Consequently, it is essential to design robust multidimensional scores that capture the most relevant failure modes for a given application.
Recently, there have been various attempts at defining domain-specific scores [30-32] as well as generic scores for evaluating the quality of synthetic images. Examples include the FID score which is based on a distributional distance between real and synthetic images [33]. Other examples for sample- level evaluation metrics include the precision and recall metrics [21] which check if synthetic data resides in the support of the real data distribution. However, these scores do not encode clinical domain knowledge, which is critical for identifying failures in generating clinically meaningful images. Traditional scores of medical image quality include signal- and contrast-to-noiss ratio [34-36], mean structural similarity [37]. These scores typically applied to real images and cannot be repurposed to judge the generative capacity of a synthetic data model in a meaningful way. The lack of an automated score for detecting clinically implausible synthetic medical images is key motivation for our work. We believe that the most reliable way to assess the quality of a synthetic image is to have it evaluated by an expert pathologist. From this perspective, the reward model I' in Section 2.2 can be thought of as a data-driven score for image quality trained using pathologist evaluations.
Learning from human feedback. The success of many modern generative models can be attributed in part to finetuning using feedback solicited from human annotators. The utilization of human feedback in model finetuning is very common in natural language processing applications, particularly in finetuning of large language models (LLMs). Examples for applications were human feedback was applied include translation [38], web question-answerr answering [39] and instruction tuning [40-42] The key idea in these applications is that by asking a human annotator to rate different responses from the same model, one can use such annotations to finetune the model align with human preference. Similar ideas have been applied to align computer vision models with human preferences [20, 43-46]. In the context of our application, the goal is to align the outputs of generative model with the preferences of pathologists, which are naturally aligned with clinical domain knowledge. Our finetuning objective builds on the recent work in [20] and [45], which use human feedback to align text-prompts with generated images using a rewghe-weighted likelihood score.
4 Experiments
In this Section, we conduct a series of experiments to evaluate the utility of pathologist feedback in im- proving the quality of synthetic medical images. In the next Subsection, we start by providing de- tailed description of the single-cell bone marrow image dataset used in our experiments.
4.1 Bone marrow cells dataset
In all experiments, we used dataset of hematopathologist consensus-annotatedd single-cell images extracted from bone marrow aspirate (BMA) whole slide images. The images were obtained from the

---

Morphological cell type
Code
Real data
Synthetic data
Puthologist feedback
Training
Testing
Plausible
Implausible
Mast Cell
B1
128
32
213
72
141
Basophil
B2
128
32
214
29
185
Immature Eosinophil
213
Mature Eosinophil
B4
128
32
224
53
171
Pronormoblast
ER1
128
32
256
97
159
Basophilic Normoblast
ER2
32
256
Polychromatophilic Normoblast
ER3
128
32
256
129
127
Orthochromic Normoblast
ER4
128
32
256
118
138
Polychromatophilic Erythrocyte
ERS
128
32
256
141
115
Mature Erythrocyte
ER6
128
32
256
120
136
Myeloid Blast
M1
128
32
256
138
118
Promyelocyte
M2
128
32
256
107
149
Myelocyte
M3
128
32
256
131
125
Metamyclocyte
M4
32
256
168
Mature Neutrophil
M5
128
32
256
176
Monocyte
MO2
128
32
256
83
173
Table 2:
Breakdown of bone marrow image patches by morphological cell type and pathologist feedback.
clinical archives of academic medical center. The dataset comprised 2,048 images, with the images evenly distributed across 16 morphological classes (cell types), with 160 images per class (Table 2). These classes encompass varied cell types found in a standard bone marrow differential. The dataset covers the complete maturation spectrum of Erythroid and Neutrophil cells, from Proerythroblast to mature Erythrocyte and from Myeloid blast to mature Neutrophils, respectively. The dataset also differentiates mature Eosinophils with segmented nuclei from immature Eosinophils and features Monocytes, Basophils, and Mast cells. Bone marrow cell counting and differentiating between various cell types is a complex task that poses challenges even for experienced hematologists. Hence, we ex- pect pathologist feedback to significantly improve the quality bone marrow image synthesis.
4.2 Synthetic data generation and pathologist feedback collection
We used conditional diffusion model trained on real images (64 x 64 pixels in size) to generate syn- thetic image patches. Training was conducted using 128 images per cell type, with 32 images per cell type held out for testing and evaluating all performance metrics. For the reward model [(x c), we used a ResNeXt-50 model [47] pre-trained on cell type classification task to obtain embeddings for individual images, and then concatenated the embeddings with one-hot encoded identifiers of image class (cell type) as inputs details on model architectures and selected hyper-pamne parameters are provided in the Appendix.
need-forwrrd neural network that predicts clinical plausibility, Further
We collected feedback from an expert pathologist on 3,936 synthetic images generated from the dif- fusion model. The pathologist identified most of these images as implausible-the rate of implausible images was as high as 85% for some cell types (e.g.. Basophil cells, see Table 2). After training the re- ward model using pathologist feedback, we finetune the diffusion model as described in Section 2.
4.3 Results
Expert evaluation of synthetic data quality. To evaluate the impact of pathologist feedback on the generated synthetic data, we created two synthetic datasets: a sample from the diffusion model (before finetuning with pathologist feedback) and a sample from the finetuned version of the model after incorporating the pathologist feedback. Each dataset comprised 400 images (25 images per cell type). An expert pathologist implausible (in a manner similar to the feedback collection process).
asked to evaluate the two samples and label each image plausible
Table 3 lists the fraction of clinically plausible images per cell type for the two synthetic datasets (be- fore and after finetuning using the pathologist feedback) as evaluated by an expert hematopathologist. As can see, the pathologist feedback leads to significant boost in the quality of synthetic images across all cell types, increasing the average rate of clinical plausibility from 0.21 to 0.75. Note that in this experiment, the human evaluator emulates the reward function I(z, c). Hence, the improved performance of the finetuned model indicates success in learning the pathologist preferences.
Evaluating synthetic data using fidelity & diversity scores. In addition to expert evaluation, we also evaluated the two synthetic datasets generated the previous experiment using standard metrics for evaluating generative models. We considered the Precision, Recall and Coverage metrics [21, 22].
7

---

Clinically plausible
Training data
Precision
Recall
Coverage
No feedback
Path, feedback
Synthetic (no feedback)
68.06
52.00
56.98
B1
0.40
0.92
0.16
1.00
Synthetic (auto. feedback)
74.80
43.90
61.33
E1
0.12
0.80
Synthetic (with feedback)
81.01
56.74
84.57
E4
0.24
0.52
FRI
0.44
0.96
Table 4: Evaluation of synthetic data using fidelity and diversity metries.
ER2
0.28
0.64
FR3
0.24
0.68
ER4
0.20
0.84
Training data
F1
Accuracy
Precision
Recall
ER5
0.20
0.76
ER6
0.20
0.96
Synthetic (no feedback)
60.33
95.17
61.33
65.56
MI
0.20
0.84
Synthetic (auto. feedback)
63.47
95.58
64.64
65.68
M2
0.20
0.64
Synthetic (with feedback, D))
71.80
96.41
71.29
74.51
M3
0.08
0.56
M4
0.20
0.84
Synthetic (with feedback, 2D) Real
75.80
96.95
75.59
76.51
0.08
79.03
97.39
79.10
79.47
M02
0.12
0.40
Table 3: Expert evaluation of synthetic data.
Table 5: Accuracy of clussifiers trained on reul and synthetic data.
Precision measures the fraction of synthetic samples that resides the support of the real data distribu- tion and is used to measure fidelity. Recall and Coverage measure the "diversity" of synthetic samples, ie.. the fraction of real images that are represented the output of a generative model. addition to the two synthetic datasets (with and without feedback) generated in the previous experiment, we also evaluated a third synthetic dataset finetuned using the automatic feedback bitassifier-guidanne approach described in Section 2.5. The results in Table 4 show that pathologist feedback improves the quality of synthetic data compared to the two baselines across all metrics under consideration. In- that feedback not improves fidelity of synthetic images, but it also improves the diversity of samples, which was not one the criteria considered in the pathologist feedback.
terestingly,
Downstream modeling with synthetic data. Morphology -assed classification of cells is key step in the diagnosis of hematologic malignancies. We evaluated the utility of synthetic medical images in training a cell-type classification model. In this experiment, we train ResNext-50 model to classify the 16 cell types using real data, synthetic data from the pretrained model with no feedback, and syn- thetic data from the model finetuned with pathologist feedback. To ensure fair comparison, we created synthetic datasets consisting of 128 images per cell type, matching the size of the real data. The classification accuracy of all models was then tested on the held-out real dataset, containing 32 images per cell type. Results are shown in Table 5. Unsurprisingly, the model trained on real data demonstrated superior performance across all accuracy metries, exhibiting significant gap compared to the model trained on synthetic data without human feedback. However, incorporating pathologist feedback helped narrow this gap and improved the quality of synthetic data to the point where the resulting classifiers only slightly underperformed compared to the one trained on real data.
To evaluate the marginal value of human feedback, we also considered two ablated versions of our syn- thetic data generation process. These included a synthetic dataset generated using automatic feedback (Section 2.5) as well reward model trained using the pathologist feedback on synthetic data only (i.c., dataset D!') without real data augmentation (Section 2.2). The results in Table 5 show that auto- matic feedback only marginally improves performance, which aligns with the results in Table 4. Real data augmentation (i.e., finetuning on D.' + D.)) slightly improves classification accuracy, but most of the performance gains are achieved by finetuning on the pathologist-labeled dataset D.!.
Impact of the number of feedback points. How much human feedback is necessary to align the gen- erative model with the preferences of pathologists? In Figure 4, we analyze the effect of different amounts of pathologist-labeled synthetic images on training the reward model. We explore four see- narios: 0%. 10%, 50%, and 100% of the 3,936 synthetic images labeled by the pathologist. For each scenario, we fine-tune the pretrained model using a reward model trained with the corresponding frac- tion of synthetic images. We then repeat the cell classification experiment to evaluate the accuracy of the classifiers trained using synthetic data generated in these four scenarios. Figure 4(a) shows that as the number of pathologist-labeled synthetic images increases, all accuracy metrics increase to improve over the pretrained model performance and become closer to the accuracy of training on real data. Additionally, we see that even modest amount of feedback (e.g.. 10% of pathologist-labele thetic images) can have a significant impact on performance. Figure 4(b) also show the qualitative improvement in the quality of synthetic images as the amount of human feedback increases.
syn-
Incorporating new clinical concepts using pathologist feedback. Finally, we evaluate the feedhack- driven generation approach outlined in Section 2.4. Here, our goal is not only to leverage pathologist
8

---

(a) Accuracy improvement with feedback
(b) Qualitative improvement in synthetic data quality with human feedback
0.8
Synthetic data
Precision
0.7
0% Real data Pretrained Finctured
Amount of pathologist feedback
0.8
10%
0.7
50%
0.6
0.8
F1.sorr
100%
0.7
0.6
0
25
50
75
100
Real
36 Pathologist feedbackk
(c) Feedback-drinn driven generation of cell sub-types
Pretrained: Generation of
Finetuned: Conditionel generation Neutrophil sub-types
Mature Neutrophil (M5) samples
Segmented Neutrophil samples
Band Neutrophil samples
Figure 4: Quantitative and qualitative impact of pathologist feedback on synthetic images. (a) Accuracy of cell-types classifiers trained on synthetic data with varying amounts of pathologist feedhack. (b) Visual inspection of random synthetic samples from diffusion models finetuned with with varying amounts of pathologist feedback. (c) Feedback-driven conditional generation of new (segmented and band) subtypes of Neutrophil cells.
feedback for rating the plausibility of synthetic images, but also harness their expertise in providing additional annotations that can enable the conditional diffusion model to generate more refined cate- gories of bone marrow image patches, e.g., abnormal cell types that develop from preexisting normal cell types. Hence, refining the generative model to synthesize new cell subtypes can help continuously update the model capture new pathological cells and build downstream diagnostic models.
In this experiment, we focus on finetuning the model to distinguish between band Neutrophils and seg- mented Neutrophils (Figure 5(c)). These sub-categories are often lumped together in bone marrow cell typing, as was the case our dataset (see Table 2). However, in many clinical settings, differenti- ating between the two subtypes essential, For instance, a high percentage of band Neutrophils can indicate an acute infection, inflammation, or other pathological conditions. We collected pathologist annotations of band and segmented Neutrophils, and trained subtype classifier augment the plausibility reward as described in Section 2.4. The finetuned model was able to condition on the new classes and generate plausible samples of the two Neutrophil subtypes (Figure 5(c)). while retaining the classification accuracy with respect to the new classes (See Appendix for detailed results).
Conclusions
Synthetic data generation holds great potential for facilitating the sharing clinical data and enriching rare diseases datasets. However, existing generative models and evaluation metries lack the ability to incorporate clinical knowledge. Consequently, they often fall short in producing clinically plausible and valuable images. This paper introduces pathologist-in soatolooist-in-the-looo framework for generating clini- cally plausible synthetic medical images. Our framework involves finetuning a pretrained generative model through feedback provided by pathologists, thereby aligning the synthetic data generation process with clinical expertise. Through the evaluation of synthetic bone marrow patches by expert hematopathologiits, leveraging thousands of feedback points, demonstrate that human input significantly enhances the quality of synthetic images. These results underscore the importance of incorporating human feedback in clinical applications, particularly when generative models encounter challenges in capturing nuanced domain knowledge solely from raw data.

---

References
[1] Robin Rombach, Andreas Blatmmnnn, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[2] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831 PMLR, 2021.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[4] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[5] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[6] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:220..12590 2022.
[7] Richard Chen, Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5()):993 2021.
[8] Debbie Rankin, Michaela Black, Raymond Bond, Jonathan Wallace, Maurice Mulvenna, Gorka Epelde, et al. Reliability of supervised machine learning using synthetic data in health care: Model to preserve privacy for data sharing. JMIR medical informatics, 18910 2020.
[9] Samuel Budd, Emma C Robinson, and Bernhard Kainz. A survey on active learning and human- in-the-loop deep learning for medical image analysis. Medical Image Analysis, 71:1 12062, 2021.
[10] Muhammad Imran Razzak, Saceda Naz, and Ahmad Zaib. Deep learning for medical image processing:
Overview, challenges and the future. Classification in BioApps: Automation of Decision Making, pages 323-350, 2018
[11] Kai Packhäuser, Lukas Folle, Florian Thamm, and Andreas Maier. Generation of anonymous chest radiographs using latent diffusion models for training thoracic abnormality classification systems. arXiv preprint wrXiv:2211.01322 2022.
[12] Hazrat Ali, Shafaq Murad, and Zubair Shah. Spot the fake lungs: Generating synthetic medical images using neural diffusion models. In Artificial Intelligence and Cognitive Science: 30th Irish Conference, AICS 2022, Munster, Ireland, December 8-9, 2022, Revised Selected Papers, pages 32-39. Springer, 2023.
[13] Faisal Mahmood, Daniel Borders, Richard Chen, Gregory N McKay, Kevan Salimian, Alexander Baras, and Nicholas J Durr. Deep adversarial training for multi-organ nuclei segmen- tation in histopathology images. IEEE transactions medical imaging, 39(11):3257-3266 2019.
[14) Hajar Emami, Ming Dong, and Carri K Glide-Hurst. Attention-guided generative adversarial network to address atypical anatomy in synthetic et generation. In 2020 IEEE 2 international conference on information reuse and integration for data science (IRI), pages 188-193. IEEE, 2020.
[15] Lisa C Adams, Felix Busch, Daniel Truhn, Marcus Makowski, Hugo JWL Aerts, and Keno K Bressem. What does dall-e 2 know about radiology? Journal of Medical Internet Research, 25:e43110, 2023.
[16] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari. Adapting pretrained iision-language foundational models to medical imaging domains. arXiv preprint arXiv:2210.04133. 2022.
[17] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv: 2200.02002 2022.
10

---

[18]
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. 2019.
arXiv preprini arXiv:/909.0880
[19]
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- -0001, 2020.
[20] Kimin Lee. Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12199 2023.
[21] Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, As- sessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
[22] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar. How faithful is your synthetic data? sample-level metries for evaluating and auditing generative models. In International Conference on Machine Learning, pages 290-306. PMLR, 2022.
[23] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[24] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 14, 2013.
[25] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):119-444 2020.
[26] Gustav Müller-Franzes, Jan Moritz Nichues, Firas Khader, Soroosh Tayebi Arasteh, Christoph Haarburger, Christiane Kuhl, Tianci Wang, Tianyu Han, Sven Nebelung, Jakob Nikolas Kather, et al. Diffusion probabilistic models beat gans on medical images. arXiv preprint 2022.
[27] Walter HL Pinaya, Petru-Daniel Tudosiu, Jessica Daffion, Pedro F Da Costa, Virginia Fernandez, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with la- tent diffusion models. In Deep Generative Models: Second MICCAI Workshop, DGM4MICCAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings. pages 117-126. Springer, 2022.
[28] Firas Khader, Gustav Mueller-Franzes, Soroosh Tayebi Arasteh, Tianyu Han, Christoph Haar- burger, Maximilian Schulze Hagen. Philipp Schad, Sandy Engelhardt, Bettina Baessler, Sebas- tian Foersch, et al. Medical diffusion-denoiisnng diffusion probabilistic models for 3d medical image generation. arXiv preprint arXiv:2211.03366 2022.
[29] Amirhossein Kazerouni, Ehsan Khodapanah Aghdam, Moein Heidari, Reza Azad, Mohsen Fayyaz, llker Hacihaliloglu, and Dorit Merhof. Diffusion models for medical image analysis: A comprehensive survey. arXiv preprint arXiv:2211.0777
2022.
[30] Tonghe Wang, Yang Lei, Zhen Tian, Xue Dong, Yingzi Liu, Xiaojun Jiang, Walter Curran, Tian Liu, Hui-Kuo Shu, and Xiaofeng Yang. Deep learning- based image quality improvement for low-dose computed tomography simulation in radiation therapy. Journal of Medical Imaging, 6(4))043504-043544, 2019.
[31] Honaa Urbaniak and Marcin Wolter. Quality assessment of compressed and resized medical images based on pattern recognition using convolutional neural network. Communications in Nonlinear Science and Numerical Simulation, 95: 15:1055,,, 2021.
[32] Daniel Lévy and Arzav Jain, Breast mass classification from mammograms using deep convolu- tional neural networks. arXiv preprint 16/2.0054, 2016.
[33] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[34] Harry Nyquist. Certain factors affecting telegraph speed. Transactions of the American Institute of Electrical Engineers, 43:412-422, 1924.
[35] Stefan Winkler and Praveen Mohandas. The evolution video quality measurement: From psnr to hybrid metries. IEEE transactions on Broadcasting, 54(3):660-668, 2008.
11

---

[36] WA Edelstein, PA Bottomley, HR Hart, and LS Smith. Signal, noise, and contrast in nuclear magnetic resonance (nmr) imaging. J Comput Assist Tomogr, 7(3):311-41,, 1983.
[37] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
[38] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1ee11144944477 04439,04333.. 2014.
[39] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:21 2021.
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[41] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2204:2200.05862, 2022.
[42] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint (tXiv:22/2.08073, 2022.
[43] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprini arXiv:2304.05597,, 2023.
[44]
Cheng-Kang Ted Chao and Yotam Gingold. Text-guided image-and-shapp editing and genera- tion: short survey, arXiv preprini arXiv:2304.09244 09244, 2023.
[45] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint 14420 2023.
[46] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.. 06767, 2023.
[47] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[48] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[49] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 6980, 2014.
[50] Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on large image data set. Blood, The Journal of the American Society of Hematology. 131021))117--1977, 2021.
12

---

A Experimental Setup
A.1 Exploratory analysis of generative models
To select the generative model underlying pathologist- -enggiss-iitttelloyy framework, conducted an exploratory analysis for the performance of various models, including conditional GANs, a conditional diffusion model, and latent diffuuson model [1]. Following evaluation of these models, selected the conditional diffusion model as our baseline for incorporating human feedback. The decision to choose the conditional diffusion model as baseline was based the fact that it outperformed the other baselines when pretrained on the bone marrow image patches. The rationale for selecting the best performing model in the pre-true pre-training phase is that the marginal value of pathologist feedback is best assesed when incorporated into a model that already performs well without feedback.
We found that the conditional GAN model did not converge to reliable set of parameters and was therefore deemed unfit as base model in our framework. The latent diffusion model demonstrated the ability generate visually appealing cell images, but majority of these images did not correspond to the intended class labels. (See Table A. I and Figure A. for a quantitative and qualitative comparison of all baseline generative models.) In addition to these generative baselines, also finetuned the Stable Diffusion model on our dataset, but the resulting images displayed an artistic style that could not be identified by expert realistic bone marrow slides.
Table A.1: Performance comparison for baseline generative models in the cell classification task.
Training Data
Test Data
AUC
F1
Precision
Recall
Conditional diffusion model
Real
0.929482 0.413551
0.567672
0.604237
0.604146
Conditional GAN model
Real
0.003225
0.002186
0.015804
Conditional Latent diffusion model
Real
0.566388
0.008565
0.048372
0.051456
Conditional Latent Diffusion
Conditional
Conditional Diffusion
Figure A.1: Representative samples from different baseline models
A.2 Dataset details, image acquisition and annotation
Our dataset was created from pathology slides for bone marrow aspirate. All slides randomly se- lected from the clinical service based on normal morphology and adequate specimen. The slides were stained using version of the Fisherbrand modified Wright-Giemsa Giemsa Stain Pack. The quality of images varied across slides, reflecting variations in stain intensity, slide preparation, and slide age.
Cell types included in our analysis. The 16 image classes listed in Table 2 cover most cell types included a standard bone marrow differential. A very broad spectrum of Erythroid and Neutrophil maturation was included, from Proerythroblast mature Erythrocyte, and from Myeloid blast to Neutrophil. Eosinophils were separated into mature Eosinophils with segmented nuclei and immature Eosinophils. In addition, the set included Monocytes, Basophils and Mast cells. We extracted 128 im- ages per class for training and 32 image per class for evaluating performance metrics.
Pathologist annotation and criteria for clinical plausibility. Clinicians consider various aspects when annotating the (real) bone marrow cell images, and these aspects also underlie the criteria for evaluating the plausibility of synthetic images (Table 1). These criteria include (1) Cell size: Cells can be classified based on their size, ranging from small to large. (2) Nucleus shape and size: The shape and size of a cell nucleus provide valuable information about the cell type. For instance, band
13

---

Neutrophils have a horseshoe-shappe shaped nucleus, while segmented Neutrophils have multi-lobes nu- cleus. (3) nucleus-to-cytoolasm ratio: The proportion of the cell occupied by the nucleus compared to the cytoplasm vary across cell types. Immature cells usually have larger nucleus-to-ce mucleus-to-cytoplasn ratio, while mature cells have smaller ratio. (4) Chromatin pattern: The appearance of chromatin (DNA and proteins) within the nucleus can offer clues to the cell type. For example, Lymphocytes typically have dense, clumped chromatin, while Myeloid cells have a more diffuse, fine chromatin pattern. (5) Cytoplasm color and consistency: The color and texture of cell cytoplasm can be indicative of type. Erythroid cells have a deeply Basophilic (blue) cytoplasm, while Myeloid cells have a lighter, Eosinophilic (pink) cytoplasm. (6) Granules: The presence, size, number, and color of granules within the cytoplasm can help identify cell types. Neutrophils have small, pale granules, Eosinophils have large, orange-ping granules, and Basophils have large, dark purple granules.
A.3 Model architecture
Diffusion model. We used conditional diffusion [48] as our baseline generative model. Our fine- tuning pipeline is based on public repository model was fine-tuned using the Adam optimizer [49]. The model is trained in half-precision on 2 x 24 GB NVIDIA GPUs, with per-GPU batch size of 16, resulting in toal batch size of 32. We used learning rate of 104, and an exponential moving average over parameters with a rate of 0.9999.
The
Classification model. To evaluate the utility of synthetic data in downstream tasks, trained clas- sification model to classify cell types in synthetic data and tested its performance in real data. We implemented this classifier using the ResNeXt-50 architecture, which have been shown to display good performance in classifying bone marrow cells in [50]. (ResNeXt a convolutional neural net- work architecture, which combines elements of of VGG, ResNet and Inception models.) The network was initialized using weights from ImageNet dataset and then trained using our dataset.
Reward model. The reward model [(2, c) takes the image and its cell type as input, and issues plausibility scores that predicts whether pathologist would label the image as clinically plausible. The reward model was constructed as feed-forward neural network that operates on the cell type as well as the embedding feature of images extracted from the ResNext network structure.
B Performance Metrics for Generative Models
We implemented the evaluation scores for generative models in Section 4.3 based on publicly released repository (https://github. httpss/ggihub..co//llaaaaaa////a wwwaaaa/geeeaaiveeeealaaaaa originally introduced [21], are formally defined as:
The precision and recall scores,
M
N N
1
Precision = M
Σ 1=1
1 1035 Manifold(X1..... Recall
(XyE Monifold((1
where the manifold is the defined as Manifold ,xx) := U NND,, with Bxx,7 r) being a Euclidean ball around the point with radius r and NND, the distance to the k th nearest neighbour. The coverage metric is defined follows:
Coverage Σ Las NND,
In the definitions above, {X+}; denotes the real sample and {Y,}, denotes the synthetic sample. The precision metric evaluates the fraction of synthetic samples that reside in the real data manifold, whereas the recall and coverage metrics evaluate the fraction of real samples that are represented in the synthetic sample. Precision be though of as automated score for clinical plausibility.
C Incorporating New Clinical Concepts using Pathologist Feedback
In Section 4.3, we discussed the use of human feedback to provide additional annotations that can enable the conditional diffusion model to generate more refined categories of bone marrow image patches. In this experiment, we finetuned the model to distinguish between band Neutrophils and
14

---

segmented Neutrophils (Figure 5(c)). These two categories are often lumped together in bone marrow cell typing, as was the case in our dataset (see Table 2). However, in many clinical settings, differentiating between the two subtypes is essential. For instance, high percentage of band Neu- trophils can indicate an acute infection, inflammation, or other pathological conditions. We collected pathologist annotations of band and segmented Neutrophils, and trained a subtype classifier to aug- ment the plausibility reward as described in Section 2.4. The finetuned model was able to condition on the new classes and generate plausible samples of the two Neutrophil subtypes (Figure 5(c)), while retaining the classification accuracy with respect to the new classes as shown Table C.2.
Table C.2: Model finetuned with the new subtypes of Neutrophil cells.
AUC score
Accuracy
Precision
Recall
81.90
71.88
67.95
82.81
D Pseudocodes
Algorithm 1 Training the reward model
function BEEWARD_FUNCTION (Training from pathologist feedback)
2: 3Mgembeddding = N N. mogode,mage((prppre Image))
3: classembedding N N. sncodeeabell(aaber
4: return
claSSembedding)
5: end function
6: for (Image, label, feedback) in Dataloader do
7: scores Reward (unction(Image, label)
8: loss MMELLsssssorres feedback) optimizer.zero _grad()gr
loss.backward() optimizer.step()
9: end for
Algorithm 2 Pretraining the conditional diffusion model for generating synthetic images
D. dataset of real images N, the number of training iterations B, the batch size a, learning rate A trained conditional diffusion model Initialize the conditional diffusion model M i = 1 to N Sample mini-batch of images X C D with size B Compute the noise schedule (t) for the diffusion process Initialize the set of generated images Y with denoised images from X (B(+1) T downto 1 Compute the conditional diffusion probabilities p()(B((+))) for Y using model M Update Y by sampling Y from p(y Mt+1) Compute the reconstruction loss between the generated images Y and the real images X Update the model parameters 0 using gradient descent:
<- V.
15

---

Algorithm 3 Finetuning the conditional diffusion model using pathologist feedback
Dr. dataset of real images Da. dataset of artificial images generated from diffusion models trained on the real images N, the number of training iterations B, the batch size a, ModelReward. learning rate 7, penalty parameter Model Reward. Reward Model trained with clinician feedback
Fine tuning Conditional Diffusion Model Load the trained conditional diffusion model M = N Sample mini-batch of images X D with size B Compute the noise schedule (() for the diffusion process Initialize the set of generated images Y with denoised images from X T downto Compute the conditional diffusion probabilities p(uelye+1, c) for Y and classC using model M Update Y by sampling Yr from p(y (9++1) Compute the reconstruction loss L between the generated images and the original images X E D,loss loss * ? Do RASIEERROR Update the model parameters 0 using
loss = Loss * M odel (X)ard(X)
gradient descent: t 0 aVaL
Algorithm& Incorporating new clinical concepts into the model
D pre. collection of artificial images, a mixture of sub types
NEW_CONCEPT CONCEPT FUNCTION (Training from clinician feedback) 2: irigeebbeddcng = N N. engcoeee
3: function NEW
( [mmage))
3: classembedding N N. enccodeaabelllabel)
4: return
classembeddingg)
5: end function
6: for (Image, label, feedback) in Dataloader do
7: scores Reward_function[[naee function(Image, label)
8: loss MSELoss(scores,, feedback) optimizer.zero __rod)rrr)) loss.backward() optimizer.step()
9: end for
E Illustration of Synthetic Samples
In this Section, present illustrative samples of synthetic bone marrow image patches generated by our model.
Figure E.1. Representative samples from the conditional diffusion model before (left) and after (right) incorporating pathologist feedback.
Figure E.2. Representative samples from the conditional diffusion model.
Figure E.3. Representative samples from the finetuned model with 10% of the pathologist feedback points.
Figure E.4. Representative samples from the finetuned model with 50% of the pathologist feedback points.
Figure E.S. Representative samples from the finetuned model with 100% of the pathologist feedback points.
16

---

w.o. clinician feedback
w. clinician feedback
Mast Cell
Basophil
Immature Eosinophil
Mature Eosinophil
Pronormoblast
Basophilic Normoblast
Polychromatophilic Normoblast
Normolll
Orthochromic Normoblast
Polychromatophilic Erythrocyte
Mature Erythrocyte
Myeloid Blast
Promyelocyte
Myelocyte
Metamyelocyte
Neutrophil
Monocyte
Figure E.1: Representative samples from the conditional diffusion model before (left) and after (right)
incorporating pathologist feedback.
17

---

Mast Cell
Basophil
Immature Eosinophil
Mature Eosinophil
Pronormoblast
Basophilic Normoblast
Polychromatophilic Normoblast
Orthochromic Normoblast
PolychromatophicccEyhhhhe Erythrocyte
Mature Erythrocyte
Myeloid Blast
Promyelocyte
Myelocyte
Metamyelocyte
Neutrophil
Monocyte
Figure E.2: Representative samples from the conditional diffusion model.
18

---

Mast Cell
Basophil
Immature Eosinophil
Mature Eosinophil
Pronormoblast
Basophilic Normoblast
Polychromatophilic Normoblast
Normollll
Orthochromic Normoblast
Polychromatophilic ExytroorttpphhiclEEyyrrtotee Erythrocyte
Mature Erythrocyte
Myeloid Blast
Promyelocyte
Myelocyte
Metamyelocyte
Neutrophil
Monocyte
Figure E.3: Representative samples from the finetuned model with 10% of the pathologist feedback points.
19

---

Mast Cell
Basophil
Immature Eosinophil
Mature Eosinophil
Pronormoblast
Basophilic Normoblast
Polychromatophilic Normoblast
Normollll
Orthochromic Normoblast Normobllas
Polychromatophilic Exyhroor Erythrocyte
Mature Erythrocyte
Myeloid Blast
Promyelocyte
Myelocyte
Metamyelocyte
Neutrophil
Monocyte
Figure E.4: Representative samples from the finetuned model with 50% of the pathologist feedback points.
20

---

Mast Cell
Basophil
Immature Eosinophil
Mature Eosinophil
Pronormoblast
Basophilic Normoblast
Polychromatophilic Normoblast
Normollll
Orthochromic Normoblast
Polychromatophilic ExyhrrorttpphhiclEEyyrottte Erythrocyte
Mature Erythrocyte
Myeloid Blast
Promyelocyte
Myelocyte
Metamyelocyte
Neutrophil
Monocyte
Figure E.5: Representative samples from the finetuned model with 100% of the pathologist feedback points.
21


Reviews:
[{"review_id": "hITP1hA1HK", "reviewer": "Reviewer_uSrF", "summary": "This paper proposes a pathologist-in-the-loop framework for generating clinically plausible synthetic medical images using diffusion models. The process involves pretraining a conditional diffusion model on real medical images, then using synthetic images labeled by expert pathologists to train a \"clinical plausibility reward model\" to predict pathologist feedback on new images. Finally, the diffusion model is fine-tuned using a reward-weighted likelihood loss that incorporates the reward model, to align synthetic outputs with clinical knowledge.\nThe method is evaluated on a bone marrow cell image dataset. Results show the proposed method incorporating pathologist feedback significantly improves the clinical plausibility, fidelity, diversity, and downstream utility of synthetic images.", "questions": "1) The proposed method is only evaluated on a single dataset. Experiments on more datasets may be necessary to validate the generalizability of the method. Experimenting on higher resolution image synthesis could also be important for potential clinical usage.\n2) It might be helpful to include more visual or qualitative comparisons between synthetic images marked as \"implausible“ or \"plausible\".\n3) Ye et al. also proposed a relevant idea using automatic feedback as the reward in [1]. It might be worth discussing this paper.\n[1] Ye et al. Synthetic Sample Selection via Reinforcement Learning. MICCAI 2020.", "limitations": "Authors could consider discussing the limitations and potential negative societal impact of this work:\n\nLimitations:\n1) The method has only been evaluated on one medical imaging modality/dataset (bone marrow). Applicability to other datasets or modalities with different visual and clinical characteristics is unclear.\n2) The framework requires extensive pathologist time for labeling synthetic images. Scalability to large diverse datasets may be problematic.\n3) Binary plausibility labels may not capture nuanced aspects of clinical knowledge. More granular ratings could be beneficial.\n4) The synthetic images are with low resolution (64x64). Quality and utility for diagnosis may degrade for higher resolution synthesis.\n\nExample negative societal impacts:\n1) If not carefully validated, inaccuracies in synthetic medical images could mislead or harm practitioners relying on them for diagnosis/treatment.\n2) Widespread availability of synthetic medical data could lead to privacy risks if reconstructed from patient data without consent.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) Clinically plausible medical image synthesis is an interesting and important problem. This paper provides a promising human-in-the-loop framework to incorporate clinical knowledge.\n2) The method is evaluated extensively both qualitatively and quantitatively. The results demonstrate clear improvements from incorporating expert feedback.\n3) The paper is well-written and easy to follow. The method is described in sufficient detail.", "weaknesses": "1) The criteria used by the pathologist for judging clinical plausibility could be described more precisely. Are there any quantitative metrics for each criterion?\n2) Only binary feedback is collected from the pathologist. More fine-grained ratings could provide richer supervision signal.\n3) Evaluations could be conducted on multiple datasets and with higher resolution images.\n4) The comparison to \"automatic feedback\" using one classifier is somewhat weak. Better baselines or more comparisons could be evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please refer to the combined response above for a general overview of the improvements and changes that have been incorporated in the manuscript to address this issue.\n\n**Weaknesses**\n\n*1- Criteria for evaluating clinical plausibility*\n\nIn the final manuscript, we will further elaborate on the qualitative criteria for evaluating clinical plausibility. However, **we would like to stress that quantitative metrics are not possible as cells can be implausible for a wide range of unpredictable reasons.** This is precisely the motivation behind our work; if there were a finite number of ways in which images can be invalid and if there were quantitative metrics for evaluating those failures, there would be no need for human feedback.\n\n*2- Limitation to binary feedback* \n\nWe appreciate the reviewer's suggestion of collecting more fine-grained ratings from the pathologist to provide a richer supervision signal. Indeed, incorporating a more detailed feedback system could potentially enhance the learning process and improve the model's performance. However, there are some practical considerations that led us to opt for binary feedback. Binary feedback simplifies the annotation process for the pathologist, reducing the cognitive load and minimizing potential inconsistencies in the ratings. In a clinical setting, it is crucial to balance the workload of the pathologist while ensuring that the feedback provided is both accurate and meaningful. Furthermore, our results suggest that binary feedback is an effective supervision signal for generating clinically plausible synthetic medical images. That being said, we acknowledge the potential benefits of more fine-grained ratings and will consider exploring this approach in future work. \n\n*3- Image resolution* \n\nWe would like to clarify the context in which these images are generated and used, as it may help address the concern about the resolution.  The resolution of the single cell patch may appear low, but these are cropped from whole slide images scanned at 400x magnification, which is the industry standard for clinical whole slide scanners. \n\nIt is important to note that clinicians routinely use this resolution, or even lower resolution, for cell counting and making diagnoses. Thus, the resolution of the synthetic images in our study is consistent with the resolution used in real-world clinical settings and should not impact the utility of our approach for diagnosis. Our methodology is designed to be compatible with the standard industry practices and resolution requirements in the clinical context.\n\n*4- The automatic feedback baseline* \n\nWe thank this reviewer for suggesting more baseline experiments. Accordingly, we add one more baseline control that incorporates feedback from a real/fake binary classifier. This classifier would distinguish between plausible (real) and implausible (fake) images without relying on pathologists' feedback or cell-type labels, offering automated feedback. Please see our reply to **Reviewer q99y** for the results.\n\n**Answers to Questions**\n\n1- We leveraged another independent dataset and showed that the framework is generalizable to the external dataset. Please see our response for **Reviewer n6aW** for the results.\n\n2- We appreciate the reviewer's suggestion to include more comparisons between synthetic images marked as \"implausible\" and \"plausible.\" As per your suggestion, we have included a visualization of 128 images, wherein 64 images are considered realistic, and the remaining 64 are deemed implausible. This visualization will provide a clear comparison between plausible and implausible synthetic images (**Rebuttal Figure 1**) and please refer to the pdf attached with the combined response to check the results.\n\nAdditionally, we have added an appendix to the paper, which contains 512 images, with 32 images for each cell type. We hope that these additional visualizations and comparisons will effectively address the reviewer's concerns and enhance the clarity of our paper.\n\n3- We appreciate the opportunity to discuss Ye et al., MICCAI 2020. While Ye et al. propose a reinforcement learning (RL) based method for selecting high-quality synthetic samples to improve the performance of medical image recognition systems, our work focuses on improving the generative model itself by incorporating pathologist feedback. We believe that our approach differs from Ye et al.'s work in three significant ways: (1) generative model improvement, (2) task complexity, and (3) expert validation. We will include a comparison in the revised manuscript.\n\n**Limitations:**\n\n1- Evaluation on a single medical imaging modality/dataset: In the revised manuscript, we will discuss this limitation and emphasize the need for future studies to evaluate our approach on various medical imaging modalities and datasets to ensure its generalizability.\n\n2- Pathologist time requirements: We will address this limitation in our paper and explore possible ways to alleviate this issue, such as involving multiple experts, leveraging active learning techniques to minimize the number of labels required, or incorporating semi-supervised or unsupervised learning methods.\n\n3- Binary plausibility labels: In the revised manuscript, we will discuss this limitation and consider potential solutions, such as adopting multi-level or continuous rating scales for clinical plausibility, which may provide a more detailed representation of expert knowledge.\n\nRegarding the potential negative societal impact, we understand that the misuse or biased generation of synthetic medical images could have adverse consequences in medical research and decision-making. In the revised manuscript, we will discuss the importance of ensuring the ethical use of synthetic medical images, addressing potential biases in the generative models, and considering privacy concerns when generating and sharing synthetic data.\n\n"}, {"review_id": "GWNknsWxSh", "reviewer": "Reviewer_q99y", "summary": "This paper introduces a pathologist-in-the-loop framework for generating clinically plausible synthetic medical images. The training process is similar to generative adversarial nets, with two major modifications. First, the discriminator was trained by human input instead of real/fake labels. Second, the generator was replaced by Diffusion Models. Evaluation of synthetic bone marrow patches by expert hematopathologists, leveraging thousands of feedback points, demonstrates the significant quality enhancement achieved through human input.", "questions": "1. Despite incorporating 100% pathologist feedback, there appears to be a discernible disparity between synthetic and real data. Could you elaborate on the underlying reasons for this gap and propose potential solutions to mitigate it?\n\n2. What if pathologists are unable to differentiate between real and synthetic images? For example, in some cases [1], the experts may not be able to distinguish the real/synthetic images.\n\n3. What is the rate of implausible images after fine-tuning the diffusion model?\n\n4. Consider integrating both real and synthetic data for training purposes. To what extent can this integration enhance the overall performance? This is particularly important as the authors have not yet achieved comparable performance of AI models trained solely on synthetic data when compared to those trained on real data.\n\n**Reference**\n\n[1] Hu, Qixin, Yixiong Chen, Junfei Xiao, Shuwen Sun, Jieneng Chen, Alan L. Yuille, and Zongwei Zhou. \"Label-free liver tumor segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7422-7432. 2023.", "limitations": "No potential negative societal impact was detected.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. I found the paper to be well-written and informative, and I thoroughly enjoyed reading it.\n\n2. The generation of synthetic data holds significant importance.\n\n3. The extensive visualization of the synthetic images, along with the notable improvement resulting from the pathologist's involvement, is highly convincing.", "weaknesses": "1. Missing an important baseline method that incorporates feedback from a real/fake binary classifier. This classifier can distinguish between plausible (real) and implausible (fake) images without relying on pathologists' feedback or cell-type labels (automated feedback). It is imperative to investigate whether utilizing this information can enhance the ability of Diffusion Models to generate realistic images.\n\n2. The clinical plausibility criteria presented in Table 1 pose challenges for implementation in various other tasks. Firstly, developing the checklist items necessitates domain expertise. Secondly, disagreements among pathologists may arise regarding these criteria.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please refer to the combined response above for a general overview of the improvements and changes that have been incorporated in the revised manuscript.\n\n**Weaknesses**\n\n1- We appreciate the reviewer's suggestion to include a baseline method that incorporates feedback from a real/fake binary classifier. This classifier would distinguish between plausible (real) and implausible (fake) images without relying on pathologists' feedback or cell-type labels, offering automated feedback. Following your suggestion, we conducted experiments where we trained a binary classifier, referred to as a naive classifier, on real and artificial images without relying on pathologists' feedback or cell-type labels. However, this naive classifier did not improve the diffusion model's performance in the downstream cell-typing task, as shown in **Rebuttal Table 1** (Please refer to the pdf attached with the combined response to check the results).\n\nIn summary, while the real/fake binary classifier offers an interesting approach, our experiments show that it does not lead to improvements in the downstream cell-typing task, which highlights the importance of pathologists' feedback to generate synthetic medical images that hold clinical validity. \n\n2-  We acknowledge that developing these criteria requires domain expertise and that disagreements among pathologists may arise. We would like to address these concerns as follows:\n\n    (1)\tDomain expertise: It is indeed true that annotating the synthetic medical image quality requires expert knowledge, which is why we have engaged clinician experts to help with the annotation process. We believe that the expert-informed criteria provide a robust foundation for evaluating the clinical plausibility of generated images. Furthermore, once this annotation process is completed, the criteria can be potentially applied to other tasks within the domain, such as cell counting and abnormal cell identification. This makes our approach scalable and applicable to a wide range of medical imaging tasks.\n\n    (2)\tDisagreements among pathologists: While we acknowledge that disagreements among experts may occur, the criteria we have used are based on standards outlined in medical textbooks. We have made every effort to ensure that our criteria are as unbiased and objective as possible.\n\n**Questions**\n\n1- There are several factors that may contribute to the observed disparity between synthetic and real data:\n\n    (1)\tLimitations of the generative model: While our generative model is capable of synthesizing visually realistic images, it may not fully capture the complex and nuanced features present in real medical images. This is an inherent challenge in generative modeling, especially when dealing with high-dimensional and intricate data such as medical images.\n    (2)\tIncomplete representation of clinical knowledge: Although we have incorporated pathologist feedback, there might be aspects of clinical knowledge that are not entirely captured by the reward function. Medical expertise is vast, and it can be challenging to encapsulate all relevant information in a single model.\n\n2- Please note our feedback collection process requires pathologists to identify images that are not clinically valid, and not differentiate between real/synthetic images. In fact, pathologists are only shown synthetic images for feedback. If the pathologist cannot distinguish between real and synthetic images, this means that a synthetic image is clinically valid.\n\n3- As shown in **Main Table 3**, we found that incorporating pathologist feedback led to a significant boost in the quality of synthetic images across all cell types. After fine-tuning the diffusion model with expert feedback, the average rate of clinical plausibility increased from 0.21 to 0.75. This improvement demonstrates the effectiveness of our approach in generating more clinically plausible synthetic medical images.\n\n4- Integrating real and synthetic data for training purposes can potentially enhance overall performance; however, it is essential to ensure that the synthetic data is of high quality and clinically plausible. Our current approach aims to improve the quality of synthetic medical images by incorporating pathologist feedback, which can make synthetic data more suitable for integration with real data during the training process. It is important to note that the synthetic data generated in our approach is based on the limited real data available. As a result, integrating synthetic and real data together may not fully address all the issues arising from the limited real data, such as underrepresented classes or rare cases. \n"}, {"review_id": "qFatLe4AwF", "reviewer": "Reviewer_n6aW", "summary": "The authors proposed to include pathologist in the loop for medical data synthesis. The pathologist can be replaced by training a reward model, which is used to fine-tune the generation model.", "questions": "A direct evaluation of the reward model is helpful. I am curious about the relationship of the reward model quality and the synthesis performances.", "limitations": "The authors claimed that they have discussed the limitations, but it is not clear for me. Maybe I missed something.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I admire that the authors broke down the barriers between disciplines, and I believe their proposed pathologist-in-the-loop synthetic data generation framework is a promising strategy for improving the generation of biomedical samples. It is the main reason why I give a positive score. As \"pathologist-in-the-loop\" is the key contribution, I personally encourage that the raw scores from doctors and the corresponding images should be published in pair after acceptance -- then we can check the quality of the manual annotations.\n\n- The paper is well-organized and the experiments are comprehensive.", "weaknesses": "- The authors only gives the overall performances. I wonder if there is a direct verification for the reward model.\n- It would be better if this idea can be evaluated on other datasets. Of course, I know it is hard.\n- The technical innovation seems to be limited. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please refer to the combined response above for a general overview of the improvements and changes that have been incorporated in the revised manuscript.\n\n**Weaknesses**\n\n1- Thanks for suggesting a direct verification of the reward model. We provide the reward function performance on a held-out validation set, which is also annotated by clinical experts (**Rebuttal Table 2**, Please refer to the pdf attached with the combined response to check the results). The reward function performance improves with more feedback, which correlates with the downstream classification performance demonstrated in **Main Figure 4**. In the revised manuscript, we will include detailed performance metrics for each cell type in the appendix, rather than showing only the overall performance. This should provide a more comprehensive understanding of the model's effectiveness.\n\n2- We concur that assessing our methodology on additional datasets is crucial to validate its applicability across various contexts. To address this concern, we have attempted to gather another independent dataset from an external institute. Regrettably, due to time constraints, we were only able to annotate 20 cell types per class for the training set, a significantly smaller number compared to the 128 per class in the dataset presented in the paper. Despite this limitation, we proceeded to train a diffusion model as the baseline and employed the same workflow to fine-tune it. The reward function remained consistent; however, it was applied to synthetic images generated from the new and independent dataset. The results are shown in the **Rebuttal Table 3** (Please refer to the pdf attached with the combined response to check the results). \n\nOwing to the limited data, the baseline performance was substantially inferior to that of the primary dataset. Nonetheless, the pathologist-in-the-loop approach still managed to significantly enhance the model's performance. This indicates that our method is effective even when applied to smaller and distinct datasets.\n\n3- We believe that the novelty and significance of our work lie in its application and the unique combination of existing techniques to address the challenges of generating clinically plausible synthetic medical images. To the best of our knowledge, our study is the first to leverage human feedback to model clinician preferences in the context of medical image generation. \n"}, {"review_id": "sWhyRjyQ1M", "reviewer": "Reviewer_ZdBm", "summary": "This paper develops a framework that generates synthetic medical images aligned with the clinical knowledge of doctors through training a reward model based on pathologists' image plausibility annotations. The intuition for their very simple but powerful approach that depends on incorporating clinical expert knowledge into the diffusion models by using the reward model to inform a finetuning objective arises from their observation: \"designing domain-specific objective functions for finetuning foundational models that ensure a generative model adheres to clinical knowledge is challenging\". Their presentation is very clear, and empirical evaluations & ablations are convincing. I recommend this paper for acceptance beyond my concerns around how well this paper fits into NeurIPS venue.\n\n", "questions": "- How do you think your findings might change as the diffusion models get stronger in creating more plausible images (given the rate of implausible images was higher than 80% for some cell types)?\n- How do you think you might generalize your method to other modalities such as interpreting MRI images where clinical plausibility criteria might be more limited? (might be harder to define the criteria beyond aliasing artifacts in certain anatomies?)", "limitations": "Authors have not explicitly described limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Very clear presentation\n- Simple but powerful idea that brings in an idea from a different domain (LLMs) to medical imaging\n- Convincing empirical results and ablations", "weaknesses": "My main worry about this paper is that NeurIPS might not be the best venue, given the algorithmic novelty is limited here. However, I do think this is an interesting application of integrating domain-specific knowledge to models through RLHF-style training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please refer to the combined response above for a general overview of the improvements and changes that have been incorporated in the revised manuscript.\n\n**Weaknesses**\nThis paper serves as an application submission to NeurIPS, aligning with the conference's well-established track for such submissions. While our paper is not motivated by a core technical problem, we believe that the application of ideas from the RLHF to clinical problems is by itself a novel idea that might have significant impact in our application domain of interest. We believe that presenting our work at NeurIPS would provide valuable insights to the broader AI and machine learning community, demonstrating the potential impact of integrating domain-specific knowledge in real-world applications. \n\n**Questions**\n\n1- As diffusion models become stronger and generate more clinically plausible images, we anticipate that the rate of implausible images would decrease. However, the importance of incorporating human feedback in the loop would still remain significant for several reasons:\n\n    (1)\tEnsuring clinical validity: Even with stronger diffusion models, it is crucial to align the generated images with clinical knowledge. Expert feedback would continue to play a vital role in refining the model's performance and ensuring clinical validity for various cell types and rare diseases.\n\n    (2)\tLearning new clinical concepts: Human feedback not only helps improve the quality of synthetic images but also teaches the model new clinical concepts not annotated in the original training data. This aspect would remain valuable regardless of the initial strength of the diffusion model.\n\n    (3)\tAdapting to evolving medical knowledge: Medical knowledge and best practices continue to evolve over time. Incorporating human feedback allows the model to stay up-to-date with the latest clinical understanding and maintain its relevance and utility in the ever-changing healthcare landscape.\n\n    In conclusion, while we acknowledge that stronger diffusion models may generate more plausible images, the integration of human feedback would still be crucial for ensuring clinical validity, learning new concepts, and adapting to evolving medical knowledge.\n\n2-  We acknowledge the potential difficulties in defining criteria for modalities like MRI; however, we believe our method can still be adapted and generalized with some modifications:\n\n    (1)\tCollaborating with domain experts: For modalities like MRI, it is essential to work closely with radiologists or other relevant domain experts who possess the necessary knowledge to identify and define the clinical plausibility criteria. Their expertise would help to set guidelines and provide feedback on the generated images.\n\n    (2)\tDeveloping modality-specific reward models: To account for the unique characteristics and challenges associated with different imaging modalities, we can develop modality-specific reward models. These models would be trained to predict expert feedback on the generated images for the particular modality, ensuring that the human-in-the-loop framework remains effective in refining the generated images' clinical plausibility.\n\n    (3)\tLeveraging auxiliary data: In cases where clinical plausibility criteria are harder to define, we can leverage auxiliary data, such as clinical reports, annotations, or patient metadata, to guide the model training and evaluation. This additional information can help the model learn more nuanced and context-specific criteria for generating clinically plausible images.\n\n    (4)\tIncorporating active learning techniques: In situations where defining clinical plausibility criteria is challenging, active learning techniques can be employed to selectively query expert feedback on the most uncertain or ambiguous cases. This approach would ensure that the model gains the most valuable information from experts, even when the criteria are not easily defined.\n\n    In conclusion, we believe that our method can be generalized to other imaging modalities, such as MRI, by closely collaborating with domain experts, developing modality-specific reward models, leveraging auxiliary data, and incorporating active learning techniques. These adaptations would help ensure the effectiveness of our human-in-the-loop framework in generating clinically plausible images across various modalities.\n"}, {"review_id": "hVndxTule6", "reviewer": "Reviewer_Mvby", "summary": "This paper proposed to use the diffusion model with expert feedback to improve the quality of synthetic medical images. The key motivation is the embedding of RLHF for medical image generation. To achieve this, there are three steps: (1) pre-trained a generation model and collected expert feedback based on several manual rules; (2) trained a model to predict the feedback scores; (3) incorporated the scores to fine-tune generative models. Extensive experiments demonstrate the effectiveness of the proposed model. Overall, this is an interesting and sound paper.", "questions": "See the above weaknesses.", "limitations": "Besides the comments on the paper's weaknesses, please also consider:\n\n1. Even though it is a kind of application paper, please add some discussions to state the technical contributions.\n2. The authors claimed such a synthetic way could be useful for rare diseases. Please provide the evidence.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. RLHF-based generation model for medical knowledge incorporation. Interesting and useful.\n2. Reasonable framework. The proposed model is based on the conditional diffusion models and the whole process is sound.\n3. Extensive evaluation. Besides the commonly-used quality-related metrics and downstream performance, the new knowledge discovery and visual results are also impressive.", "weaknesses": "1. Reproductivity. It's better to make the codes public and improve the reproductivity.\n2. Manual rules. The expert feedback is highly related to the manual design rules. It's better to see the performance with different amounts of rules or with more rules. Also, please elaborate more on how to define these rules. In other words, it is not clear why the designed rules can indicate the complex image quality and small image differences.\n3. Downstream performance. There is only one downstream task to denote effectiveness. More tasks should be considered. Meanwhile, why not show the performance of downstream models trained by both synthetic and real images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. Please refer to the combined response above for a general overview of the improvements and changes that have been incorporated in the revised manuscript.\n\n**Weaknesses**\n\n1- We have prepared a Github repository with all the codes required to reproduce our experiments. The codes will be released upon acceptance and the end of the anonymity period.\n\n2- As you mentioned, we have indeed demonstrated the relationship between the amount of expert feedback and model performance in **Figure 4 a) and b)**. Our results show that as we incorporate more feedback, the quality of synthetic images improves. Regarding the use of more rules or alternative sets of rules, we agree that this could potentially enhance the model performance. However, due to time constraints, we were unable to explore this aspect in depth. Nevertheless, we have ensured that the rules used in our study are comprehensive and based on established clinical knowledge found in relevant textbooks. These rules were carefully chosen to capture the complex image quality and subtle differences between cell types, which are crucial for generating clinically plausible synthetic medical images. In our revised manuscript, we will provide a more detailed explanation of the process of defining these rules and their relevance to the evaluation of complex image quality and small image differences. We will also discuss the potential benefits of exploring additional or alternative sets of rules in future studies, as well as the limitations imposed by time constraints.\n\n3- We appreciate your feedback on the downstream performance evaluation and the suggestion to consider more tasks and explore the performance of downstream models trained using both synthetic and real images. \n\n(1)  We believe that we have demonstrated the effectiveness of our approach through three downstream tasks. First, we showed that the synthetic data quality is improved by both clinician evaluation (**Main Table 3**) and computational-based qualitative measurements (**Main Table 4**). Second, we demonstrated that human feedback could improve the synthetic data's ability for cell type classification in an amount-dependent manner (**Main Table 5, Figure 4 a,b**). Lastly, we showed that specific human feedback could drive the conditional generation of new concepts (**Main Figure 4 c**). \n\nHowever, your suggestion inspired us to perform another downstream task, where we conducted a Turing test-style experiment: asking a pathologist to distinguish synthetic data from real data and evaluating whether the fine-tuned model can confuse the pathologist more. **Rebuttal Table 5** showed that pathology-in-the-loop framework could indeed improve model’s performance in generating high quality clinical images data that could confuse the pathology expert more, representing by the decrease in accuracy.\n\n(2) We appreciate your suggestion to explore the performance of downstream models trained using both synthetic and real images. We conducted the experiment (**Rebuttal Table 4**). As can be seen from the table, leveraging both real and synthetic data together benefits morphology-based cell classification. We will incorporate these findings into the refined manuscript.\n\n**Limitations**\n\nThanks for your suggestions. In the revised manuscript, we will add a more elaborate discussion on the technical contribution. Regarding the potentially utilization in diagnosing rare diseases: we would like to highlight two key aspects of our pathologist-in-the-loop framework that support its potential application in this context:\n\n1.\tOur framework facilitates the generation of high-quality synthetic images for rare cell types, which may have a frequency of less than 1% in healthy individuals. While common image generation pipelines might struggle to generate high-quality images due to the low frequency of these cell types, our pathologist-in-the-loop framework helps improve image quality. This can be particularly valuable in diagnosing rare diseases where high-quality images of rare cell types are essential.\n\n2.\tOur method, as described in **Subsection 2.4**, allows for the rapid incorporation of new concepts into trained medical image generative models. The concept introduced in this study for differentiating subtypes based on nuclear morphology can potentially be extended to identify pathological variations observed in cancer diagnoses, such as changes in nuclear size, shape, texture, and nuclear: cytoplasmic ratio. Additionally, this approach could be applied to cytoplasmic features. For rare diseases, generating synthetic data using our framework may provide the best option for creating sufficient training data to develop and evaluate AI models effectively.\n"}]

Author rebuttal:
We thank all reviewers for the comprehensive and constructive feedback on our submission. The valuable input received has significantly contributed to improving our work. We have prepared an extensive, point-by-point response for each reviewer, outlining our plans to address their concerns and suggestions for additional analyses to enhance the manuscript. **All new figures and tables can be found in the attached one-page PDF file.** We believe that our response will address the reviewers' concerns and allow us to promptly resolve any remaining minor issues.

**Summary of improvements based on reviewers' comments**

1. External validation:

We agree that evaluating our methodology on additional datasets is essential for validating its applicability across various contexts. To address this, we collected an independent dataset from a separate hospital, and our pathologist-in-the-loop approach still demonstrated significant improvements in the model's performance, indicating its effectiveness across multiple datasets (**Rebuttal Table 3**).

2. More comprehensive comparison and evaluation

We introduced an additional baseline control using feedback from a real/fake binary classifier to distinguish between plausible and implausible images without relying on pathologist feedback or cell-type labels  (**Rebuttal Table 1**). Furthermore, we included a comparison where we trained the downstream classifier with both synthetic and real images (**Rebuttal Table 4**). Lastly, we added a Turing test-style experiment, asking a pathologist to differentiate synthetic data from real data (**Rebuttal Table 5**). These additional experiments collectively enhance the quality and rigor of our work.

3. Validation of the reward model

We presented the reward function performance on a held-out validation set annotated by clinical experts (**Rebuttal Table 2**). The results show that the reward function's performance improves with more feedback, correlating with the downstream classification performance demonstrated in **main Figure 4**. This offers a more comprehensive understanding of the model's effectiveness.

4. A more comprehensive comparison illustrating clinician feedback

To better understand the distinctions between plausible and implausible images, we included a visualization of 128 images (64 plausible and 64 implausible) in **Rebuttal Figure 1**. Additionally, we added an appendix containing 512 images, with 32 images for each cell type, to provide a deeper understanding of the different cell types and variations in image quality. These additional visualizations and comparisons effectively address the reviewers' concerns and enhance the clarity of our paper.

Once again, we would like to thank all reviewers for their comments and we are looking forward to the discussion period.

Meta-review: All reviewers land on the positive side. They all enjoy the proposed pathologist-in-the-loop framework. This framework was evaluated extensively both qualitatively and quantitatively, showing its ability to generate clinically-plausible synthetic medical images. The AC suggests acceptance and encourages the authors to include the discussion in the rebuttal into the final version.
Decision: Accept (spotlight)