Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback
Authors: TaeHo Yoon
Kibeom Myoung
Keon Lee
Jaewoong Cho
Albert No
Ernest K. Ryu
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative models
Diffusion probabilistic models
Controlled generation
Human Feedback
RLHF
TL;DR: We show very small amount of human feedback is sufficient for aligning a diffusion model's sample generation with specified requirements.

Abstract: Diffusion models have recently shown remarkable success in high-quality image generation. Sometimes, however, a pre-trained diffusion model exhibits partial misalignment in the sense that the model can generate good images, but it sometimes outputs undesirable images. If so, we simply need to prevent the generation of the bad images, and we call this task censoring. In this work, we present censored generation with a pre-trained diffusion model using a reward model trained on minimal human feedback. We show that censoring can be accomplished with extreme human feedback efficiency and that labels generated with a mere few minutes of human feedback are sufficient.

Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback
TaeHo Yoon'
Kibeom Myoung' Keon Lee" Jaewoong Cho Albert No
Ernest K. Ryu
Department of Mathematical Science, Seoul National University
2 Interdisciplinary Program in Artificial Intelligence, Seoul National University
9 Department of Electronic and Electrical Engineering, Hongik University
KKRAFTON
Abstract
Diffusion models have recently shown remarkable success in high-quality im- age generation. Sometimes, however, pre-trained diffusion model exhibits par- tial misalignment in the sense that the model can generate good images, but it sometimes outputs undesirable images. If so, simply need prevent the generation of the bad images, and we call this task censoring. In this work, we present censored generation with pre-trained diffusion model using a reward model trained on minimal human feedback. We show that censoring can be ac- complished with extreme human feedback efficiency and that labels generated with mere few minutes of human feedback are sufficient. Code available at: https: //t::://ttttbb http//ttttiiimiiiie
human coman--feddbac. feedback.
1 Introduction
Diffusion probabilistic models [19, 12, 42) have recently shown remarkable success in high-quality image generation. Much of the progress is driven by scale [35, 36, 38], and this progression points to future of spending high costs to train a small number of large-scale foundation models [4] and deploying them, sometimes with fine-tuning, in various applications. In particular cases, however, such pre-trained diffusion models may be misaligned with goals specified before or after the training process. An example of the former text-guided diffusion models occasionally generating content with nudity despite the text prompt containing no such request. An example scenario of the latter is deciding that generated images should contain certain type of concepts (for example, human faces) even though the model was pre-trained on images with such concepts.
Fixing misalignment directly through training may require impractical cost of compute and data. To train a large diffusion model again from scratch requires compute costs of up to hundreds of thousands of USD [30, 29]. To fine-tune a large diffusion model requires data size ranging from 1,000 [28] to 27, 7,0000 [25].' We argue that such costly measures are unnecessary when the pre-trained model is already capable of sometimes generating "good" images. If so, simply need to prevent the generation of "bad" images, and we call this task censoring. (Notably, censoring does not aim to improve the "good" images.) Motivated by the success of reinforcement learning with human feedback (RLHF) in language domains [9, 50, 43, 33], we perform censoring using human feedback.
In this work, we present censored generation with pre-trained diffusion model using reward model trained on extremely limited human feedback. Instead fine-tuning the pre-trained diffusion model, train reward model on labels generated with few minutes of human feedback and perform guided generation. By not finettning the diffusion model (score network), we reduce both compute
'The prior work [28] fine-tunes pre- pre-trained diffusion model on a new dataset of size 1k using so-called adapter module while [25] improves text-to-image alignment using 27k
data.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

7
7
7
7 7
7
7 7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
77
7
7
7
7
7
7
777
7
7
7
7
7
7
7
7
7
7
7777177
7
7
7 7 7 7
7
7
7
7
7
7
7
7
7
7 7 7 7
7
7
7 7 7
7 7 7
7
7
7
7
7
7
7
7
7
7
7
4
7
7 7 7
7
7
7
7
7
7
2
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
(a) Baseline:
MNIST class "7"
(h) Censored: Crossed 7
X3O
I D
corstakk
(c) Baseline: LSUN Church with LDM
(d) Censored: Stock photo watermarks
(c) Baseline: ImageNet class "tench" (fish)
(f) Censored: Human faces
Bascline: LSUN bedroom
(h) Censored: Broken images
Uncensored baseline vs. censored generation. Setups are precisely defined in Section Due to space constraints, we present selected representative images here. Full sets of sel-selected samples are shown in the appendix.
Figure 1:

---

HUAAP 1OO HMMN
A HU MATI À A OMMIA HA
(i) Baseline: Stable Diffusion, "A photo of a human"
(i) Censored: Embedded text
Figure 1: (Continued) Uncensored baseline vs. censored generation. Setups are precisely defined in Section 5. Due to space constraints, we present selected representative images here. Full sets of non-selected samples are shown in the appendix.
and data requirements for censored generation to negligible levels. (Negligible compared any amount of compute and man-hours an ML scientist would realistically spend building system with diffusion model.) We conduct experiments within multiple setups demonstrating how minimal human feedback enables removal of target concepts. The specific censoring targets we consider are: A handwriting variation ("crossed 7"s) in MNIST [11]; Watermarks in the LSUN [47] church images; Human faces in the ImageNet [10] class "tench"; "Broken" images in the generation of LSUN bedroom images.
Contribution. Most prior work focus on training new capabilities into diffusion models, and this inevitably requires large compute and data. Our main contribution is showing that very small amount of human feedback data and computation is sufficient for guiding pre-trained diffusion model do what it can already do while suppressing undesirable behaviors.
1.1 Background on diffusion probabilistic models
Due to space constraints, defer the comprehensive review prior works Appendix D. this section, we briefly review the standard methods of diffusion probabilistic models (DPM) and set up the notation. For the sake of simplicity and specificity, we only consider the DPMs with the variance preserving SDE.
Consider the variance preserving (VP) SDE
dX, X,dt + v B,dW,,
Xo ~ po
for t E [0, where B, > 0, Xx Rd, and W, is d-dimensional Brownian motion. The process (X,,7 has the marginal distributions given by
X 2 Va,Xo + v1 0get,
a: - B.ds 8t N(0,
for E j0, [J [39, Chapter 5.5]. Let pr denote the density for X, fort [0, T1. Anderson's theorem [1] tells that the the reverse-time SDE by
dx, = Be (- Vog X, dt + B..We.
Xx<pi
D where {W, 1e is reverse-time Brownian motion, satisfies X, Xx ~ Pt.
In DPMs, the initial distribution is set as the data distribution, i.e., Po Pdata in (1), and score network so trained so that so (X, t) x V (X,). For notational convenience. one often uses the evror network (,(X,,)) t) =
(X), t). Then, the reverse-time SDE is approximated by
dx, B,
E0 Xex t)
2 X, dt + B,dW,, ~
VI- a2
3

---

for E [0,
When image X has corresponding label Y, classifier guidance [40, 12] generates images from Pe(X, x Pr(X,, P.(X,))p X,)
fort E [0, 7] using
Eo (x(x,,t t) = (x) - wV ag V p: X()
1 (X,
1 2 X,
dx, = 3r
(v-
dt + v B,
~(0,(
where 0. This requires training separate dime-dependent classifier approximating (Y X2). In the context of censoring, we use a reward model with binary labels in place of a classifier.
Problem description: Censored sampling with human feedback
Informally, our goal is:
Given pre-trained diffuusin model that is partially misaligned in the sense that generates both "good" and "bad" images, fix/modify the generation process so that only good images are produced.
The meaning of "good" and "bad"" depends on the context and will be specified through human feedback. For the sake of precision, we define the terms "benign" and "malign" to refer to the good and bad images: A generated image is malign if it contains unwanted features to be censored and is benign if not malign.
Our assumptions are: (i) the pre- pre-trained diffusion model does not know which images are benign or malign, (ii) human is willing to provide minimal (~ minutes) feedback distinguish benign and malign images, and (iii) the compute budget is limited.
Mathematical formalism. Suppose pre-trained diffusion model generates images from distribu- tion Pdata (2: containing both benign and malign images. Assume there is function r(x) E (0, 1) representing the likelihood of x being benign, i.e., r(z) considered for sampling while r(z) R 0 means image is malign and should not be sampled. We mathematically formalize our goal as: Sample from the censored distribution
means image is benign and should be
Poensor (x) Xx (r)r).
Human feedback. The definition of benign and malign images are specified through human feedback. Specifically, we ask human annotator provide binary feedback E {0, for each image X through a simple graphical user interface shown in Appendix E. The feedback takes 1-3 human-minutes minutes for the relatively easier censoring tasks and most 10-20 20 human-minuter for the most complex task that we consider. Using the feedback data, we train reward model r 2 which we further detail in Section 3.
The evaluation criterion of our methodology are the human time spent providing feedback, quantified by direct measurement, and sample quality, quantified by precision and recall.
Evaluation.
In this context, precision is the proportion of benign images, and recall is the sample diversity of the censored generation. Precision can be directly measured by asking human annotators to label the final generated images, but recall is more difficult to measure. Therefore, we primarily focus on precision for quantitative evaluation. We evaluate recall qualitatively by providing the generated images for visual inspection.
Reward model and human feedback
Let Y be random variable such that Y 1 if X is benign and Y 0 X is malign. Define the time-independent reward function as
r(X) P(Y 1|x).

---

Algorithm Reward model ensemble
Require: Images: malign (xx(), ,..,,,.. (, benign {x((M+ ..... (NN)r (NMs NB) for k = .....: K do
Randomly select with replacement NM benign samples X ((Nuttt), y... X (Nartis)))
've Train reward model (8) with (x(1), ................................ ( (Na))) ( (N(Nt)))), ...., ( ((NutiN end for return No = [I U2
(6)
As we later discuss in Section 4, time- iime-eepndentt guidance requires time-dependent reward function. Specifically, let ~ Pdata and be its label. Let (X, (e))r be images corrupted by the VP SDE (1) with X, = X. Define the rime-dependent reward function as
(X,) = P(Y = x))))
for E [O,
We approximate the reward function r with reward model rv, i.., we train
Ty (X) R r(X) or
To (X,, (x.),,,
using human feedback data (x YY),) (x(M) (So the time-dependent reward model uses (x((),, y()) as training data.) We use weighted binary cross entropy loss. In this section, describe the most essential components of the reward model while deferring details to Appendix
The main technical challenge is achieving extreme human-feedbace efficiency. Specifically, we have N < 100 most setups we consider. Finally, we clarify that the diffusion model (score network) not trained or fine-tuned. We use relatively large pre-trained diffusion models [12, 36), but we only train the relatively lightweight reward model ry--
3.1 Reward model ensemble for
setups
In some setups, benign images constitute the majority of uncensored generation. Section 5.2 considers such a beeign-ddmiinnnt setup, where 11.4% of images have stock photo watermarks and the goal is to censor the watermarks. A random sample of images provided to a human annotator will contain far more benign than malign images.
To efficiently utilize the imbalanced data a sample-efficient way, we propose an ensemble method loosely inspired by ensemble-base sample efficient RL methods [23, 6]. The method trains K (K) each using a shared set of NM (scarce) malign images joined with NM benign images randomly subsampled bootstrap-style from the provided pool of Ng (abundant)
reward models r ,...,, F0x
benign data as in Algorithm 1. The final reward model is formed as Tv = II "ix Given that
[k)
a product becomes small when any of its factor is small, Tw is effectively asking for unanimous approval across (1)
Pi T (K)
In experiments,
whose K 5. We the same neural network architecture for ra dix parameters i VK either independently randomly initialized transferred from the same pre-trained weights as discussed in Section 3.3. We observe that the ensemble method significantly improves the precision of the model without perceivably sacrificing recall.
(K)
3.2 Imitation learning for malign-dominant setups
In some setups, malign images constitute the majority of uncensored generation. Section 5.3 considers such a malign-dominant setup, where 69% of images are tench (fish) images with human faces and the goal is to censor the images with human faces. Since the ratio of malign images starts out high, a single round of human feedback and censoring may not sufficiently reduce the malign ratio.
Therefore, we propose an imitation learning method loosely inspired by imitation learning RL methods such as DAgger [37]. The method collects human feedback data in multiple rounds and improves the reward model over the rounds as described Algorithm 2. Our experiment of Section 5.3 indicates that 2-3 rounds of imitation learning dramatically reduce the ratio of malign images. Furthermore,
5

---

Algorithm 2 Imitation learning of reward model
Require:
Pre-trained Eg. Initialize D 0.
Sample x ((N) using Eo and no censoring.
Receive y ((N)) from human feedback. Add data buffer: <- (xx(), y(0)
Train reward model Ti with D.
for ................................ do
Sample (1)
( using Eg and censoring with Tap-
Receive y(1) (N-) from human feedback. Add data buffer: D 1|)
Train reward model To with D.
end for return Tg
imitation learning is a practical model of an online scenario where one continuously trains and updates the reward model rs while the diffusion model continually deployed.
Ensemble vs. imitation learning. In the benign-dominant setup, imitation learning is too costly in terms of human feedback since acquiring sufficiently many (~ 10) malign labels may require the human annotator to go through too many benign labels (~ 1000) for the second round of human feedback and censoring. In the malign-dominant setup,
reward model ensemble, where reward models share the benign data while hootstrap-subssmpling the malign data, but we empirically observe this to be ineffective. We attribute this asymmetry to the greater importance of malign data over benign data; the training objective is designed so as our primary goal is to censor malign images.
3.3 Transfer learning for tine-independent reward
To further improve human-feedbace efficiency, we use transfer learning. Specifically, we take ResNet18 model [17, 18] pre-trained on ImageNetlk [10] and replace the final layer with randomly initialized fully connected layers which have 1-dimensional output features. We observe training all layers be more effective than training only the final layers. We use transfer learning only for training time-independent reward models, as pre-trained time-dependent classifiers are less common. Transfer learning turns out to essential for complex censoring tasks (Sections 5.2, 5.4 and 5.5), but requires guidance techniques other than the simple classifier guidance (see Section 4).
4 Sampling
In this section, we describe how to perform censored sampling with trained reward model rig- We follow the notation of Section 1.1.
Time- denendee guidance. tion follows the SDE
Given time-dependent reward model (X,, f), censored genera-
E 8 e, 1)
wT 02 V.V log (x,)
1 E0 _,,,/ dt + V B.dW..
dx, 3r
X T X7~0((011)
V1-
arguments [42, Section IJ. it follows that Xo Peensor (I:) (r)t approximately when w 1. The parameter w O, which we refer to as the guidance weight, controls the strength the guidance, and is analogous to the "gradient scale" used in prior works [12]. Using w 1 can be viewed as a heuristic strengthen the effect of the guidance, or can be viewed as an effort to sample from Peensor O Pd
for E [0, 2" with w > O. From the standard
Time-independent guidance. Given a time-independent reward model Tw (X.), we adopt the ideas of universal guidance [2] and perform censored generation via replacing the ég of (2) with
E (X. Eo (X,,, t) wv1 a, V log r(Xo))
where
Xo = E|Xo |x, = X.] = X, VI OLEN X,,t)
00t

---

1.6
6.0
malign generation
1.4
Proportion of malign generation
5.0
Proportion
1.2
4.0
0.4
1.0
Single
Union
Ensemble +Universal
Ensemble
Single
Union
Ensemble
Ensemble +Universal
MNIST: Censoring "crossed 7"
(b) L.SUN church:
Censoring watermarks
Figure 2: Mean proportion of malign images after censoring with standard deviation over trials, each measured with 500 samples. Reward ensemble outperforms non-ensemble models, and the universal guidance components further improve the results. Left: Censoring "crossed 7" from MNIST. Before censoring, the proportion is 11.9%. The mean values each point are: 1.30%, 0.98%, 0.60%, and 0.42%. Right: Censoring watermarks from LSUN Church. Before censoring, the proportion is 11.4%. The mean values of each point are: 3.02%, 3.84%, 1.36%, and 0.76%.
for f E [0] with O. To clarify, v differentiates through Xo. While this method has mathematical guarantees, prior work [2] has shown strong empirical performance in related setups."
Backward guidance and recurrence. The prior work [2] proposes backward guidance and self- recurrence to further strengthen the guidance. We find that adapting these methods to our setup improves the censoring performance. provide the detailed description in Appendix G.
Experiments
We now present the experimental results. Precision (censoring performance) evaluated with human annotators labeling generated images. The human feedback time we report includes annotation of training data for the reward model Ty. but does not include the annotation of the evaluation data.
5.1 MNIST: Censoring 7 with strike-through cross
In this setup, we censor handwriting variation called "crossed 7", which has horizontal stroke running across the digit, from an MNIST generation, as shown in Figure la. We pre-train our own diffusion model (score network). In this benign-dominant setup, the baseline model generates about 11.9% malign images.
We use 10 malign samples to perform censoring. This requires about 100 human feedback labels in total, which takes less than 2 minutes to collect. We observe that such minimal feedback is sufficient for reducing the proportion of crossed 7s to 0.42% as shown in Figure 1b and Figure 2a. Further details are provided in Appendix H.
Ablation studies. We achieve best results by combining the dimendependent reward model ensemble method described in Section 3. I and the universal guidance components (backward guidance with recurrence) detailed in Appendix G. We verify the effectiveness of each component through an ablation study, summarized in Figure 2a. Specifically, we compare the censoring results using a reward model ensemble (labeled "Ensemble" in Figure 2a) with the cases of using (i) single reward model within the ensemble (trained on 10 malign and 10 benign images; labeled "Single") and (ii) standalone reward model separately trained the union of all training data (10 malign and 50 benign images; labeled "Union") used in ensemble training. We also show that the backward and recurrence components do provide an additional benefit (labeled "Ensemble+U Universs=="i
'If' we simply perform time-dependent guidance with a time-indepen inneeindeeendent reward function re (X), the observed performance is poor. This is because r (x fails to provide meaningful guidance when the input is noisy, and this empirical behavior agrees with the prior observations of [32, Section 2.4] and [2, Section 3.1].
7

---

5.2 LSUN church: Censoring watermarks from latent diffusion model
In the previous experiment, we use a full-dimensional diffusion model that reverses the forward diffusion (1) in the pixel space. In this experiment, we demonstrate that censored generation with minimal human feedback also works with latent diffusion models (LDMs) [46, 36), which perform diffusion on nower-dimensional latent representation of (variational) autoencoders. We use an LDM pre-trained on the 256 256 LSUN Churches [36] and censor the stock photo watermarks. In this benign-dominant dominant setup, the baseline model generates about 11 11.4% malign images.
Training time-dependent reward model in the latent space be used with an LDM would introduce additional complicating factors. Therefore, for simplicity and to demonstrate multiple censoring methods, we train a imeeiindeeendentt reward model ensemble and apply -ideeindeenndntt guidance efficiency, we use pre-trained ResNet 88
as outlined in Section 4. To enhance model and use transfer learning as discussed in Section 3.3. We use 30 malign images, and the human feedback takes approximately 3 minutes. We observe that this is sufficient for reducing the proportion of images with watermarks to 0.76% as shown in Figure 1d and Figure 2b. Further details are provided in Appendix I.
Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. As in Section 5.1, we verify the effectiveness of each component through an ablation study, summarized in Figure 2b. The label names follow the same rules as in Section 5.1. Notably, on average, the "single" models trained with 30 malign and 30 benign samples outperform the "union" models trained with 30 malign and 150 malign samples.
5.3 ImageNet: Tench (fish) without human faces
Although the ImageNetlk dataset contains explicit human classes, the dataset does contain human faces, and diffusion models have a tendency to memorize them [7]. This creates potential privacy risks through the use of reverse image search engines [3]. A primary example is the ImageNet class "tench" (fish), in which the majority of images are humans holding their catch with their celebrating faces clearly visible and learnable by the diffusion model.
In this experiment, we use a conditional diffusion model pre-trained on the 128 x 128 ImageNet dataset [12] as baseline and censor the instances of class "tench" containing human faces (but not other human body parts such as hands and arms). In this malign-dominant setup, the baseline model generates about 68.6% malign images.
We perform rounds of imitation learning with 10 malign and 10 benign images in each round to train a single reward model. The human feedback takes no more than 3 minutes in total. We observe that this is sufficient for redueing the proportion of images with human faces to 1.0% as shown in Figure If and Figure 3. Further details are provided in Appendix J.
Ablation studies. We verify the effectiveness of imitation learning by comparing it with training the reward model at once using the same number of total samples. Specifically, we use 20 malign and 20 benign samples from the baseline generation to train a reward model (labeled "non-imitation (20 malign)" in Figure 3a) and compare the censoring results with round 2 of imitation learning: similarly we compare training at once with 30 malign and 30 benign samples (labeled "non-imitation (30 malign)") and compare with round 3. We consistently attain better results with imitation learning. As in previous experiments, the best precision is attained when backward and recurrence are combined with imitation learning (labeled "30+Univ"").
We additionally compare our censoring method with another approach: rejection sampling, which simply generates samples from the baseline model and rejects samples X such that T. (X) is less than the given acceptance threshold. Figure 3b shows that rejection sampling yields worse precision compared to the guided generation using the same reward model, even when using the conservative threshold 0.8. We also note that rejection sampling in this setup accepts only 28.2% and 25.5% of the generated samples respectively for thresholds 0.5 and 0.8 on average, making it suboptimal for situations where reliable real-time generation is required.
8

---

imitation learning
Proportion malignn generation (%)
non-imitation [20 malign)
Proportion of malign generation (%)
non-imitation 130 malign)
40
30
20
Rejection (0.5)
Rejection (0.8)
Guidance (Round 1)
30+Univ
& malign training samples
(a) Ablation results for imitation learning
(b) Comparsion with rejection sampling
Mean proportion of malign tench images (w/ human face) with standard deviation over trials, each measured with 1000 samples. Left: Before censoring, the proportion is 68.6%. Using imitation learning and universal guidance, it progressively drops 17.8%, 7.5%, 2.2%, and 1.0%. Non-imitation learning is worse: with 20 and 30 malign images, the proportions are 10.7% and 6.8%. Right: With acceptance thresholds 0.5 and 0.8, rejection sampling via reward models from round produces 32.0% and 29.8% 998%% of malign images, worse than our proposed guidance- guidance-based censoring.
Figure 3:
malign generation
Proportion of
Single
Union
Ensemble
Ensemble Universsl
Rejection (0.5)
Rejection (0.8)
Figure 4: Mean proportion of malign (broken) bedroom images with standard deviation over 5 trials, each measured with 500 samples. Before censoring, the malign proportion is 12.6%. The mean values of each point are: 8.68%, 8.48%, 4.56%, 1,36%, 4. 4.16%, and 2.30%.
5.4 LSUN bedroom: Censoring broken bedrooms
Generative models often produce images with visual artifacts that are apparent to humans but are difficult to detect and remove via automated pipelines. In this experiment, diffusion model trained on 256 256 LSUN Bedroom images [12] and censor "broken" images as perceived by humans. In Appendix K, we precisely define the types of images consider to be broken, thereby minimizing subjectivity. In this bemign-dominant setup, the baseline model generates about 12.6% malign images.
pre-trained
This censoring task is the most difficult one we consider, and we design this setup as a "worst case" on the human work our framework requires. We use 100 malign samples to train reward-model ensemble. This requires about 900 human feedback labels, which takes about 15 minutes to collect. To enhance human-feede feedbback efficiency,
pre-trained ResNet18 model and use transfer learning as discussed Section 3.3. We observe that this is sufficient for reducing the proportion of malign images to 1.36% as shown in Figure Ih and Figure 4. Further details provided in Appendix K.
Ablation studies. We achieve our best results by combining the (ime-independent) reward ensem- ble and backward guidance with recurrence. We verify the effectiveness of each component through an ablation study summarized in Figure 4. We additionally find that rejection sampling, which rejects (k)
ΣK Tx (X) is less than threshold, yields worse precision compared to the guided generation using the ensemble model and has undesirably low average acceptance ratios of 74.5% and 55.8% when using threshold values 0.5 and 0.8. respectively.
sample X such that K Σk=1
'https: tattps://githus wwt.th/www///ema//iiiddddddddaad

---

Proportion of malign generation (%)
6-
1
Single
Union
Ensemble
Ensemble +Universal
Figure 5: Mean proportion of malign images (with embedded text) with standard deviation over trials, each measured with 500 samples. Before censoring, the malign proportion is 23.7%. The mean values of each point are: 7.10%, 7.95%, 3.86%, 1.24%.
5.5 Stable Diffuusio: Censoring unexpected embedded texts
Text-to-iman imoge diffusion models, despite their remarkable prompt generality and performance, are known often generate unexpected and unwanted artifacts or contents. In this experiment, we use Stable Diffusion" v1.4 to demonstrate that our methodology is readily applicable to aligning text-to-image models. The baseline Stable Diffusion, when given the prompt "A photo occasionally produces prominent embedded texts only display texts without any visible human figures (as in Figure 1i). We set the output resolution 512 512 and censor the instances of this behavior. In this benign-dominant setup, the baseline model generates about 23.7% malign images,
human",
This censoring problem deals with the most corplex-structured model and demonstrates the effec- tiveness of our methodology under the text-conditional setup. We use 100 malign samples to train a -ooell ensemble. This requires about 600 human feedback labels, which takes no more than minutes to collect. We use pre-trained ResNet1 8 model and use transfer learning as discussed in Section 3.3. We observe that this is sufficient for reducing the proportion of malign images 1.24% as shown in Figure Ij and Figure 5. Further details are provided in Appendix I..
Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. We verify the effectiveness of each component through an ablation study, summarized in Figure 5. Similarly in Section 5.2, observe that the "single" models outperform the "union" models on average.
Note the guidance weight w.. We speculate that the effective scale of the guidance weight grows (roughly doubles) with 1) significant scale growth in terms of data size and 2) the introduction of new modality (e.g. unconditional or class-conditional - text-conditional model), We use w 1.0 for the simplest task of Section 5.1, while we use w = 2.0 for Sections 5.2 and 5.4 where the data size grows to 256 x 256. For this section where we use text-conditioning, we use w 4.0.
Conclusion
In this work, we present censored sampling of diffusion models based on minimal human feedback and compute. The procedure is conceptually simple, versatile, and easily executable, and we anticipate our approach find broad use in aligning diffusion models. our view, that diffusion models can be controlled with extreme data- titttefffiiency without fine-tuning of the main model weights, is an interesting observation its own right (although the concept of guided sampling itself is, of course, not new [40, 12, 32, 35]). We are not aware of analogous results from other generative models such as GANs language models; this ability to adapt/guide diffusion models with external reward functions seems to be unique trait, and believe it offers promising direction of future work on leveraging human feedback with extreme sample efficiency.
10

---

Acknowledgments and Disclosure of Funding
TY, KM, and EKR were supported by the Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea (MSI) INO.2021-0-01344 Artificial Intelligence Graduate School Program (Seoul National University)] and a grant from KRAFTON AL AN was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education 202IRIFIA1059550 We thank Sehyun Kwon and Donghwan Rho for providing valuable feedback.
References
[1] B. D. Anderson. Reverse-time diffusion equation models. Applications, 12(3):31--32, 1982.
Stochastic Processes and their
[2] A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Gold- stein. Universal guidance for diffusion models. arXiv preprint arXiv:2302.007121, 2023.
[3] A. Birhane and U. Prabhu. Large image datasets: A pyrrhic win for computer vision? WACV,
2021.
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein.
J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfssonn, S. Buch, D. Card, R. Castellon, N. Chatterji, A. S. Chen, K. A. Creel, J. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie,
K. Goel, N. D. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. F. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. F. Nyarko, G. Ogut, L. Orr, Papadimitriou, Park, Piech, E. Portelance, C. Potts. A. Raghunathan, R. Reich, H. Ren, Rong, Y. H. Roohani, C. Ruiz, J. Ryan, C. R'c, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. P. Taori, W. Thomas,
Tramèr, E. Wang. W. Wang. B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. A. Zaharia, M. Zhang, T. Zhang, X. Zhang, Zhang, Zheng, K. Zhou, and Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv: 21.22108.075 07258, 2021.
Srinivasan, Tamkin,
[5] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3 39(3-4):324 41:324-34444 345, 1952.
[6] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. NeurlPS, 2018.
[7] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, Sehwag, F. Tramèr, B. Balle, D. Ippolito, and
E. Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 2023.
[8] Chong, I.-C. Shen, I. Sato, and T. Igarashi. Interactive optimization of generative image modelling using sequential subspace search and content-based based guidance. Computer Graphics Forum, 2021.
[9] F. Christiano, Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. NeurIPS, 2017.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.F. Li. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
[11] L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916)14 2012.
[12] Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. NeurIPS, 2021.
[13] R. O. Duda, P. F. Hart, and Stork. Puttern Classification. John Wiley Sons, 2006.
[14] B. Efron. Estimating the error rate of prediction rule: Improvement on cross-validation.
Journal of the American statistical association, 78(382):316333) 1983.

---

[15] B. Efron and R. Tibshirani. Improvements on cross-validation: The 632+ bootstrap method. Journal of the American Statistical Association, 92(438):548-56) 1997.
[16] R. Gandikota, J. Materzynska, models. arXiv preprini arXiv:2303.07345, 2023.
Kauffan, and Bau. Erasing concepts from diffusion
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
[18] K. Hc, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. ECCV, 2016.
[19] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[20] J. Ho and T. Salimans. Classifier-free diffusion guidance. NeurlPS Workshop Deep Genera- tive Models, 2021.
[21] B. Kawar, R. Ganz, and M. Elad. Enhancing diffusion-based image synthesis with robust classifier guidance. Transactions on Machine Learning Research, 2023.
[22] D. Kim, Y. Kim, S. Kwon, W. Kang. and I.-C. Moon. Refining generative process with discriminator guidance in score- based diffusion models. JCML, 2023.
[23] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. ICLR, 2018.
[24] A. K. Lampinen, So, D. Eck, and F. Bertsch. Improving image generative models with human interactions. arXiv preprint arXiv: arXiv:/709. 10459, 2017.
[25] K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S.
Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 2023.
[26] K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedhack-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. ICML, 2021.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. ICLR, 2019.
[28] T. Moon, M. Choi, G. Lee, J.-W. Ha, and J. Lee. Fine-tuning diffusion models with limited data. NeurIPS 2022 Workshop on Score-Based Methods, 2022.
[29] MosaicML. Training stable diffusion from scratch costs <160k. mosai cml com/blog wlllgg/rraining..... sewtaiiinn--stableee cessed: 2023-05-01.
https://ww. wtt.riiiiiiooooossaattctooooooo
160k. Ac-
[30] E. Mostaque. (@EMostaque) "We actually used 256 A100s for this per the model card, 150k hours in total so at market price $600k". httpps///sitteeecccoo//sttttt/a Accessed: 2023-05-01.
156387067411182266.
[31] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. ICML,, 2021.
[32] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Megrew, I. Sutskever, and M. Chen. GLIDE: Towards photorcalistic image generation and editing with text-guided diffusion models. ICML, 2022.
[33] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with
human feedback. NeurlPS, 2022.
[34] R. Ramamurthy, P. Ammanabrolu, K. Brantley, Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. ICLR, 2023.
[35] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint 2rXiv:2204.06125 2022.
[36] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. CVPR, 2022.
[37] Ross, G. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
12

---

[38] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
[39] S. Särkkä and A. Solin. Applied Stochastic Difserential Equations. Cambridge University Press, 2019.
[40] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
[41] Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. ICLR, 2021.
[42] Y. Song, Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based
generative modeling through stochastic differential equations. ICLR, 2021.
[43] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning summarize with human feedback. NeurlPS, 2020.
[44] Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303. 2023.
[45] S. Um and J. C. Ye. Don't play favorites: Minority guidance for diffusion models. arXiv preprint orXiv:2301.1233
2023.
[46] A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. NeurIPS, 2021.
[47] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans the loop. arXiv preprint arXiv: 1505603366333 2015.
[48] E. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.15591, 2023.
[49] J.-Y. Zhu, P. Krähenbühl,, E. Shechtman, and A. A. Efros. Generative visual manipulation the natural image manifold. ECCV, 2016.
[50] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1009.90099988 08593, 2019.
13

---

A Broader impacts & safety
As our research aims to suppress undesirable behaviors diffusion models, our methodology carries the risk of being used maliciously to guide the diffusion model toward malicious behavior. Generally, research on alignment carries the risk of being flipped "align" the model with malicious behavior, and our work is no exception. However, despite this possibility, it is unlikely that our work will be responsible for producing new harmful materials that baseline model not already capable of, as we do not consider training new capabilities into diffusion models. In this sense, our work does not pose a greater risk of harm compared other work on content filtering.
B Limitations
Our methodology accomplishes its main objective, but there few limitations point out. First, although the execution of our methodology requires minimal (few minutes) human feedback, objective evaluation of methodology does require non-trivial amount of human feedback. Indeed, even though we trained our reward models with 10s human labels, our evaluation used 1000s of human labels. Also, the methodology is built the assumption of having access pre- trained diffusion models, and it does not consider how to train new capabilities into the base model or improve the quality of generated images.
C Human subject and evaluation
The human feedback used in this work was provided by the authors themselves. We argue that our work does not require external human subjects as the labeling is based on concrete, minimally ambiguous criteria. For the setups of Sections 5.1 ("crossed 7"), 5.2 ("watermarks"), and 5.3 ("tench") criteria is very clear and objective. For the setup Section 5.4 ("broken" bedroom images), we describe our decision protocol in Section K. For transparency, we present comprehensive censored generation results in Sections H to K.
We used existing datasets- ImageNet, LSUN, and MNIST- our study, These are free of harmful or sensitive content, there is reason expect the labeling task to have any adverse effect on the human subjects.
14

---

D Prior Works
DPM. The initial diffusion probabilistic models (DPM) considered forward image corruption processes with finite discrete steps and trained neural networks to reverse them [40, 19, 41]. Later, this idea was connected to continuous- SDE formulation [42]. As the SDE formalism tends to be more mathematically and notationally elegant, we describe our methods through the SDE formalism, although all actual implementations require using an discretizations.
The generation process of DPMs is controllable through guidance. One approach to guidance is to use conditional score network, conditioned on class labels text information [31, 20, 32, 35, 38). Alternatively. one can use guidance from another external network. Instances include CLIP guidance [32, 35]. which performs guidance with CLIP model pre-trained on image-caption pairs; discriminator guidance [22], which uses a discriminator network to further enforce consistency between generated images and training data; minority guidance [45], which uses perceptual distances to encourage sapling from low-density regions, and using adversarially robust classifier [21] to better align the sample quality with human perception. In this work, we adapt the ideas of (tmme-ependent) classifier guidance of [40, 12] and universal guidance [2].
RLHE Reinforcement learning with human feedback (RLHF) was originally proposed as a method- ology for using feedback to train a reward model, when an explicit reward of the reinforcement learning setup difficult to specify [9, 26]. However, RLHF techniques have been succesfully used in natural language processing setups with no apparent connection to reinforcement learning (50, 43, 33J. While the RLHF mechanism in language domains is not fully understood, the success indicates that the general strategy of fine-tuning or adjusting the behavior of a pre-trained model with human feedback and reward models is a promising direction.
Controlling generative models with human feedback. The use of human feedback to fine-tune generative models has not yet received significant attention. The prior work [24] aims to improve the aesthetic quality of the images produced by generative adversarial networks (GANs) using human feedback. There are methods that allow interactive editing of images produced by GANs (i.e., modifying images based on human feedback) but such methods do not fine-tune or modify the generation procedure of GANs [8, 49].
For DPMs, the prior work of [25] fine-tunes the pre-trained Stable Diffusion [36] model to have better image-text alignment using 27,000 human annotations. In [44], rank-based human feedback was used to improve the generated image of a diffusion model on an image-by-image basis. There has been prior work on removing certain concepts from pre-trained DPMs [16, 48], which involve human evaluations, but these approaches do not use human feedback in their methodologies.
Reward models. Many prior work utilizing human feedback utilize reward models in the form of binary classifier, also called the Bradley-Terry model [5]. However, the specifics of the deep neural network architecture varies widely. In the original RLHF paper [9], the architecture seems to be simple MLPs and CNNs. In [33], the architecture is the same the GPT-3 architecture except that the unembedding layer is replaced with a projection layer to output a scalar value. In [50, 433, the reward model is linear function of the language embedding used in the policy network. In [34], the authors use transformer based architectures to construct the reward models. Overall, the conclusion is that field has not yet converged to a particular type of reward model architecture that is different from the standard architecutres used in related setups. Therefore, we use simple UNet and ResNet18 models for our reward model architectures.
15

---

E GUI interface
We collect human feedback using very minimal graphical user interface (GUT), shown in the following.
X Feedback collector
Red boundary indicates contatins Face (For sure), Blue boundary indicates that undecidable whether the image contaire hunan Face,
Submit
Figure Simple GUI used to collect human feedback for the setup of Section 5.3. Upon user's click, the red boundary appears around an image, indicating that it will be labeled as malign.
16

---

Reward model: Further details
Weighted loss function. We train the reward model using the weighted binary cross entropy loss
BCEa(Tr((;))) v)
log (x; t) (1 - log(1 (=;
(4)
We use < to prioritize the model to accurately classify malign images as malign at the expense of potentially misclassifying some benign images as malign.
Data augmentation. We augment the training dataset with 10 to 20 random variations of each training image using rotation, horizontal flip, crop, and color jitter. We augment the data once and train the reward model to fit this augmented data as opposed to applying a random augmentation every time the data is loaded.
Bootstrap subsampling. As discussed in Section 3.1, we use the reward model ensemble in the benign-dominant setup, where labeled benign images are more plentiful while there is a relatively limited quantity of Nm. malign images. The K reward models of the ensemble utilize the same set of Nm malign images. As the benign images, we implement a resampling strategy that inspired by bootstrapping [14, 15, 13]. Each model selects Nm benign images independently with replacement from the pool of labeled benign images.
G Backward guidance and recurrence
We describe backward guidance and recurrence, techniques inspired by the universal guidance of [2].
G.1 Backward guidance
Compute (x,tt) as (2) or (3) (time-independent or time-dependent guidance) and form
X V &g (X,,t)
fwd
Vot
We then take fdd as a starting point and perform B steps of gradient ascent with respect to log (-) and obtain Xbbwd Finally, replace Eg by ebwud such that X. VaeXbuu /1 ag (x,tt holds, i.e.,
bwd 60
(X,,))
Xx
VI az
G.2 Recurrence
Once abwd computed, the guided sampling implemented discretized step of the backward SDE
c bhud (X,,t) dx, B, (
Xx 2 xx) dt + B,dW,.
Say the discretization step-size At, so the update computes Xt-At from Xx, In recurrent generation, we use the notation x(() X₁ and X A At Ar and then obtain X(2) by following the forward
noise process of the (discretized) VP SDE (1) starting from (x() x for time At. We repeat the process times, sequentially generating x x!2)
( x(R)
17

---

H
MNIST crossed 7: Experiment details and image samples
H.1
Diffusson model
For this experiment, we train our own diffusion model. We use the 5,000 images of the digit "7" from the MNIST training set and rescale them to 32 32 resolution. The architecture the error network Ey follows the UNet implementation' of prior work [12], featuring a composition of residual blocks with downsampling and upsampling convolutions and global attention layers, and time embedding injected into each residual block. We set the input and output channel size of the initial convolutional layer to and 128, respectively, use channel multipliers [1, 2, 2, 2] for residual blocks at subsequent resolutions, and use 3 residual blocks for each resolution. We train the diffusion model for 100,000 iterations using the AdamW [27] optimizer with 3n = 0.9 and 8n = 0.999, using learning rate 10-1, EMA with rate 0.9999 and batch size 256. We use 1,000 DDPM steps.
H.2 Reward model and training
The dime-dependent reward model architecture is a half-UNet model with the upsampling blocks replaced with attention pooling to produce a scalar output. The weights are randomly initialized, i.e., we do not transfer learning. We augment the training (human feedback) data with random rotation in [-20, 20] degrees. When using 10 malign and 10 benign feedback data, we use = 0.02 for the training loss BCF, and train all reward models for 1,000 iterations using Adam with learning rate 3 x 10-1, weight decay 0.05, and batch size 128. When we use 10 malign and 50 benign data for the ablation study, we use a = 0.005 and train for the same number of epochs as used in the training of 10 malign& & 10 benign case, while using the same batch size 128.
H.3 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose 1.0. We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 2a:
"Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 10 malign images, and a set of 10 benign images.
"Union" model refers a model which is trained on 10 malign images and collection of 50 benign images, combining the set of benign images used to train the ensemble. This model is trained for 3,000 iterations, with a 0.005 for the BCF, F. loss.
For these non-ensemble models, we use = 5.0, which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence discussed in Section G, we use 1.0, learning rate 2 x 10-4, B 5, and R
H.4 Censored generation samples
Figure 7 shows uncensored, baseline generation. Figures 8 and 9 shows images sampled with censored generation without and with backward guidance and recurrence.
18

---

7
2
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
1
7
7
7
7
7
7
7
T
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
77771477777
4 7
7
7
7 7
""""""""""""""""""""""""""."""""
772
7
7
77771770
""""""""""""""""""""""""""""""""
7 7
?1171171717777
""""""""""""""""""""""""""""""""
""""""77777"""""""""""
7 0777777
7
7
7
77 7 7
"
7
7 7 7
""
7
7 7
"
7
7
7
1 7
7 7 7
7
7
7
7
7
7
7
1
7
7
7
7
7
7
7
7
7
7
7 ?
7
7
7
7
7
7
7
7
7 7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
Figure 7: Uncensored baseline image samples from the diffusion model trained using only images of the digit "7" from MNIST.
19

---

7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 7
7
7 7 7
7 7 7 77777777700
7
7
7
7
7 7 7 0
77 7
7
7
"F""""""""""""3"""""""""""""""""
""""""""""""""""""""""""""""""""
7 7
7 7
2 7 2 7777777772
7 7
7
""""""""'''"""""""""""""""""""""
7
/7
7
7 7 7 
777
7
7
-
7
7
7 7
7
7
7
7
7
7
T
7
7
7
7
7
7
7
7
7 7
7 7
7
7
7 7
Figure 8:
Non-curated censored generation samples without backward guidance and recurrence.
Reward model ensemble is trained on 10 malign images.
20

---

7
7
7
7
7
7
7
7
7
7
7
7
7
7 7
----""""""""""""""""""
7
7
7777777777
-
7
7 """"""""''"""""""""""""
""""""""""""""""""""""""""""""""
7 -
7 7 "7721772'1117777"""777
7
7 7
7
7777 -"""""1''''""""">>>>>>>>"
7 717717711122777777777777
7 /
17 ""2"127722"2111177777777
/
77 """"7777"'"777""7777777
7 """""""""""""""""">>>>>>>...
7 (777)711""1777"""777777
7 7
7 7
".2177117777777711111
2117777777
7
7
0777700777777777777777
7
7
7 7
7
7
/
7 7
7
7 ""
7
7
7 77
7
Figure 9: Non-curated censored generation samples with backward guidance and recurrence. Reward model ensemble is trained on 10 malign images.
21

---

I LSUN church: Experiment details and image samples
L1 Pre-trained diffusion model
We use the pre-trained Latent Diffusion Model (LDM)* from [36]. We follow the original settings and use 400 DDIM [41] steps.
1.2 Malign image definition
As shown in Figure 10, the "Shutterstock" watermark is composed of three elements: the Shutterstock logo in the center, the Shutterstock website address at the bottom, and a white X lines in the background. In the baseline generation, all possible combinations of these three elements arise. We classify an image as "malign" it includes either the logo in the center the website address the bottom. We do not directly censor the white X lines, as they are often not clearly distinguishable when providing the human feedback. However, we do observe a reduction in the occurrence of the white X lines as they are indirectly censored due to their frequent co-occurrence with the other two elements of the Shutterstock watermark. While the majority of the watermarks are in the Shutterstock format, did occasionally observe watermarks from other companies as well. We choose to censor only the Shutterstock watermarks as the other types were not sufficiently frequent.
clo roueeerrrrre
a urla
shu
067er24m
www.shutternockcom 202300213
Srsss دc
itter e
ucck
L09300001
Figure 10: Examples of LSUN church images with Shutterstock watermarks.
I.3 Reward model training
We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision. * "DEFAULTS" setting", which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with randomly initialized fully connected layer with a one-dinensi output. We train all layers of the reward model using the human feedback dataset of 60 images (30 malign, 30 benign) without data augmentation. We use BCE, in (4) as the training loss with
0.1. The models are trained for 600 iterations using AdamW optimizer [27] with learning rate 3 x 10-4, weight decay 0.05, and batch size 128.
wtw.iissiio.ommmmss...... wwwels resnet18
22

---

1.4 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose 2.0. We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 2b:
"Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 30 malign images, and a set of 30 benign images.
"Union" model refers to model which is trained on 30 malign images and collection of 150 benign images, combining the set of benign images used to train the ensemble. This model is trained for 1,800 iterations, with a 0.01 for the BCE, loss.
For these non-ensemble models, we use = 10.0, which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with recurrence as discussed in Section G, we use = 2.0 and R = 4.
1,5 Censored generation samples
Figure 11 shows uncensored, baseline generation. Figures 12 and 13 present images sampled with censored generation without and with backward guidance and recurrence.
23

---

Figure II: Uncensored baseline image samples. Malign images are labeled with red borders for visual clarity.
24

---

2012
Figure 12: Non-curated censored generation samples without backward guidance and recurrence. Reward model ensemble trained on malign images. Malign images are labeled with red borders for visual clarity.
25

---

Figure 13: Non- con-curated censored generation samples with recurrence. Reward model ensemble is trained on 30 malign images. Malign images are labeled with red borders for visual clarity.
26

---

ImageNet tench: Experiment details and image samples
J.1 Pre-trained diffuusin model
We use the pre-trained diffusion model' from [12], trained on ImagtNet dataset [10]. We use (time-dependent) classifier guidance with gradient scale 0.5 as recommended by [12] and 1,000 DDPM steps for sampling to generate samples from the class "tench".
J.2 Reward model training
We same half-UNet architecture as in Section H for the time-dependent reward model. The weights are randomly initialized, i.e., we do not use transfer learning. All hyperparameters are set identical to the values used for training the time-dependent classifier for 128 x 128 ImageNet in the prior work [12], except that we set the output dimension of the attention pooling layer to 1. We augment the training (human feedback) data with random horizontal flips with probability 0.5 followed by one of the following transformations: 1) random rotation within [-30, 30) degrees, 2) random resized crop with an area of 75-100% and 3) color jitter with contrast range [0.75, 1.33] and hue range (-0.2, 0.2]. We use a 0.1 for the training loss BCE,. When using 10 malign and 10 benign feedback data, we train reward models for 500 iterations using Adam with learning rate 3 x 10-1, weight decay 0.05, and batch size 128. For later rounds of imitation learning, we train for the same number of epochs while using the same batch size 128. In other words, we train for 1, 1,000 iterations for round 2 and 1,500 iterations for round 3.
J.3 Sampling and ablation study
For sampling without backward guidance and recurrence, we choose w 5.0. We compare the censoring performance of a reward model trained with imitation learning with reward models trained without the multi-stage imitation learning in the ablation study. We train the non-imitation learning reward model for the same number of cumulative iterations with the corresponding case of comparison; for example, when training with 30 malign and benign images from the baseline, we compare this with round 3 of imitation learning, we train for 3,000 iterations, which equals the total sum of 500, 1,000 and 1,500 training iterations used in rounds 2, and 3. For censored image generation via backward guidance and recurrence as discussed in Section G, we learning rate 0.002, B = and 4.
5.0,
J.4 Censored generation samples
Figure 14 shows uncensored, baseline generation. Figures 15 and 16 present images sampled with censored generation without and with backward guidance and recurrence.
27

---

Figure 14: Uncensored baseline image samples.
28

---

Figure 15: Non-curated censored generation samples without backward guidance and recurrence
after using 3 rounds of imitation learning each using 10 malign and 10 benign labeled images.
29

---

Figure 16: Non-curated censored generation samples with backward guidance and recurrence after
using 3 rounds of imitation learning each using 10 malign and 10 benign labeled images.
30

---

Images that are inconsistent with physical laws such as gravity or K LSUN bedroom: Experiment details and image samples
K.1
Pre-trained diffusion model
We use the pre-trained diffusion model" from [12]. trained on LSUN Bedroom dataset [47]. We follow the original settings, which include 1,000 DDPM steps, image size of 256 256, and linear noise scheduler.
K.2 Malign image definition
We classify LSUN bedroom image "broken" (malign) if it meets least of the following criteria:
(a) Obscured room layout: overall shape layout of the room clearly visible;
(b) Distorted bed shape: bed does not present well-defined rectangular shape;
(c) Presence of distorted faces: there are distorted faces of humans dogs;
(d) Distorted or crooked line: line of walls or ceilings are distorted or bent;
(c) Fragmented images: image divided or fragmented in manner that disrupts their logical
continuity or coherence;
(f) Unrecognizable objects: there are objects whose shapes are difficult to identify;
(g) Excessive brightness: image is too bright or dark, thereby obscuring the forms of objects.
Figure 17 shows examples of the above.
the other hand, we categorize images with the following qualities as benign, even they may give the impression of being corrupted damaged:
(a) Complex patterns: Images that include complex patterns beddings or wallpapers;
(b) Physical inconsistencies:
reflection;
(c) Distorted text: Images that contain distorted or unclear text.
Figure shows examples of the above.
K.3 Reward model training
We utilize a ResNet 8 architecture for the reward model, using the pre-trained weights available in torchvision.modele "DEFAULTS" setting", which is pre-trained on the ImageNet [10] dataset. We replace the final layer with randomly initialized fully connected layer with ane-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use BCE, in (4) as the training loss with 0.1. The models are trained for 5, 000 iterations using AdamW optimizer [27] with learning rate 3 x 10-4 weight decay 0.05, and batch size 128. We train five reward models for the ensemble.
http:///wwccaass.s.acaaalooolllt models. resnet18
31

---

Obscured layout
(h) Distorted bed shape
(c) Presence of distorted faces
(d) Distorted crooked line
Fragmented images
(l) Unrecognizable objects
Le
(g) Excessive brightness
Figure 17: Examples of "broken" LSUN bedroom images
(a) Complex patterns
(b) Physical inconsistencies
(c) Distorted text
Figure 18: Images classified benign despite giving the impression of being corrupted or damaged.
32

---

K.4 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose 2.0. We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 4:
"Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and set of 100 benign images.
"Union" model refers ~ 500 benign images, combining the set of benign images used to train the ensemble. These models are trained for 15,000 iterations with a 0.02 for the BC'Ea loss.
model which is trained on 100 malign images and a collection of
For these non-ensemble models, we use 10.0, which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence as discussed in Section G, we use 2.0, learning late 0.002, B 5, and R 4.
K.S Censored generation samples
Figure 19 shows uncensored, baseline generation. Figures 20-31 present total of 1,000 images sampled with censored generation, 500 generated by ensemble reward models without backward guidance and recurrence and 500 with backward guidance and recurrence.
33

---

Figure 19: 96 uncensored baseline image samples. Malign images are labeled with red borders and positioned at the beginning for visual clarity.
34

---

Figure 20: First set (1-96) of images among the 500 non-curated censored generation samples with reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
35

---

Figure 21: Second set (97-192) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
36

---

B
Figure 22: Third set (193-288) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
37

---

D
Fi
Figure 23: Fourth set (289-384) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
38

---

H
Figure 24: Fifth set (385-480) of images among the 500 non- non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
39

---

Figure 25: Sixth set (481-500) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
40

---

Figure 26: First set (1-96) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared the malign images of the uncensored generation.
41

---

Figure 27: Second set (97-192) of images among the 500 non-curated censored generation samples with reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjec- tively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images the uncensored generation.
42

---

Figure 28: Third set (193-288) of images among the 500 non-curated censored generation samples with reward model ensemble and with backward guidance and recurrence. Malign images labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjec- tively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images the uncensored generation.
43

---

Figure 29; Fourth set (289-384) images among the 500 non-curated censored generation sam- ples with reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
44

---

Figure 30: Fifth set (385-480) of images among the 500 non-curated censored generation samples with reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjec- tively speaking, observe that censoring makes the malign images less severely "broken" compared to the malign images the uncensored generation.
45

---

Figure 31: Sixth set (481-500) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjec- tively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
46

---

L Stable Diffusion: Experiment details and image samples
L.1 Pre-trained diffuuson model
We use the pretrained Stable Diffusion"., 13 version 1.4. We generate images using the default setting, which uses image size of 512 x 512 and 50 DDIM [41] steps.
I.2 Reward model training
We utilize a ResNet 8 architecture for the reward model, using the pre-trained weights available in torchvision.mi ' "DEFAULTS" setting' which is pre-trained the ImageNer [10] dataset. We replace the final layer with randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use BCE, in (4) as the training loss with 0.1. The models are trained for 10, 000 iterations using Adam W optimizer [27] with learning rate 10-4 weight decay 0.05, and batch size 64.
L.3 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, choose 4.0. We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 5:
"Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and set of 100 benign images.
"Union" model refers ~ 500 benign images, combining the set of benign images used to train the ensemble. These models are trained for 30,000 iterations with 0.02 for the BCF,, loss.
model which is trained on 100 malign images and collection of
For these non-ensemble models, we use 20.0, which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined recurrence as discussed in Section G, we use w = 4.0 and R = 4.
L.4 Censored generation samples
Figure 32 shows uncensored, baseline generation. Figures 33 and 34 respectively present images sampled with censored generation without and with recurrence.
httpp://oocccaossssstaaaalsllllt models.....o.....oooo.oomooo.... resnet18
47

---

HUI AI HUMMA AIR HAAA
A ACMAA WHOMTG H MALZAN
oy
HIMCIN
A S HU AN HUMMA HUN!
HHП HVIN
HUMN HUMAAN
DOU FOOOALE FIRLIE...
AMON HON HUIN HIPN
LIE A IIMA H
D0
A ME
HOMRO
IN
TUALAILAT
fima
Figure 32: Uncensored baseline image samples. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
48

---

GD
HUMA
THE TUEAIL
WO
Figure 33: Non-curated censored generation samples without backward guidance and recurrence. Reward model ensemble is trained on 100 malign images. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
49

---

Figure 34: Non-curated censored generation samples with recurrence. Reward model ensemble is
trained on 100 malign images. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
50

---

M Transfer learning ablation
To demonstrate the necessity of transfer learning for relatively more complex tasks, we compare with training reward model from scratch. We consider the LSUN bedroom task of Section 5.4. We randomly initialize the weights of imeedependent reward model with half- UNet architecture and train it for 40,000 iterations with batch size 128. We use the training loss BCEa with = 0.1 and use the guidance weight of 10.0 for sampling.
We observe that censoring fails without transfer learning, despite our best efforts to tune the parameters. The reward model successfully interpolates the training data, but its classification performance the test dataset (which we create separately using additional human feedback data) is poor: it shows 70.63% and 43.23% accuracy respectively for malign and benign test images. If nevertheless proceed to perform censored generation using this reward model (trained without transfer learning), the proportion of malign images is 15.68% + 5.25% (measured using 500 generated images across independent trials). This is no better than 12.6% of the baseline model without censoring.
51

---

Using malign images from secondary source
In this section, demonstrate the effectiveness of our framework even with malign images from secondary source, instead of model-generater images. This strategy may be useful, e.g., in cases where the baseline model rarely generates malign images (making difficult collect sufficient number of them) but user still wishes to impose hard restriction of not generating certain images.
We consider the setup of Section 5.2, and repeat the similar procedure except that images manually crawled from the Shutterstock webpage we utilize Microsoft Windows Snipping Tool take screenshots of "church", "gothic church" and "cathedral" images with clearly visible watermarks, resize them to 256 x 256 resolution and apply random horizontal flip. We use fixed set of these 30 secondary malign images (Figure 35) and 30 benign images generated by the baseline model to train each reward model. We use the same hyperparameters as in Section I for reward training and sampling, except the only difference of using the guidance weight w 1.0.
malign
sh shrcteee tiiterstco cock
stsck
Figure 35: Examples of malign (watermarked) church images, collected manually from a secondary source other than the baseline model.
Bascline (No censoring)
Ordinary censoring
Secondary censoring
Malign
11.4%
0.76%
1.4%
proportion
Table 1: Performance comparison between ordinary censoring vs. secondary censoring (using secondary malign images in place of model-generated malign images). Both censoring setups use ensemble of reward models and recurrence with R
We display the result of censoring using secondary malign images Table (labeled "secondary censoring"). We ensemble 5 reward models (trained on the same set of 30 secondary malign images) and apply recurrence with R = Recall that the baseline model generates 11.4% malign images, while with secondary censoring, the proportion drops to 1.4%. This is slightly worse than 0.76% of our ordinary censoring methodology (Figure 2), but still shows that secondary malign images could be effective alternatives to model-generated images in certain setups.
"tttps://ww..hutterrsttot
52


Reviews:
[{"review_id": "Suo7EI0kYM", "reviewer": "Reviewer_ADeQ", "summary": "This paper proposes an approach to censor the sample generation of pre-trained diffusion probabilistic models to better align with human preferences. The authors use minimal human feedback (<3min spent in providing the feedback for basic tasks, and <15min for more complicated tasks that they consider) to train a light-weight reward model, then use the trained reward model to provide either time-dependent or time-independent guidance for the generation process. The authors also use techniques such as ensemble, iterative imitation learning, transfer learning, backward guidance and recurrence to improve censoring performance. Experiments on MNIST, LSU church, ImageNet, and LSUN bedroom showed the effectiveness of their proposed method.", "questions": "- For the training data $X^{(1)},\\cdots,X^{(N)}$ in Algorithms 1 and 2, are they images corrupted by the VP SDE or images generated by the pre-trained model $\\varepsilon_\\theta$? What are the objectives in training each reward model in the ensemble method?\n- In Algorithm 1, for each $k$ the learner randomly select with replacement $N_M$ benign samples. Does this step also need human feedback?\n- Typo in line 204. \"30 malign and 150 malign samples\"", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper studies an interesting and important problem of aligning diffusion models with human preferences. By training a relatively lightweight reward model to guide the generation process instead of fine-tuning the large pre-trained model, their proposed approach could potentially save a lot of computation and human feedback.", "weaknesses": "- The paper's structure could be more coherent and consistent. The authors introduce classifier guidance in Section 1.1 but leave its relation to censored sampling unclear until Section 4. This disconnect could confuse readers. Similarly, there is a lack of clarity regarding the reward function $r$ and the reward model $r_\\phi$ introduced in Section 2. While initially, it seems that $r_\\phi \\approx r$, the possibility of $r_\\phi$ becoming time-dependent, as mentioned in Line 97 of Section 3, introduces confusion because $r$ is time-independent. This is not resolved until Section 4, when time-dependent guidance is addressed. To improve readability, it would be beneficial for the authors to introduce the underlying notations and mathematical concepts of the entire model—from the diffusion probabilistic models to the reward models and their role in censored sampling—before discussing training and evaluations.\n- In section 3.1, the authors choose $r_\\phi$ as the product of $K=5$ independently trained reward models. However, they did not have enough discussions or experiments to back up these choices, which makes the resulting model seem arbitrary. In particular, since the authors mentioned $r_\\phi(X)\\approx r(X)=P(Y=1|X)$, it's unclear why taking the product--as opposed to taking the average--makes sense because each $r_{\\psi_k}$ is also approximating $r$. Moreover, if we ignore the mathematical implications for now, the authors explained in line 115 that taking the product is essentially asking for unanimous approval, but readers might still question whether other methods, such as increasing the value of $w$ in the \"union\" method, could work just as well.\n- The paper highlights extreme human feedback efficiency by saying that a few minutes of human feedback are sufficient. However, this claim overlooks the complexity of tasks considered in this paper, which are relatively simple compared to tasks where RLHF is typically employed in large-scale models. Therefore, it would be better to directly compare with previous approaches (such as fine-tuning) under the same task. Additionally, the more complex tasks, like censoring distorted bedrooms, actually require more than the estimated 3 minutes of human feedback. Hence, the title could be seen as potentially misleading or exaggerating the method's efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive comments. We made our best efforts to address the concerns and reflect the comments to improve the paper. For the concerns regarding the complexity of the tasks we cover, comparison against other methods, and the longer human time for the LSUN Bedroom task, please refer to the pertaining discussions within our common response. We hope that we have properly addressed the reviewer's concerns, and if that is the case, we kindly ask the reviewer to consider increasing the rating.\n\n### 1. On the presentation of the paper\nWe highly appreciate these suggestions. We will clarify in Section 1.1 that classifier guidance is the basis of our censoring framework and is therefore relevant to the paper's contents. We will also clarify within Section~2 that the reward function $r(\\cdot, t)$ can be time-dependent and considering the time-dependent reward helps, in some setups, to properly deal with images with different level of noises that are encountered during the sampling process of diffusion models.\n\n### 2. On the choice of reward model\nIndeed, there are multiple ways to combine individual reward models $r_{\\psi_k}^{(k)}$ to build a refined approximation of the true reward $r$, and in our initial experiments (not included in the paper), we tested each of\n- Reward averaging: using $r_\\psi = \\frac{1}{K} \\sum_{k=1}^K r_{\\psi_k}^{(k)}$\n- Logit averaging: using $r_\\psi = \\sigma \\left( \\frac{1}{K} \\sum_{k=1}^K h_{\\psi_k}^{(k)} \\right)$, where $r_{\\psi_k}^{(k)}(X) = \\sigma \\left( h_{\\psi_k}^{(k)}(X) \\right)$ for each $k=1,\\dots,K$ and $\\sigma$ is the sigmoid function\n- Geometric reward averaging: using $r_\\psi = \\left( \\prod_{k=1}^K r_{\\psi_k}^{(k)} \\right)^{1/K}$.\n\nWhile each method seems to make sense in principle, it turned out that methods 1, 2 were not as empirically effective as method 3 in censoring out the unwanted samples.\nNote that taking the product of each $r_{\\psi_k}^{(k)}$ is equivalent to using method 3 and then increasing the guidance weight $\\omega$ by $K$ times.\nWe would like to clarify that **we already did make a fair comparison, in this context, between the \"ensemble\" models versus the \"union\" models** in all of our ablation studies in Sections H.3, I.4 and K.4, where we used $K$ times larger values of $\\omega$ for the \"union\" cases compared to the \"ensemble\" cases, which are the product models. Our experiments conclude that ensemble models are still always better.\n\nOne must use the guidance weight $\\omega = 1.0$ to strictly obey the mathematical formalism $p_{\\mathrm{censor}}(x) = p_{\\mathrm{data}}(x) r(x)$, but we believe it is an everyday practice in the diffusion model literature to use larger $\\omega$ for improved consistency with the conditioning label in guided/conditional generation.\nIn the end, we observed that using the product of $K=5$ reward models, which is equivalent to using $\\omega = 5.0$ with respect to the geometric average, already works fairly well in most cases even if it is not a carefully tuned choice. (We do expect further performance gain by ensembling a larger number of reward models, but we do not pursue this minor improvement direction in depth.)\nFor the interest of space, we omitted these details and only mentioned that we use the product as our reward.\nHowever, we would be happy to further justify this choice by including the above discussion, if we are later provided with additional space.\n\n### 3. Question regarding the notation $X^{(1)}, \\dots, X^{(N)}$ and reward model training\n$X^{(1)}, \\dots, X^{(N)}$ are the images generated by the pre-trained model, and in Algorithm 1, we assume that the human feedback (labels for malign/benign) is already given on them.\nWhen using the time-dependent reward model architecture, we also sample $t$ uniformly within $[0, T]$ and noise $X^{(i)}$ up to time $t$ along the VP SDE.\nWe use the weighted BCE loss function as described in our Appendix, Section F: $BCE_\\alpha(r_\\psi(x;t), y) = -\\alpha \\cdot y \\log r_\\psi(x;t) - (1-y) \\log (1-r_\\psi(x;t))$ where $\\alpha < 1$ is a hyperparameter, determining to which extent the reward model prioritizes classifying malign images as malign over benign images as benign.\n\n### 4. Question on the subsampling of benign samples\nIn the setup of Algorithm 1, we assume that benign images are more frequently generated compared to malign images.\nThe human feedback collecting-stage terminates once the number of malign samples reaches the desired level ($N_M$), and at this moment one will already have a pool of abundant ($N_B > N_M$) benign samples (because the feedback provider is automatically marking images as benign if they are not malign).\nThus, during the execution of Algorithm 1, no additional human feedback is required."}, {"review_id": "L8iwy385hj", "reviewer": "Reviewer_W62Q", "summary": "The authors combine pre-trained diffusions with classifiers trained on human feedback about which type of images to omit, and use the classifiers to guide diffusion sampling using the Universal Guidance technique. They observe that they are able to filter out several types of malign images on a variety of datasets, notably removing undesirable artifacts on LSUN and faces on Imagenet.\n\nI would recommend this paper for acceptance.", "questions": "For the first concern, it would be great if you could include something showing the number of model-produced malign samples going down as a function of number of malign samples used in classifier training, or as a function of number of total samples used for classifier training, or both. I know this is hard to do since the outputs need to be checked manually. Even doing this with a batch of 128 for a few classifier-training-set-sizes would be good, and see e.g. the malign proportion lowers for several feedback dataset sizes.\n\nFor the second concern, the crossed 7's are distinct enough in MNIST such that a very small classifier could quickly assign zero weight to them. However, the faces on Imagenet are more convincing since the properties to be censored (faces) do appear in several ways that aren't as clearly disentangled from benign properties. It would be useful to share somewhere in the text a case where the censoring did not work despite what looks like an adequate amount of samples. Did this happen for censoring some other of the Imagenet classes? Are there any cases you explore where the choice of malign samples constitute nearly all of the training set?\n\nFeedback on a second dataset rather than model samples: for the case where a diffusion is trained on a dataset A with low frequency of the malign property (but nevertheless we need to guarantee that samples do not ever contain the property), the model samples will feature malign content infrequently. What if you have a second dataset B where the property is common? If the datasets are the same resolution and are diverse enough, and share some concepts this might work and might be a nice way to get past low malign frequency in dataset A? Consider the case where A only has some faces but B is at least half human faces. So samples from the A-trained-model do not produce malign content often and good feedback is hard to collect. How well does a feedback model trained on a second dataset B (which are not model samples) work for censoring the A-trained-diffusion from generating faces?\n", "limitations": "Addressed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- People care about censoring sampled outputs, and don't like re-training large diffusions\n- Together with the appendix, the specifics of how exactly to guide are well explored in this work\n- I like that that while \"bad\" images are hard to quantify with some kind of numerical per-sample score, it is easy for a human to recognize instances of it when they do see it.\n- emphasizing that not much feedback is needed for the cases explored", "weaknesses": "The main concerns would naturally be:\n- how much feedback is enough\n- are the good results specific to the type of things being censored in the particular datasets chosen\n- for the infrequent malign content case, can a second dataset be used in place of model samples?\n\nSee questions below.\n\nOther small comment: change \"man hours\" to \"human work hours\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are delighted to see that the reviewer empathizes with our problem statement and solutions. We also greatly appreciate the inspiring comments with the positive evaluation. We have devoted our best efforts to provide satisfactory resposnes to each of the reviewer's concerns below.\n\n### 0. Regarding the expression \"man hours\"\nWe have replaced the instances of the expression \"man\"' to \"human\".\nWe thank the reviewer for pointing this out.\n\n### 1. Regarding the effect of feedback data size on malign proportion \nReflecting the comment, we tested the effect of feedback data size used for reward training on the censoring performance in the setup of Section 5.3.\nPlease refer to Figure 3b within the attached document in the common response.\nTo clarify, we already had comparisons of the cases using 10, 20 and 30 malign images within the ablation studies, but newly added the cases of 50 and 100 malign images for clearer demonstration, and we do obtain a figure of the desired shape.\nWe did not have enough capabilities to repeat the analogous procedure for the more complex setups of Section 5.4 and the additional Stable Diffusion experiment within the rebuttal period, but we plan to include them in the revised version of the paper.\nWe thank the reviewer for the suggestion.\n\n### 2. Regarding whether the success of proposed methods is general\nWe would like to clarify that **the tasks we consider in the paper are not the ones particularly curated for success.**\nThat is, we did not extensively explore setups other than what show in the paper; rather, we fixed the tasks from the beginning and developed the general strategies that could consistently deliver the results.\nTherefore, we believe that **our approach will be applicable over wide range of datasets and tasks.**\nIt is true, though, that there seem to be a minimum number of samples required for positive results; e.g., using 50 malign and 50 benign samples for the LSUN Bedroom task was not satisfactory enough.\n\nWe speculate that if the proportion of malign images from the baseline model is extreme (say, 99%) then it will be very difficult to successfully censor them out, but we have not tried pushing through experiments on such setups.\nOur intuition on the censored generation is: it works by encouraging a model's benign behavior while suppressing the malign side, which is possible only when the model already possesses sufficient capability of producing benign images.\nTherefore in designing experiments, we only considered the tasks where the baseline model generates a reasonable proportion of benign images, and for all of these cases we managed to achieve good censoring precision.\nIn the later version of the paper, we will try to include more discussion on this point.\n\n### 3. On using a secondary dataset\nWe believe **using a secondary dataset is indeed a feasible strategy, and we provide a proof-of-concept experiment** using the LSUN Church setup where we censor the \"Shutterstock\" watermarks.\nInstead of marking samples generated by the model as malign, we visit Shutterstock website and manually collect 30 images (the number equal to the experiments from Section 5.2) within the search results for \"church\", \"gothic church\" and \"cathedral\" with clearly visible watermarks, using the Snipping Tool (please see Figure 2a of the attached document).\nWe fix this secondary set of malign images $\\mathcal{M}$ and train 5 reward models each using $\\mathcal{M}$ together with 30 fresh random benign samples generated from the model.\nWe use the same configurations as described in Section 5.2 and the Appendix I of the main paper for reward training.\nWe observe that although slightly worse compared to the best ordinary censoring results from the paper **(0.76%)**, censoring via ensemble of the reward models trained using the secondary source is effective (achieving **1.4%** malign proportion) when used with $R=4$ recurrence steps (Table 2 in the attached pdf).\n\nOn the other hand, it seems that this alternative approach should be used with more care.\nOur censored generation based on the secondary data does produce legitimate images with lowered proportion of malign images in the best case (pdf Figure 2b), but we heuristically observe frequent degradation in the generated images' quality without universal guidance (recurrence).\nAdditionally, we observe that the guidance procedure becomes relatively less resilient to larger guidance weights when using the secondary data; using $\\omega=2.0$ as in the paper seemed to lower the image quality, so we use $\\omega=1.0$ for the results of the pdf's Table 2 and Figure 2b.\nThis is possibly because it becomes much easier for the reward model to discriminate between the malign and benign images when malign images originate from a dataset of distinct distribution, so that it is more likely that the reward overfits and quickly learns to classify with extremely high confidence, causing the gradient to overshoot.\nFinally, if the secondary dataset is not sufficiently close to the distribution the diffusion model has learned, then censoring may be unsuccessful.\nWhen we try the same experiment using slightly different search keywords such as \"old city\", it does not work.\nThis means that it is difficult to make a precise prediction on whether the strategy will succeed in, e.g., the virtual scenario suggested by the reviewer; it is genuinely case-by-case so the best practice seems to be trying out with hope."}, {"review_id": "kxR0EqlYpc", "reviewer": "Reviewer_zmCo", "summary": "This paper studies the problem of preventing the generation of unwanted images in diffusion models. It formulates the task of 'censoring' and proposes using reward model trained from human labelling to guide the diffusion model. The method requires no fine-tuning and a few minutes of human feedback, while displaying significant reduction of unwanted images in four experiment setups.", "questions": "- Why is it better to use time-independent reward model for censoring watermarks and time-dependent reward model for other tasks?  Is there a general reason why time-independent reward models work better for some tasks and not others?\n\n- Any guidance of choosing $\\omega$ ?\n\n- It appears from the paper that the training of reward models need large datasets containing images related to a single unwanted concept (for example fish with human faces). If I do not have such high-quality, single-purpose datasets, maybe only random images with all kinds of undesirable concepts, can the method still work well?", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This work identifies and formulates an important problem of misalignment in diffusion probabilistic models, which underwent few studies in the past. Solution to the problem can significant improve the utility of generative models and lead to positive social impact.\n\n- RLHF is an emerging technology which is theoretically sound and empirically useful. The proposed method successfully bring the idea of RLHF into diffusion models while combining the idea of classifier guidance. The method itself is clean and simple to train, and requires few modifications to the sampling process. \n\n- From the experiments, the method seem also quite efficient: (i) it sample efficient: requiring up to 100 malign samples, (ii)  it does not need fine-tuning; (iii) it works under limited human feedback, taking only a few minutes of human labeling. These properties are desirable in practical implementation.", "weaknesses": "- Under the general framework of training reward models and sampling with guidance, the authors use four different methods to train the reward model for the four different tasks in the paper. The methods are quite heuristic. It appears that for any new concept and dataset, it still requires a manual selection of the training algorithms for the reward models to achieve the best effect, which can be costly. However, the paper does not provide a conclusive algorithm or any unifying principles for the selection. In this sense the study seems unfinished.\n\n- There are no large-scale, principled comparisons with baselines. Even if there are limited existing works studying the problem, I think there are still some naive methods such as using natural language prompts, fine-tuning, and post-selection with classifier. It is not straightforward to see why the method proposed in this paper is the best.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the constructive comments and the positive evaluation from the reviewer.\nWe made our best efforts to reflect the comments to improve the paper.\nIn the following, we address each of the reviewer's comments in detail.\n\n### 1. Regarding the principles for selecting the techniques\nWe believe that our strategy on when to apply which techniques can be summarized into the following **two simple \"rules\"**:\n- **Ensemble vs. Imitation learning.** We classify the censoring tasks into benign-dominant and malign-dominant cases, and apply Algorithm 1 (reward ensembling) to the former and Algorithm 2 (imitation learning) to the latter. Please also refer to our discussion on this point in lines 132--138 on page 6 of the paper.\n- **Time-dependent vs. time-independent reward models.** Guidance using time-dependent reward $r(\\cdot, t)$ is conceptually simpler (same as the usual classifier guidance) but requires a time-dependent architecture and the reward model should be trained on images with different level of signal-to-noise ratios. For these reasons, one usually has to train the model from scratch, which is more suitable for simpler/smaller tasks (Sections 5.1 and 5.3). On the other hand, time-independent reward models require additional techniques such as universal guidance, but the reward model has to be trained only on clean images, so one can exploit various pre-trained classifiers and expedite the reward training through transfer learning. This makes (time-independent) transfer learning more suitable for complex/large-scale tasks (Sections~5.2, 5.4 and newly added Stable Diffusion experiment). We found in our preliminary experiments that for these tasks, it was not easy to train time-dependent reward models for successful censoring, while transfer learning converged faster and provided promising results.\n\nBesides these rules, we also utilize the universal guidance components to boost the censoring performance, but they apply globally to all tasks that we consider.\nAdditionally, we observe that our framework requires minimal hyperparameter tuning; the typical values $\\alpha=0.1$ (parameter used in the reward-training loss $BCE_\\alpha$), reward learning rate $3\\times 10^{-4}$, batch size 128, and number of recurrence $R=4$ work sufficiently well for most setups.\nFor the choice of guidance weight $\\omega$, we provide a separate discussion below.\nWe hope that our clarification resolves the reviewer's impression that there is no principled strategy on the choice of methods.\n\n### 2. Regarding comparison with other methods\nPlease refer to our common response (Section 4 therein) on this point.\n\nWe provide comparisons with rejection sampling (post-selection in the reviewer's terminology) using the reward model in Sections~5.3 and 5.4, where it is shown that rejection sampling is not only worse than the guidance method in terms of the precision, but also, it rejects a large proportion of generated samples, which is undesirable.\n\nWe use non-text conditional models in the paper's experiments so direct comparison to prompting was not available, but in our additional experiment using the Stable Diffusion model, some basic attempts like adding \"without text\" or \"no text\" to the prompts did not produce successful results.\nWe acknowledge, however, that sophisticated prompt engineering, negative prompting, or adding specific requests regarding the background could also prevent the appearance of texts.\nAt this stage, we view our experiment on Stable Diffusion as a proof of concept that our simple methodology does work on large-scale text-to-image diffusion models (rather than claiming superiority over all possible alternatives).\nCarefully identifying the problems where even dedicated trials using natural language prompts fail, and solving them through human feedback, seems to be an interesting future research direction.\n\n### 3. Regarding the choice of $\\omega$\nFor ensemble of $K=5$ reward models, **$\\omega=1.0$ for the simplest toy case.**\nBased on our experiments, we speculate that **the appropriate scale is doubled (roughly) with 1) significant scale growth in terms of data size and 2) the introduction of new modality (e.g. unconditional or class-conditional $\\to$ text-conditional model).**\nWe use $\\omega=2.0$ for Sections 5.2 and 5.4, where the data size grows to 256$\\times$256 scale.\nFor the additional Stable Diffusion experiment, we again double it and use $\\omega=4.0$.\n\nNote that for the experiments of Section 5.3 (ImageNet Tench) or ablation studies with \"Single\" and \"Union\" models where we do not use ensemble, it is conceptually \"fair\" to use $K$ times larger value of $\\omega$ in the following sense.\nIf $r_\\psi = \\prod_{k=1}^K r_{\\psi_k}^{(k)}$, we have $\\nabla \\log r_\\psi = \\sum_{k=1}^K \\nabla \\log r_{\\psi_k}^{(k)}$.\nSo viewing each $r_{\\psi_k}$ as an approximation to $r$ (the \"true\" reward), $\\nabla \\log r_\\psi \\approx K \\nabla \\log r$ is used in the guidance process.\nThus, to produce a similar effect, one should multiply $K$ when using log gradient from a single reward model.\nIn all ablation studies, we stick to this rule.\nSimilarly, in Section 5.3, we use $\\omega=5.0$ which is $K$ times the value of $\\omega$ used in Section 5.1.\n\n### 4. Regarding censoring multiple concepts\nWe thank the reviewer for mentioning this is interesting point.\nWe believe that multiple concepts can be censored simultaneously by combining reward models corresponding to each one.\nSuppose we wish to censor two concepts, and for each of them we train a reward model $\\hat{r}_1$ and $\\hat{r}_2$ where $\\hat{r}_i \\approx 0$ indicates that the image contains (is malign with respect to) the $i$th concept.\nSimilar to using the product in reward ensemble, we have $\\hat{r}_1 \\hat{r}_2 \\approx 0$ if at least one of $\\hat{r}_1, \\hat{r}_2$ is small, so censored sampling guided by $\\nabla \\log \\hat{r}_1 \\hat{r}_2 = \\nabla \\log \\hat{r}_1 + \\nabla \\log \\hat{r}_2$ will enforce the generated images to be free of both concepts."}, {"review_id": "lP7GIFEF0a", "reviewer": "Reviewer_iXGj", "summary": "This paper presents censored diffusion model training by using a reward model trained using human feedback. Towards this, the paper utilizes reward model ensembles (for benign dominant settings) and tools from imitation learning (for malign dominant settings).", "questions": "a. i wonder if the malign dominant settings can be handled by flipping the reward to be something resembling 1-\\prod_{i=1}^K (1-r_{\\psi_k}^{k})?\n\nb. does it help to train every single reward model with some form of hard negative mining? I mean this in the context of algorithm 1.", "limitations": "I believe the authors must present a discussion surrounding this in the context of their work - what it seeks to address, what remains to be addressed, and how much to trust the proposed algorithm as to whether it truly achieves what it sets out to. I am not certain about going through an ethics review, and I hope the AC and the authors try to see what is necessary in this direction as it clearly seems like there is potential for fleshing some of these concerns out.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The human feedback part is pretty interesting and is a topic of increasing interest. The empirical results appear to be pretty compelling, though, I am not an expert with respect to evaluating this in particular, so I cannot comment on this with high confidence.\n\n==> post rebuttal: I increased the score from a 5 to a 6.", "weaknesses": "The techniques utilized by the paper are pretty well known and in that sense, this paper's contributions appear incremental. \n\nOne other question that appears to not be addressed is - by performing this paper's procedure iteratively (as multiple rounds), would the probability of producing malign images be brought down to zero? What are the other ways of achieving this?\n\nAnother weakness of the paper is that it is not clear to me if the proposed approach has been compared against other concept removal methods in the literature (which I agree I am not an expert on), but, this is something the authors must either present clarifications on, or, provide more comparisons on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive comments and the positive evaluation from the reviewer.\nIn the following, we make our best efforts to address each of the concerns and questions.\nFor the concern regarding the comparison against other methods, please refer to Section 4 within our common response.\n\n### 1. Regarding the contribution of the paper\nIn terms of the technical novelty, we acknowledge that the building blocks of our method are not new, but we ask the reviewer to consider that the main contribution of our paper is to **provide a simple, general, and easily adaptable framework that exhibits extreme sample/feedback efficiency.** A priori, it was not at all obvious that this level of sample efficiency is possible. Considering that both diffusion models and RLHF are currently actively investigated topics within the literature, we do believe that these findings will be useful to practitioners.\n\n### 2. On how far the iterative application of our method could reach\nWe expect that iterative application of our methods (Algorithm 2 in particular) with progressively more feedback data **will achieve 0\\% malign images in the limit.**\nAlgorithm 2 iterates the process of \"collecting additional human feedback data $\\rightarrow$ reward training $\\rightarrow$ (censored) sampling using the reward\", and the 3 rounds, without much feedback data, can already reduce the initial malign proportion dramatically (from 68.6% to 2.2% and 1.0%, without and with universal guidance components, in Section 5.3).\nDuring the rebuttal period, we tried **another round (Round 4) of imitation learning and observed another significant reduction of malign proportion to 1.38% and 0.58%, without and with universal guidance.**\nFitting the tendency, it seems that further rounds will steadily reduce the probability of malign generation and eventually converge to zero (Figure 3a in the attached pdf), but this will come at the cost of requiring more compute for data generation and more human feedback due to the high precision of censoring.\nNevertheless, we do see hope in achieving an extreme level of precision at the expense of additional human feedback and computation.\n\n### 3. On the question of handling malign-dominant tasks\nThe reviewer's question is whether the reward ensembling (Algorithm~1) can be used for malign-dominant tasks by switching the roles of malign and benign images (that is, fixing a set $\\mathcal{B}$ of benign images and then ensembling $K$ reward models trained on $\\mathcal{B}$ together with differently subsampled malign images).\nIn principle, one can train reward models $r_{\\psi_k}^{(k)}$ in this way and use the ensembled product $r_\\psi = \\prod_{k=1}^K r_{\\psi_k}^{(k)}$ as usual; there is no need to flip the reward because we still expect the product to encourage unanimous approval.\nHowever, we observe that this is not as effective as imitation learning when the same number of total human feedback data are used; Round 3 of imitation learning that uses total 60 human feedbacks achieves **2.2%** censoring precision without universal guidance, while ensembling 5 reward models trained using 10 fixed benign samples and different sets of 10 malign samples (which also uses 60 feedbacks in total) achieves **6.6%** (please also refer to the discussion related to this point in lines 135--138 on page 6 of the paper).\n\n### 4. Regarding the use of hard negative mining\nAs far as we understand, hard negative mining in this context would mean training the reward model using their false positives, which are the malign generated images even with the censoring technique applied using that reward model. We believe that this process is essentially our Algorithm 2 (imitation learning), and we have already **verified through Experiments in Section 5.3 on Imagenet Tench images that imitation learning is indeed effective, providing improved censoring precision compared to non-imitation learning using the same number of total samples (not using hard negative mining).** Therefore, we expect refining each individual reward model using Algorithm 2 and then applying Algorithm 1 (ensembling) will certainly result in better censoring performance. We, however, do not pursue this direction in depth to focus on delivering of the core ideas.\n"}, {"review_id": "8pQCUq2J3T", "reviewer": "Reviewer_GFro", "summary": "The authors examine the problem of preventing the generation of certain types of images generated by a diffusion model. To achieve this, the authors propose using a reward model trained on human feedback. The authors demonstrate their approach from examples that require minimal human feedback to achieve sufficient censoring performance.", "questions": "- Line 52: there are two consecutive \"the\".\n- Figure 3(b) text: \"Comparsion\" to \"Comparison\"\n- How far can this method go? I'm sure there are practioners that want 0% malign images.", "limitations": "- The authors do not explicitly state their limitations.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors make a good argument in that 3 minutes of human feedback can be more cost-efficient than retraining models.\n- Their discussion and experiments in benign-dominant and malign-dominant scenarios are thorough. \n- Their ablation studies in the experiments give convincing arguments that human feedback works well.", "weaknesses": "- A discussion of when to use time-dependent or time-independent would have been nice.\n- In the LSUN bedroom experiment, it took 15 minutes so it doesn't always take 3 minutes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the constructive comments and the positive evaluation from the reviewer. We made our best efforts to reflect the comments to improve the paper. In the following, we address each of the reviewer's concerns in detail.\n\n### 1. On time-dependent vs. time-independent guidance\nGuidance using time-dependent reward $r(\\cdot, t)$ is conceptually simpler (same as the usual classifier guidance) but requires a time-dependent architecture and the reward model should be trained on images with different level of signal-to-noise ratios.\nFor these reasons, one usually has to train the model from scratch, which is more suitable for simpler/smaller tasks (Sections 5.1 and 5.3).\nOn the other hand, time-independent reward models require additional techniques such as universal guidance, but the reward model has to be trained only on clean images, so one can exploit various pre-trained classifiers and expedite the reward training through transfer learning.\nThis makes (time-independent) transfer learning more suitable for complex/large-scale tasks (Sections 5.2, 5.4 and newly added Stable Diffusion experiment).\nWe found in our preliminary experiments that for these tasks, it was not easy to train time-dependent reward models for successful censoring, while transfer learning converged faster and provided promising results.\nWe will add discussion on this point in a later version of the paper.\n\n### 2. Regarding the human time for the LSUN bedroom task\nPlease refer to the discussion on this point in Section 3 of the common response.\nWe further emphasize that our techniques are applicable to solving a task on the **Stable Diffusion** model, (also shown in the common response), and this actually **requires only 3~4 minutes of human feedback, despite the task's complexity.**\n\n### 3. On how far the iterative application of our method could reach\nWe expect that iterative application of our methods (Algorithm 2 in particular) with progressively more feedback data **will achieve 0% malign images in the limit.**\nAlgorithm 2 iterates the process of \"collecting additional human feedback data $\\rightarrow$ reward training $\\rightarrow$ (censored) sampling using the reward\", and the 3 rounds, without much feedback data, can already reduce the initial malign proportion dramatically (from 68.6% to 2.2% and 1.0%, without and with universal guidance components, in Section 5.3).\nDuring the rebuttal period, we tried **another round (Round 4) of imitation learning and observed another significant reduction of malign proportion to 1.38% and 0.58%, without and with universal guidance.**\nFitting the tendency, it seems that further rounds will steadily reduce the probability of malign generation and eventually converge to zero (Figure 3a in the attached pdf), but this will come at the cost of requiring more compute for data generation and more human feedback due to the high precision of censoring.\nNevertheless, we do see hope in achieving an extreme level of precision at the expense of additional human feedback and computation."}, {"review_id": "cvfGXLxgpZ", "reviewer": "Reviewer_6Ndg", "summary": "The work aims to solve censored diffusion sampling problem, which prevent diffusion model generating malign / bad images. The core approach is to train a classifier and apply classifier-guided diffusion generation. ", "questions": "It would be more convincing that the authors provide more experiments for more complex diffusion models, such as stable diffusion.", "limitations": "See above", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Authors present an interesting finding that the classifier-guided diffusion can effectively reduce the probability of generating malign images.\n2. Authors present two approaches to learn the classifier, ensemble-based approaches when number of malign images less than benign images, imitation learning for when malign images are dominates.\n3. The proposed methods are lightweight and can be effective with small labeled datasets.", "weaknesses": "The core technique employed in the work is not new.  And compared with existing works that are based on text2image diffusion models, authors use relatively simple diffusion models that are either class-condition diffusion models or diffusion models for one category dataset. I am not sure the effectiveness and efficiency of the proposed method can be scaled to a large diffusion model that handles for complicate diffusion models. The classifiers in three experiments are relatively easy and that may explain why it is so effective. It may not be true in large text2image settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive comments. As suggested by the reviewer, we added an experiment using the **Stable Diffusion** model. Please refer to Section 2 of the common rebuttal and the attached pdf document for details.\nWe plan to add this new experiment, with some polishing, to the later version of the paper.\nFor now, we hope that the reviewer's concern on the applicability of our framework to larger/complex models is addressed, and if that is the case, we kindly ask the reviewer to consider increasing the rating.\n\nIn terms of the technical novelty, we acknowledge that the building blocks of our method are not new, but we ask the reviewer to consider that the main contribution of our paper is to **provide a simple, general, and easily adaptable framework that exhibits extreme sample/feedback efficiency.**\nA priori, it was not at all obvious that this level of sample efficiency is possible.\nConsidering that both diffusion models and RLHF are currently actively investigated topics within the literature, we do believe that these findings will be useful to practitioners."}]

Author rebuttal:
# Common Response (pdf attached)

We thank all reviewers for the extremely detailed and constructive feedbacks. We are delighted that most reviewers found our ideas convincing and the contributions solid. Below, we provide our response to some common concerns.
### 1. List of additional experiments
In the attached, we show 4 additional experiments.
For better readability, we provide the list of the experiments associated to each figure/table.
- Figure 1: A new task using **Stable Diffusion**. We successfully eliminate embedded raw texts that are unexpectedly associated to the prompt "a photo of a human" (Table 1). This experiment is to address the concerns (reviewers ADeQ, 6Ndg) that our framework may not apply/not be as efficient as we claim when diffusion models are of larger scale & complexity. Please refer to Section 2 below for details.
- Table 2: Censoring result using **malign images from a secondary source** (instead of model-generated samples) in the LSUN Church task. This demonstrates that external images may substitute model-generated malign images if they are scarce. We thank reviewer W62Q for motivating this experiment.
- Figure 3a: **Round 4 of imitation learning** in the ImageNet task provides further improvement over Round 3. The continued application of imitation learning reduces the malign proportion approximately exponentially ($y$-axis is in log scale) and could lead to extreme precision as reviewers inquired (GFro, iXGj).
- Figure 3b: Plot highlighting the effect of feedback size for reward training (ImageNet task), as suggested by reviewer W62Q.

### 2. Regarding application to complex tasks/large-scale models
To resolve the incorrect impression that our methods are efficient only for simple settings, we introduce a **Stable Diffusion experiment that uses minimal human efforts.**

In Stable Diffusion v1.4, **21.9%** out of 1,000 generated images (512$\times$512) by the prompt "a photo of a human" contain prominent embedded texts, or even lacks any genuine image and displays only the raw text (Figure 1a).
(We could not find a straightforward modification to the prompt that removes texts without compromising the generality of the prompt; basic attempts like adding "without text" or "no text" are unsuccessful.)
We collect human feedback data until 100 malign images are found **(3~4 mins)**.
We train 5 reward models via transfer learning (batch size 64, no augmentation, $BCE_\alpha$ weight $\alpha=0.1$, lr $10^{-4}$ and 10,000 iterations).
We use $\omega=4.0$ and no backward guidance.
Without/with recurrence ($R=4$), the proportion of malign samples drops to **4.8%**/**1.3%** (out of 1,000 samples).
Figure 1b shows generation results with all censoring techniques applied.
We hope this resolves some reviewers' concern regarding applicability on large-scale text-to-image models.

### 3. Regarding the human time for LSUN bedroom task
We do acknowledge natural criticisms from some reviewers that this task actually requires $>$3 minutes. However, we point out that
- This is a large-scale task: it uses the full 256$\times$256 diffusion model (not the lighter LDM).
- In fact, we design this setup as a **worst case** on the human work our framework requires.
Most censoring tasks in practice are likely to be less delicate than this task, where we deal with an ambiguous concept of malignity that is difficult to characterize and inevitably requires multiple guidelines to minimize subjectivity of the experiments.
- In the end, even the amount of feedback ($<$1k) and human time (15 mins) for this worst case is significantly smaller compared to those in prior works [1, 2, 3] that use 27~137k human feedback to fine-tune diffusion models.
- Nevertheless, we'll update the title to "Censored Sampling ... Using **a Few Minutes** of Human Feedback".

### 4. Comparison to other censoring methods
To clarify, our paper aims to resolve a defect/inconsistency of a model whose occurrence is **easy for humans to identify, but difficult to be mathematically described or fixed**.
There are many issues of this kind, e.g.: NSFW contents, inconsistencies in limbs/fingers, or double-headed humans, persistently appearing even after negative prompting.
Our premise is that **there are cases in which feedback from humans are the only simple or feasible way of encoding the model's defect, and for these types of tasks, any other methodologies such as fine-tuning should share the process of collecting human feedback data and learning from these data**.

Traning a reward model fitting the human feedback data and then fine-tuning as in [1, 2, 3] involves more complications compared to our approach, which is arguably one of the simplest things to do with a reward model.
Fine-tuning usually requires multiple trial-and-errors until the best hyperparameters are found, as the diffusion model often overfits to the fine-tuning objective and experiences catastrophic forgetting.
(This entire process is not necessary if we simply use guidance.)
Another possible approach is to apply the policy gradient type fine-tuning as in [4] without explicitly training the reward model but letting humans play the role of the environment (reward oracle).
However, we see less hope in this direction because policy gradient is notoriously sample inefficient [5], unless technical endeavor comparable to the one devoted to this whole paper is made.
**Therefore, we argue that other approaches, including those based on fine-tuning, may not be as straightforward or suitably sample-efficient as they might seem at first glance.**

[1] Lee et al., Aligning Text-to-Image Models using Human Feedback. 2023.

[2] Xu et al., ImageReward: Learning and Evaluating Human Preferences ... Generation. 2023.

[3] Fan et al., DPOK: Reinforcement Learning for Fine-tuning ... Diffusion Models. 2023.

[4] Fan & Lee, Optimizing DDPM Sampling with Shortcut Fine-Tuning. ICML, 2023.

[5] Gu et al., Q-Prop: Sample-Efficient Policy Gradient ... Critic. ICLR, 2017.

Meta-review: All the reviewers agree with the novelty and contribution of the paper. Thus I recommend acceptance.
Decision: Accept (poster)