Title: Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative models

Diffusion probabilistic models

Controlled generation

Human Feedback

RLHF
Abstract: Diffusion models have recently shown remarkable success in high-quality image generation. Sometimes, however, a pre-trained diffusion model exhibits partial misalignment in the sense that the model can generate good images, but it sometimes outputs undesirable images. If so, we simply need to prevent the generation of the bad images, and we call this task censoring. In this work, we present censored generation with a pre-trained diffusion model using a reward model trained on minimal human feedback. We show that censoring can be accomplished with extreme human feedback efficiency and that labels generated with a mere few minutes of human feedback are sufficient.

Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback

Abstract

Diffusion models have recently shown remarkable success in high-quality image generation. Sometimes, however, a pre-trained diffusion model exhibits partial misalignment in the sense that the model can generate good images, but it sometimes outputs undesirable images. If so, we simply need to prevent the generation of the bad images, and we call this task censoring. In this work, we present censored generation with a pre-trained diffusion model using a reward model trained on minimal human feedback. We show that censoring can be accomplished with extreme human feedback efficiency and that labels generated with a mere few minutes of human feedback are sufficient. Code available at: https://github.com/tetrzim/diffusion-human-feedback .

1 Introduction

Diffusion probabilistic models [19, 12, 42] have recently shown remarkable success in high-quality image generation. Much of the progress is driven by scale [35, 36, 38], and this progression points to a future of spending high costs to train a small number of large-scale foundation models [4] and deploying them, sometimes with fine-tuning, in various applications. In particular use cases, however, such pre-trained diffusion models may be misaligned with goals specified before or after the training process. An example of the former is text-guided diffusion models occasionally generating content with nudity despite the text prompt containing no such request. An example scenario of the latter is deciding that generated images should not contain a certain type of concepts (for example, human faces) even though the model was pre-trained on images with such concepts.

Fixing misalignment directly through training may require an impractical cost of compute and data. To train a large diffusion model again from scratch requires compute costs of up to hundreds of thousands of USD [30, 29]. To fine-tune a large diffusion model requires data size ranging from 1,000 [28] to 27,000 [25]. 1 We argue that such costly measures are unnecessary when the pre-trained model is already capable of sometimes generating 'good' images. If so, we simply need to prevent the generation of 'bad' images, and we call this task censoring . (Notably, censoring does not aim to improve the 'good' images.) Motivated by the success of reinforcement learning with human feedback (RLHF) in language domains [9, 50, 43, 33], we perform censoring using human feedback.

In this work, we present censored generation with a pre-trained diffusion model using a reward model trained on extremely limited human feedback. Instead of fine-tuning the pre-trained diffusion model, we train a reward model on labels generated with a few minutes of human feedback and perform guided generation. By not fine-tuning the diffusion model (score network), we reduce both compute

7

liiterstock

HMAI

100

HMMN

AHU MATI

AAOMINA

HA

and data requirements for censored generation to negligible levels. (Negligible compared to any amount of compute and man-hours an ML scientist would realistically spend building a system with a diffusion model.) We conduct experiments within multiple setups demonstrating how minimal human feedback enables removal of target concepts. The specific censoring targets we consider are: A handwriting variation ('crossed 7's) in MNIST [11]; Watermarks in the LSUN [47] church images; Human faces in the ImageNet [10] class 'tench'; 'Broken' images in the generation of LSUN bedroom images.

Contribution. Most prior work focus on training new capabilities into diffusion models, and this inevitably requires large compute and data. Our main contribution is showing that a very small amount of human feedback data and computation is sufficient for guiding a pre-trained diffusion model to do what it can already do while suppressing undesirable behaviors.

1.1 Background on diffusion probabilistic models

Due to space constraints, we defer the comprehensive review of prior works to Appendix D. In this section, we briefly review the standard methods of diffusion probabilistic models (DPM) and set up the notation. For the sake of simplicity and specificity, we only consider the DPMs with the variance preserving SDE.

Consider the variance preserving (VP) SDE

for t ∈ [0 , T ] , where β t > 0 , X t ∈ R d , and W t is a d -dimensional Brownian motion. The process { X t } t ∈ [0 ,T ] has the marginal distributions given by

for t ∈ [0 , T ] [39, Chapter 5.5]. Let p t denote the density for X t for t ∈ [0 , T ] . Anderson's theorem [1] tells us that the the reverse-time SDE by

where { W t } t ∈ [0 ,T ] is a reverse-time Brownian motion, satisfies X t D = X t ∼ p t .

In DPMs, the initial distribution is set as the data distribution, i.e., p 0 = p data in (1), and a score network s θ is trained so that s θ ( X t , t ) ≈ ∇ log p t ( X t ) . For notational convenience, one often uses the error network ε θ ( X t , t ) = - √ 1 -α t s θ ( X t , t ) . Then, the reverse-time SDE is approximated by

for t ∈ [0 , T ] .

When an image X has a corresponding label Y , classifier guidance [40, 12] generates images from

for t ∈ [0 , T ] using

where ω > 0 . This requires training a separate time-dependent classifier approximating p t ( Y | X t ) . In the context of censoring, we use a reward model with binary labels in place of a classifier.

2 Problem description: Censored sampling with human feedback

Informally, our goal is:

Given a pre-trained diffusion model that is partially misaligned in the sense that generates both 'good' and 'bad' images, fix/modify the generation process so that only good images are produced.

The meaning of 'good' and 'bad' depends on the context and will be specified through human feedback. For the sake of precision, we define the terms 'benign' and 'malign' to refer to the good and bad images: A generated image is malign if it contains unwanted features to be censored and is benign if it is not malign.

Our assumptions are: (i) the pre-trained diffusion model does not know which images are benign or malign, (ii) a human is willing to provide minimal ( ∼ 3 minutes) feedback to distinguish benign and malign images, and (iii) the compute budget is limited.

Mathematical formalism. Suppose a pre-trained diffusion model generates images from distribution p data ( x ) containing both benign and malign images. Assume there is a function r ( x ) ∈ (0 , 1) representing the likelihood of x being benign, i.e., r ( x ) ≈ 1 means image x is benign and should be considered for sampling while r ( x ) ≈ 0 means image x is malign and should not be sampled. We mathematically formalize our goal as: Sample from the censored distribution

Human feedback. The definition of benign and malign images are specified through human feedback. Specifically, we ask a human annotator to provide binary feedback Y ∈ { 0 , 1 } for each image X through a simple graphical user interface shown in Appendix E. The feedback takes 1-3 human-minutes for the relatively easier censoring tasks and at most 10-20 human-minutes for the most complex task that we consider. Using the feedback data, we train a reward model r ψ ≈ r , which we further detail in Section 3.

Evaluation. The evaluation criterion of our methodology are the human time spent providing feedback, quantified by direct measurement, and sample quality, quantified by precision and recall.

In this context, precision is the proportion of benign images, and recall is the sample diversity of the censored generation. Precision can be directly measured by asking human annotators to label the final generated images, but recall is more difficult to measure. Therefore, we primarily focus on precision for quantitative evaluation. We evaluate recall qualitatively by providing the generated images for visual inspection.

3 Reward model and human feedback

Let Y be a random variable such that Y = 1 if X is benign and Y = 0 if X is malign. Define the time-independent reward function as

Algorithm 1 Reward model ensemble

As we later discuss in Section 4, time-dependent guidance requires a time-dependent reward function. Specifically, let X ∼ p data and Y be its label. Let { X t } t ∈ [0 ,T ] be images corrupted by the VP SDE (1) with X 0 = X . Define the time-dependent reward function as

We approximate the reward function r with a reward model r ψ , i.e., we train

using human feedback data ( X (1) , Y (1) ) , . . . , ( X ( N ) , Y ( N ) ) . (So the time-dependent reward model uses ( X ( n ) t , Y ( n ) ) as training data.) We use weighted binary cross entropy loss. In this section, we describe the most essential components of the reward model while deferring details to Appendix F.

The main technical challenge is achieving extreme human-feedback efficiency. Specifically, we have N < 100 in most setups we consider. Finally, we clarify that the diffusion model (score network) is not trained or fine-tuned. We use relatively large pre-trained diffusion models [12, 36], but we only train the relatively lightweight reward model r ψ .

3.1 Reward model ensemble for benign-dominant setups

In some setups, benign images constitute the majority of uncensored generation. Section 5.2 considers such a benign-dominant setup, where 11.4% of images have stock photo watermarks and the goal is to censor the watermarks. A random sample of images provided to a human annotator will contain far more benign than malign images.

To efficiently utilize the imbalanced data in a sample-efficient way, we propose an ensemble method loosely inspired by ensemble-based sample efficient RL methods [23, 6]. The method trains K reward models r (1) ψ 1 , . . . , r ( K ) ψ K , each using a shared set of N M (scarce) malign images joined with N M benign images randomly subsampled bootstrap-style from the provided pool of N B (abundant) benign data as in Algorithm 1. The final reward model is formed as r ψ = ∏ K k =1 r ( k ) ψ k . Given that a product becomes small when any of its factor is small, r ψ is effectively asking for unanimous approval across r (1) ψ 1 , . . . , r ( K ) ψ K .

In experiments, we use K = 5 . We use the same neural network architecture for r (1) ψ 1 . . . , r ( K ) ψ K , whose parameters ψ 1 , . . . , ψ K are either independently randomly initialized or transferred from the same pre-trained weights as discussed in Section 3.3. We observe that the ensemble method significantly improves the precision of the model without perceivably sacrificing recall.

3.2 Imitation learning for malign-dominant setups

In some setups, malign images constitute the majority of uncensored generation. Section 5.3 considers such a malign-dominant setup, where 69% of images are tench (fish) images with human faces and the goal is to censor the images with human faces. Since the ratio of malign images starts out high, a single round of human feedback and censoring may not sufficiently reduce the malign ratio.

Therefore, we propose an imitation learning method loosely inspired by imitation learning RL methods such as DAgger [37]. The method collects human feedback data in multiple rounds and improves the reward model over the rounds as described in Algorithm 2. Our experiment of Section 5.3 indicates that 2-3 rounds of imitation learning dramatically reduce the ratio of malign images. Furthermore,

imitation learning is a practical model of an online scenario where one continuously trains and updates the reward model r ψ while the diffusion model is continually deployed.

Ensemble vs. imitation learning. In the benign-dominant setup, imitation learning is too costly in terms of human feedback since acquiring sufficiently many ( ∼ 10 ) malign labels may require the human annotator to go through too many benign labels ( ∼ 1000 ) for the second round of human feedback and censoring. In the malign-dominant setup, one can use a reward model ensemble, where reward models share the benign data while bootstrap-subsampling the malign data, but we empirically observe this to be ineffective. We attribute this asymmetry to the greater importance of malign data over benign data; the training objective is designed so as our primary goal is to censor malign images.

3.3 Transfer learning for time-independent reward

To further improve human-feedback efficiency, we use transfer learning. Specifically, we take a ResNet18 model [17, 18] pre-trained on ImageNet1k [10] and replace the final layer with randomly initialized fully connected layers which have 1-dimensional output features. We observe training all layers to be more effective than training only the final layers. We use transfer learning only for training time-independent reward models, as pre-trained time-dependent classifiers are less common. Transfer learning turns out to be essential for complex censoring tasks (Sections 5.2, 5.4 and 5.5), but requires guidance techniques other than the simple classifier guidance (see Section 4).

4 Sampling

In this section, we describe how to perform censored sampling with a trained reward model r ψ . We follow the notation of Section 1.1.

Time-dependent guidance. Given a time-dependent reward model r ψ ( X t , t ) , our censored generation follows the SDE

for t ∈ [0 , T ] with ω > 0 . From the standard classifier-guidance arguments [42, Section I], it follows that X 0 ∼ p censor ( x ) ∝ p data ( x ) r ( x ) approximately when ω = 1 . The parameter ω > 0 , which we refer to as the guidance weight , controls the strength of the guidance, and it is analogous to the 'gradient scale' used in prior works [12]. Using ω > 1 can be viewed as a heuristic to strengthen the effect of the guidance, or it can be viewed as an effort to sample from p ( ω ) censor ∝ p data r ω .

Time-independent guidance. Given a time-independent reward model r ψ ( X t ) , we adopt the ideas of universal guidance [2] and perform censored generation via replacing the ˆ ε θ of (2) with

Single

Union

Ensemble

Ensemble

+Universal

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Proportion of

malign generation (%)

Single

Union

Ensemble

Ensemble

+Universal

1.0

2.0

3.0

4.0

5.0

6.0

Proportion of

malign generation (%)

Figure 2: Mean proportion of malign images after censoring with standard deviation over 5 trials, each measured with 500 samples. Reward ensemble outperforms non-ensemble models, and the universal guidance components further improve the results. Left: Censoring 'crossed 7' from MNIST. Before censoring, the proportion is 11.9%. The mean values of each point are: 1.30%, 0.98%, 0.60%, and 0.42% . Right: Censoring watermarks from LSUN Church. Before censoring, the proportion is 11.4%. The mean values of each point are: 3.02%, 3.84%, 1.36%, and 0.76% .

for t ∈ [0 , T ] with ω > 0 . To clarify, ∇ differentiates through ˆ X 0 . While this method has no mathematical guarantees, prior work [2] has shown strong empirical performance in related setups. 2

Backward guidance and recurrence. The prior work [2] proposes backward guidance and selfrecurrence to further strengthen the guidance. We find that adapting these methods to our setup improves the censoring performance. We provide the detailed description in Appendix G.

5 Experiments

We now present the experimental results. Precision (censoring performance) was evaluated with human annotators labeling generated images. The human feedback time we report includes annotation of training data for the reward model r ψ , but does not include the annotation of the evaluation data.

5.1 MNIST: Censoring 7 with a strike-through cross

In this setup, we censor a handwriting variation called 'crossed 7', which has a horizontal stroke running across the digit, from an MNIST generation, as shown in Figure 1a. We pre-train our own diffusion model (score network). In this benign-dominant setup, the baseline model generates about 11.9% malign images.

We use 10 malign samples to perform censoring. This requires about 100 human feedback labels in total, which takes less than 2 minutes to collect. We observe that such minimal feedback is sufficient for reducing the proportion of crossed 7s to 0 . 42% as shown in Figure 1b and Figure 2a. Further details are provided in Appendix H.

Ablation studies. We achieve our best results by combining the time-dependent reward model ensemble method described in Section 3.1 and the universal guidance components (backward guidance with recurrence) detailed in Appendix G. We verify the effectiveness of each component through an ablation study, summarized in Figure 2a. Specifically, we compare the censoring results using a reward model ensemble (labeled ' Ensemble ' in Figure 2a) with the cases of using (i) a single reward model within the ensemble (trained on 10 malign and 10 benign images; labeled ' Single ') and (ii) a standalone reward model separately trained on the union of all training data (10 malign and 50 benign images; labeled ' Union ') used in ensemble training. We also show that the backward and recurrence components do provide an additional benefit (labeled ' Ensemble+Universal ').

5.2 LSUN church: Censoring watermarks from latent diffusion model

In the previous experiment, we use a full-dimensional diffusion model that reverses the forward diffusion (1) in the pixel space. In this experiment, we demonstrate that censored generation with minimal human feedback also works with latent diffusion models (LDMs) [46, 36], which perform diffusion on a lower-dimensional latent representation of (variational) autoencoders. We use an LDM 3 pre-trained on the 256 × 256 LSUN Churches [36] and censor the stock photo watermarks. In this benign-dominant setup, the baseline model generates about 11.4% malign images.

Training a time-dependent reward model in the latent space to be used with an LDM would introduce additional complicating factors. Therefore, for simplicity and to demonstrate multiple censoring methods, we train a timeindependent reward model ensemble and apply time-independent guidance as outlined in Section 4. To enhance human-feedback efficiency, we use a pre-trained ResNet18 model and use transfer learning as discussed in Section 3.3. We use 30 malign images, and the human feedback takes approximately 3 minutes. We observe that this is sufficient for reducing the proportion of images with watermarks to 0.76% as shown in Figure 1d and Figure 2b. Further details are provided in Appendix I.

Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. As in Section 5.1, we verify the effectiveness of each component through an ablation study, summarized in Figure 2b. The label names follow the same rules as in Section 5.1. Notably, on average, the 'single' models trained with 30 malign and 30 benign samples outperform the 'union' models trained with 30 malign and 150 malign samples.

5.3 ImageNet: Tench (fish) without human faces

Although the ImageNet1k dataset contains no explicit human classes, the dataset does contain human faces, and diffusion models have a tendency to memorize them [7]. This creates potential privacy risks through the use of reverse image search engines [3]. A primary example is the ImageNet class 'tench' (fish), in which the majority of images are humans holding their catch with their celebrating faces clearly visible and learnable by the diffusion model.

In this experiment, we use a conditional diffusion model 4 pre-trained on the 128 × 128 ImageNet dataset [12] as baseline and censor the instances of class 'tench' containing human faces (but not other human body parts such as hands and arms). In this malign-dominant setup, the baseline model generates about 68.6% malign images.

We perform 3 rounds of imitation learning with 10 malign and 10 benign images in each round to train a single reward model. The human feedback takes no more than 3 minutes in total. We observe that this is sufficient for reducing the proportion of images with human faces to 1 . 0% as shown in Figure 1f and Figure 3. Further details are provided in Appendix J.

Ablation studies. We verify the effectiveness of imitation learning by comparing it with training the reward model at once using the same number of total samples. Specifically, we use 20 malign and 20 benign samples from the baseline generation to train a reward model (labeled ' non-imitation (20 malign) ' in Figure 3a) and compare the censoring results with round 2 of imitation learning; similarly we compare training at once with 30 malign and 30 benign samples (labeled ' non-imitation (30 malign) ') and compare with round 3. We consistently attain better results with imitation learning. As in previous experiments, the best precision is attained when backward and recurrence are combined with imitation learning (labeled ' 30+Univ ').

We additionally compare our censoring method with another approach: rejection sampling, which simply generates samples from the baseline model and rejects samples X such that r ψ ( X ) is less than the given acceptance threshold. Figure 3b shows that rejection sampling yields worse precision compared to the guided generation using the same reward model, even when using the conservative threshold 0.8. We also note that rejection sampling in this setup accepts only 28.2% and 25.5% of the generated samples respectively for thresholds 0.5 and 0.8 on average, making it suboptimal for situations where reliable real-time generation is required.

0

10

20

30

30+Univ

# of malign training samples

0

20

40

60

Proportion of

malign generation (%)

imitation learning

non-imitation (20 malign)

non-imitation (30 malign)

(a) Ablation results for imitation learning

Rejection

(0.5)

Rejection

(0.8)

Guidance

(Round 1)

20

30

40

Proportion of

malign generation (%)

(b) Comparsion with rejection sampling

Single

Union

Ensemble

Ensemble

+Universal

Rejection

(0.5)

Rejection

(0.8)

2

4

6

8

10

Proportion of

malign generation (%)

5.4 LSUN bedroom: Censoring broken bedrooms

Generative models often produce images with visual artifacts that are apparent to humans but are difficult to detect and remove via automated pipelines. In this experiment, we use a pre-trained diffusion model 5 trained on 256 × 256 LSUN Bedroom images [12] and censor 'broken' images as perceived by humans. In Appendix K, we precisely define the types of images we consider to be broken, thereby minimizing subjectivity. In this benign-dominant setup, the baseline model generates about 12.6% malign images.

This censoring task is the most difficult one we consider, and we design this setup as a 'worst case' on the human work our framework requires. We use 100 malign samples to train a reward-model ensemble. This requires about 900 human feedback labels, which takes about 15 minutes to collect. To enhance human-feedback efficiency, we use a pre-trained ResNet18 model and use transfer learning as discussed in Section 3.3. We observe that this is sufficient for reducing the proportion of malign images to 1.36% as shown in Figure 1h and Figure 4. Further details are provided in Appendix K.

Ablation studies. We achieve our best results by combining the (time-independent) reward ensemble and backward guidance with recurrence. We verify the effectiveness of each component through an ablation study summarized in Figure 4. We additionally find that rejection sampling, which rejects a sample X such that 1 K ∑ K k =1 r ( k ) ψ k ( X ) is less than a threshold, yields worse precision compared to the guided generation using the ensemble model and has undesirably low average acceptance ratios of 74.5% and 55.8% when using threshold values 0.5 and 0.8, respectively.

Single

Union

Ensemble

Ensemble

+Universal

2

4

6

8

Proportion of

malign generation (%)

5.5 Stable Diffusion: Censoring unexpected embedded texts

Text-to-image diffusion models, despite their remarkable prompt generality and performance, are known to often generate unexpected and unwanted artifacts or contents. In this experiment, we use Stable Diffusion 6 v1.4 to demonstrate that our methodology is readily applicable to aligning text-to-image models. The baseline Stable Diffusion, when given the prompt 'A photo of a human', occasionally produces prominent embedded texts or only display texts without any visible human figures (as in Figure 1i). We set the output resolution to 512 × 512 and censor the instances of this behavior. In this benign-dominant setup, the baseline model generates about 23.7% malign images.

This censoring problem deals with the most complex-structured model and demonstrates the effectiveness of our methodology under the text-conditional setup. We use 100 malign samples to train a reward-model ensemble. This requires about 600 human feedback labels, which takes no more than 5 minutes to collect. We use a pre-trained ResNet18 model and use transfer learning as discussed in Section 3.3. We observe that this is sufficient for reducing the proportion of malign images to 1.24% as shown in Figure 1j and Figure 5. Further details are provided in Appendix L.

Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. We verify the effectiveness of each component through an ablation study, summarized in Figure 5. Similarly as in Section 5.2, we observe that the 'single' models outperform the 'union' models on average.

Note on the guidance weight ω . We speculate that the effective scale of the guidance weight ω grows (roughly doubles) with 1) significant scale growth in terms of data size and 2) the introduction of new modality (e.g. unconditional or class-conditional → text-conditional model). We use ω = 1 . 0 for the simplest task of Section 5.1, while we use ω = 2 . 0 for Sections 5.2 and 5.4 where the data size grows to 256 × 256 . For this section where we use text-conditioning, we use ω = 4 . 0 .

6 Conclusion

In this work, we present censored sampling of diffusion models based on minimal human feedback and compute. The procedure is conceptually simple, versatile, and easily executable, and we anticipate our approach to find broad use in aligning diffusion models. In our view, that diffusion models can be controlled with extreme data-efficiency, without fine-tuning of the main model weights, is an interesting observation in its own right (although the concept of guided sampling itself is, of course, not new [40, 12, 32, 35]). We are not aware of analogous results from other generative models such as GANs or language models; this ability to adapt/guide diffusion models with external reward functions seems to be a unique trait, and we believe it offers a promising direction of future work on leveraging human feedback with extreme sample efficiency.

Acknowledgments and Disclosure of Funding

TY, KM, and EKR were supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] and a grant from KRAFTON AI. AN was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1F1A1059567). We thank Sehyun Kwon and Donghwan Rho for providing valuable feedback.

References

B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

A. Bansal, H.-M. Chu, A. Schwarzschild, S. Sengupta, M. Goldblum, J. Geiping, and T. Goldstein. Universal guidance for diffusion models. arXiv preprint arXiv:2302.07121 , 2023.

A. Birhane and V. U. Prabhu. Large image datasets: A pyrrhic win for computer vision? WACV , 2021.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. S. Chatterji, A. S. Chen, K. A. Creel, J. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie, K. Goel, N. D. Goodman, S. Grossman, N. Guha, T. Hashimoto, P. Henderson, J. Hewitt, D. E. Ho, J. Hong, K. Hsu, J. Huang, T. F. Icard, S. Jain, D. Jurafsky, P. Kalluri, S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. S. Krass, R. Krishna, R. Kuditipudi, A. Kumar, F. Ladhak, M. Lee, T. Lee, J. Leskovec, I. Levent, X. L. Li, X. Li, T. Ma, A. Malik, C. D. Manning, S. P. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nie, J. C. Niebles, H. Nilforoshan, J. F. Nyarko, G. Ogut, L. Orr, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. H. Roohani, C. Ruiz, J. Ryan, C. R'e, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. P. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. E. Wang, W. Wang, B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. A. Zaharia, M. Zhang, T. Zhang, X. Zhang, Y. Zhang, L. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3-4):324-345, 1952.

J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. NeurIPS , 2018.

N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 , 2023.

T. Chong, I.-C. Shen, I. Sato, and T. Igarashi. Interactive optimization of generative image modelling using sequential subspace search and content-based guidance. Computer Graphics Forum , 2021.

P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. NeurIPS , 2017.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li. ImageNet: A large-scale hierarchical image database. CVPR , 2009.

L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

P. Dhariwal and A. Nichol. Diffusion models beat GANs on image synthesis. NeurIPS , 2021.

R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification . John Wiley & Sons, 2006.

B. Efron. Estimating the error rate of a prediction rule: Improvement on cross-validation. Journal of the American statistical association , 78(382):316-331, 1983.

B. Efron and R. Tibshirani. Improvements on cross-validation: The 632+ bootstrap method. Journal of the American Statistical Association , 92(438):548-560, 1997.

R. Gandikota, J. Materzynska, J. Fiotto-Kaufman, and D. Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345 , 2023.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR , 2016.

K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. ECCV , 2016.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. NeurIPS , 2020.

J. Ho and T. Salimans. Classifier-free diffusion guidance. NeurIPS Workshop on Deep Generative Models , 2021.

B. Kawar, R. Ganz, and M. Elad. Enhancing diffusion-based image synthesis with robust classifier guidance. Transactions on Machine Learning Research , 2023.

D. Kim, Y. Kim, S. J. Kwon, W. Kang, and I.-C. Moon. Refining generative process with discriminator guidance in score-based diffusion models. ICML , 2023.

T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. ICLR , 2018.

A. K. Lampinen, D. So, D. Eck, and F. Bertsch. Improving image generative models with human interactions. arXiv preprint arXiv:1709.10459 , 2017.

K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023.

K. Lee, L. Smith, and P. Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. ICML , 2021.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. ICLR , 2019.

T. Moon, M. Choi, G. Lee, J.-W. Ha, and J. Lee. Fine-tuning diffusion models with limited data. NeurIPS 2022 Workshop on Score-Based Methods , 2022.

MosaicML. Training stable diffusion from scratch costs <160k. https://www. mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k . Accessed: 2023-05-01.

E. Mostaque. ( @EMostaque ) 'We actually used 256 A100s for this per the model card, 150k hours in total so at market price $600k'. https://twitter.com/EMostaque/status/ 1563870674111832066 . Accessed: 2023-05-01.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. ICML , 2021.

A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. ICML , 2022.

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. NeurIPS , 2022.

R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage, H. Hajishirzi, and Y. Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. ICLR , 2023.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. CVPR , 2022.

S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS , 2011.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 2022.

S. Särkkä and A. Solin. Applied Stochastic Differential Equations . Cambridge University Press, 2019.

J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML , 2015.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. ICLR , 2021.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. ICLR , 2021.

N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. NeurIPS , 2020.

Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.03751 , 2023.

S. Um and J. C. Ye. Don't play favorites: Minority guidance for diffusion models. arXiv preprint arXiv:2301.12334 , 2023.

A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. NeurIPS , 2021.

F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

E. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17591 , 2023.

J.-Y. Zhu, P. Krähenbühl, E. Shechtman, and A. A. Efros. Generative visual manipulation on the natural image manifold. ECCV , 2016.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

A Broader impacts & safety

As our research aims to suppress undesirable behaviors of diffusion models, our methodology carries the risk of being used maliciously to guide the diffusion model toward malicious behavior. Generally, research on alignment carries the risk of being flipped to 'align' the model with malicious behavior, and our work is no exception. However, despite this possibility, it is unlikely that our work will be responsible for producing new harmful materials that a baseline model is not already capable of, as we do not consider training new capabilities into diffusion models. In this sense, our work does not pose a greater risk of harm compared to other work on content filtering.

B Limitations

Our methodology accomplishes its main objective, but there are a few limitations we point out. First, although the execution of our methodology requires minimal (few minutes) human feedback, an objective evaluation of our methodology does require a non-trivial amount of human feedback. Indeed, even though we trained our reward models with 10s of human labels, our evaluation used 1000s of human labels. Also, the methodology is built on the assumption of having access to pretrained diffusion models, and it does not consider how to train new capabilities into the base model or improve the quality of generated images.

C Human subject and evaluation

The human feedback used in this work was provided by the authors themselves. We argue that our work does not require external human subjects as the labeling is based on concrete, minimally ambiguous criteria. For the setups of Sections 5.1 ('crossed 7'), 5.2 ('watermarks'), and 5.3 ('tench') the criteria is very clear and objective. For the setup of Section 5.4 ('broken' bedroom images), we describe our decision protocol in Section K. For transparency, we present comprehensive censored generation results in Sections H to K.

We used existing datasets-ImageNet, LSUN, and MNIST-for our study. These are free of harmful or sensitive content, and there is no reason to expect the labeling task to have any adverse effect on the human subjects.

D Prior Works

DPM. The initial diffusion probabilistic models (DPM) considered forward image corruption processes with finite discrete steps and trained neural networks to reverse them [40, 19, 41]. Later, this idea was connected to a continuous-time SDE formulation [42]. As the SDE formalism tends to be more mathematically and notationally elegant, we describe our methods through the SDE formalism, although all actual implementations require using an discretizations.

The generation process of DPMs is controllable through guidance . One approach to guidance is to use a conditional score network, conditioned on class labels or text information [31, 20, 32, 35, 38]. Alternatively, one can use guidance from another external network. Instances include CLIP guidance [32, 35], which performs guidance with a CLIP model pre-trained on image-caption pairs; discriminator guidance [22], which uses a discriminator network to further enforce consistency between generated images and training data; minority guidance [45], which uses perceptual distances to encourage sapling from low-density regions, and using a adversarially robust classifier [21] to better align the sample quality with human perception. In this work, we adapt the ideas of (time-dependent) classifier guidance of [40, 12] and universal guidance [2].

RLHF. Reinforcement learning with human feedback (RLHF) was originally proposed as a methodology for using feedback to train a reward model, when an explicit reward of the reinforcement learning setup is difficult to specify [9, 26]. However, RLHF techniques have been succesfully used in natural language processing setups with no apparent connection to reinforcement learning [50, 43, 33]. While the RLHF mechanism in language domains is not fully understood, the success indicates that the general strategy of fine-tuning or adjusting the behavior of a pre-trained model with human feedback and reward models is a promising direction.

Controlling generative models with human feedback. The use of human feedback to fine-tune generative models has not yet received significant attention. The prior work of [24] aims to improve the aesthetic quality of the images produced by generative adversarial networks (GANs) using human feedback. There are methods that allow interactive editing of images produced by GANs (i.e., modifying images based on human feedback) but such methods do not fine-tune or modify the generation procedure of GANs [8, 49].

For DPMs, the prior work of [25] fine-tunes the pre-trained Stable Diffusion [36] model to have better image-text alignment using 27,000 human annotations. In [44], rank-based human feedback was used to improve the generated image of a diffusion model on an image-by-image basis. There has been prior work on removing certain concepts from pre-trained DPMs [16, 48], which involve human evaluations, but these approaches do not use human feedback in their methodologies.

Reward models. Many prior work utilizing human feedback utilize reward models in the form of a binary classifier, also called the Bradley-Terry model [5]. However, the specifics of the deep neural network architecture varies widely. In the original RLHF paper [9], the architecture seems to be simple MLPs and CNNs. In [33], the architecture is the same as the GPT-3 architecture except that the unembedding layer is replaced with a projection layer to output a scalar value. In [50, 43], the reward model is a linear function of the language embedding used in the policy network. In [34], the authors use transformer-based architectures to construct the reward models. Overall, the conclusion is that field has not yet converged to a particular type of reward model architecture that is different from the standard architecutres used in related setups. Therefore, we use simple UNet and ResNet18 models for our reward model architectures.

E GUI interface

We collect human feedback using a very minimal graphical user interface (GUI), as shown in the following.

XFeedback data collector

口

X

Please select all images that contain human face,

Red boundary indicates that the image contains human face （for sure)

Blue boundary indicates that it is undecidable whether the image contains human face.

Submit

Save

F Reward model: Further details

Weighted loss function. We train the reward model using the weighted binary cross entropy loss

We use α < 1 to prioritize the model to accurately classify malign images as malign at the expense of potentially misclassifying some benign images as malign.

Data augmentation. We augment the training dataset with 10 to 20 random variations of each training image using rotation, horizontal flip, crop, and color jitter. We augment the data once and train the reward model to fit this augmented data as opposed to applying a random augmentation every time the data is loaded.

Bootstrap subsampling. As discussed in Section 3.1, we use the reward model ensemble in the benign-dominant setup, where labeled benign images are more plentiful while there is a relatively limited quantity of N m malign images. The K reward models of the ensemble utilize the same set of N m malign images. As for the benign images, we implement a resampling strategy that is inspired by bootstrapping [14, 15, 13]. Each model selects N m benign images independently with replacement from the pool of labeled benign images.

G Backward guidance and recurrence

We describe backward guidance and recurrence, techniques inspired by the universal guidance of [2].

G.1 Backward guidance

Compute ˆ ε θ ( X t , t ) as in (2) or (3) (time-independent or time-dependent guidance) and form

We then take ˆ X fwd 0 as a starting point and perform B steps of gradient ascent with respect to log r ψ ( · ) and obtain ˆ X bwd 0 . Finally, we replace ˆ ε θ by ˆ ε bwd θ such that X t = √ α t ˆ X bwd 0 + √ 1 -α t ˆ ε bwd θ ( X t , t ) holds, i.e.,

G.2 Recurrence

Once ˆ ε bwd θ is computed, the guided sampling is implemented as a discretized step of the backward SDE

Say the discretization step-size is ∆ t , so the update computes X t -∆ t from X t . In recurrent generation, we use the notation X (1) t = X t and X (1) t -∆ t = X t -∆ t and then obtain X (2) t by following the forward noise process of the (discretized) VP SDE (1) starting from X (1) t -∆ t for time ∆ t . We repeat the process R times, sequentially generating X (1) t -∆ t , X (2) t -∆ t , . . . , X ( R ) t -∆ t .

H MNIST crossed 7: Experiment details and image samples

H.1 Diffusion model

For this experiment, we train our own diffusion model. We use the 5,000 images of the digit '7' from the MNIST training set and rescale them to 32 × 32 resolution. The architecture of the error network ε θ follows the UNet implementation 7 of a prior work [12], featuring a composition of residual blocks with downsampling and upsampling convolutions and global attention layers, and time embedding injected into each residual block. We set the input and output channel size of the initial convolutional layer to 1 and 128, respectively, use channel multipliers [1 , 2 , 2 , 2] for residual blocks at subsequent resolutions, and use 3 residual blocks for each resolution. We train the diffusion model for 100,000 iterations using the AdamW [27] optimizer with β 1 = 0 . 9 and β 2 = 0 . 999 , using learning rate 10 -4 , EMA with rate 0.9999 and batch size 256. We use 1,000 DDPM steps.

H.2 Reward model and training

The time-dependent reward model architecture is a half-UNet model with the upsampling blocks replaced with attention pooling to produce a scalar output. The weights are randomly initialized, i.e., we do not use transfer learning. We augment the training (human feedback) data with random rotation in [ -20 , 20] degrees. When using 10 malign and 10 benign feedback data, we use α = 0 . 02 for the training loss BCE α and train all reward models for 1,000 iterations using AdamW with learning rate 3 × 10 -4 , weight decay 0 . 05 , and batch size 128. When we use 10 malign and 50 benign data for the ablation study, we use α = 0 . 005 and train for the same number of epochs as used in the training of 10 malign & 10 benign case, while using the same batch size 128.

H.3 Sampling and ablation study

For sampling via reward ensemble without backward guidance and recurrence, we choose ω = 1 . 0 . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called ' Single ' and ' Union ' in Figure 2a:

' Single ' model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 10 malign images, and a set of 10 benign images.

' Union ' model refers to a model which is trained on 10 malign images and a collection of ∼ 50 benign images, combining the set of benign images used to train the ensemble. This model is trained for 3,000 iterations, with α = 0 . 005 for the BCE α loss.

For these non-ensemble models, we use ω = 5 . 0 , which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence as discussed in Section G, we use ω = 1 . 0 , learning rate 2 × 10 -4 , B = 5 , and R = 4 .

H.4 Censored generation samples

Figure 7 shows uncensored, baseline generation. Figures 8 and 9 shows images sampled with censored generation without and with backward guidance and recurrence.

77777

+

+

7777777

7

7777

7

>77777

7

4

7

+

+

7

7

t

7777

7777777

7777777

7

7

77777777

7

7

7

2

LL

7

777777

2

7

7777777

7

7

7

7

7

I LSUN church: Experiment details and image samples

I.1 Pre-trained diffusion model

We use the pre-trained Latent Diffusion Model (LDM) 8 from [36]. We follow the original settings and use 400 DDIM [41] steps.

I.2 Malign image definition

As shown in Figure 10, the 'Shutterstock' watermark is composed of three elements: the Shutterstock logo in the center, the Shutterstock website address at the bottom, and a white X lines in the background. In the baseline generation, all possible combinations of these three elements arise. We classify an image as 'malign' if it includes either the logo in the center or the website address at the bottom. We do not directly censor the white X lines, as they are often not clearly distinguishable when providing the human feedback. However, we do observe a reduction in the occurrence of the white X lines as they are indirectly censored due to their frequent co-occurrence with the other two elements of the Shutterstock watermark. While the majority of the watermarks are in the Shutterstock format, we did occasionally observe watermarks from other companies as well. We choose to censor only the Shutterstock watermarks as the other types were not sufficiently frequent.

shueete

www.shutterstock.com-302s0021s

Mteruck

co080ocor-mos.xaojzisure.www

I.3 Reward model training

We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' 'DEFAULTS' setting 9 , which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 60 images (30 malign, 30 benign) without data augmentation. We use BCE α in (4) as the training loss with α = 0 . 1 . The models are trained for 600 iterations using AdamW optimizer [27] with learning rate 3 × 10 -4 , weight decay 0 . 05 , and batch size 128 .

I.4 Sampling and ablation study

For sampling via reward ensemble without backward guidance and recurrence, we choose ω = 2 . 0 . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called ' Single ' and ' Union ' in Figure 2b:

' Single ' model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 30 malign images, and a set of 30 benign images.

' Union ' model refers to a model which is trained on 30 malign images and a collection of ∼ 150 benign images, combining the set of benign images used to train the ensemble. This model is trained for 1,800 iterations, with α = 0 . 01 for the BCE α loss.

For these non-ensemble models, we use ω = 10 . 0 , which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with recurrence as discussed in Section G, we use ω = 2 . 0 and R = 4 .

I.5 Censored generation samples

Figure 11 shows uncensored, baseline generation. Figures 12 and 13 present images sampled with censored generation without and with backward guidance and recurrence.

山

ura

金

J ImageNet tench: Experiment details and image samples

J.1 Pre-trained diffusion model

We use the pre-trained diffusion model 10 from [12], trained on ImagtNet1k dataset [10]. We use (time-dependent) classifier guidance with gradient scale 0.5 as recommended by [12] and 1,000 DDPM steps for sampling to generate samples from the class 'tench'.

J.2 Reward model training

We use same half-UNet architecture as in Section H for the time-dependent reward model. The weights are randomly initialized, i.e., we do not use transfer learning. All hyperparameters are set identical to the values used for training the time-dependent classifier for 128 × 128 ImageNet in the prior work [12], except that we set the output dimension of the attention pooling layer to 1. We augment the training (human feedback) data with random horizontal flips with probability 0 . 5 followed by one of the following transformations: 1) random rotation within [ -30 , 30] degrees, 2) random resized crop with an area of 75-100%, and 3) color jitter with contrast range [0 . 75 , 1 . 33] and hue range [ -0 . 2 , 0 . 2] . We use α = 0 . 1 for the training loss BCE α . When using 10 malign and 10 benign feedback data, we train reward models for 500 iterations using AdamW with learning rate 3 × 10 -4 , weight decay 0 . 05 , and batch size 128. For later rounds of imitation learning, we train for the same number of epochs while using the same batch size 128. In other words, we train for 1,000 iterations for round 2 and 1,500 iterations for round 3.

J.3 Sampling and ablation study

For sampling without backward guidance and recurrence, we choose ω = 5 . 0 . We compare the censoring performance of a reward model trained with imitation learning with reward models trained without the multi-stage imitation learning in the ablation study. We train the non-imitation learning reward model for the same number of cumulative iterations with the corresponding case of comparison; for example, when training with 30 malign and 30 benign images from the baseline, we compare this with round 3 of imitation learning, so we train for 3,000 iterations, which equals the total sum of 500, 1,000 and 1,500 training iterations used in rounds 1, 2, and 3. For censored image generation via backward guidance and recurrence as discussed in Section G, we use ω = 5 . 0 , learning rate 0 . 002 , B = 5 , and R = 4 .

J.4 Censored generation samples

Figure 14 shows uncensored, baseline generation. Figures 15 and 16 present images sampled with censored generation without and with backward guidance and recurrence.

K LSUN bedroom: Experiment details and image samples

K.1 Pre-trained diffusion model

We use the pre-trained diffusion model 11 from [12], trained on LSUN Bedroom dataset [47]. We follow the original settings, which include 1,000 DDPM steps, image size of 256 × 256 , and linear noise scheduler.

K.2 Malign image definition

We classify an LSUN bedroom image as 'broken' (malign) if it meets at least one of the following criteria:

(a) Obscured room layout: overall shape or layout of the room is not clearly visible;

(b) Distorted bed shape: bed does not present as a well-defined rectangular shape;

(c) Presence of distorted faces: there are distorted faces of humans or dogs;

(d) Distorted or crooked line: line of walls or ceilings are distorted or bent;

(e) Fragmented images: image is divided or fragmented in a manner that disrupts their logical continuity or coherence;

(f) Unrecognizable objects: there are objects whose shapes are difficult to identify;

(g) Excessive brightness: image is too bright or dark, thereby obscuring the forms of objects.

Figure 17 shows examples of the above.

On the other hand, we categorize images with the following qualities as benign, even if they may give the impression of being corrupted or damaged:

(a) Complex patterns: Images that include complex patterns in beddings or wallpapers;

(b) Physical inconsistencies: Images that are inconsistent with physical laws such as gravity or reflection;

(c) Distorted text: Images that contain distorted or unclear text.

Figure 18 shows examples of the above.

K.3 Reward model training

We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' 'DEFAULTS' setting 12 , which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use BCE α in (4) as the training loss with α = 0 . 1 . The models are trained for 5 , 000 iterations using AdamW optimizer [27] with learning rate 3 × 10 -4 , weight decay 0 . 05 , and batch size 128 . We train five reward models for the ensemble.

&

teeo oedanodohrorlo He olasb oodo

K.4 Sampling and ablation study

For sampling via reward ensemble without backward guidance and recurrence, we choose ω = 2 . 0 . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called ' Single ' and ' Union ' in Figure 4:

' Single ' model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and a set of 100 benign images.

' Union ' model refers to a model which is trained on 100 malign images and a collection of ∼ 500 benign images, combining the set of benign images used to train the ensemble. These models are trained for 15,000 iterations with α = 0 . 02 for the BCE α loss.

For these non-ensemble models, we use ω = 10 . 0 , which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence as discussed in Section G, we use ω = 2 . 0 , learning late 0 . 002 , B = 5 , and R = 4 .

K.5 Censored generation samples

Figure 19 shows uncensored, baseline generation. Figures 20-31 present a total of 1,000 images sampled with censored generation, 500 generated by ensemble reward models without backward guidance and recurrence and 500 with backward guidance and recurrence.

品

口口

园

口

国

1日口

国

L Stable Diffusion: Experiment details and image samples

L.1 Pre-trained diffusion model

We use the pretrained Stable Diffusion 13 , version 1.4. We generate images using the default setting, which uses image size of 512 × 512 and 50 DDIM [41] steps.

L.2 Reward model training

We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' 'DEFAULTS' setting 14 , which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use BCE α in (4) as the training loss with α = 0 . 1 . The models are trained for 10 , 000 iterations using AdamW optimizer [27] with learning rate 10 -4 , weight decay 0 . 05 , and batch size 64 .

L.3 Sampling and ablation study

For sampling via reward ensemble without backward guidance and recurrence, we choose ω = 4 . 0 . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called ' Single ' and ' Union ' in Figure 5:

' Single ' model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and a set of 100 benign images.

' Union ' model refers to a model which is trained on 100 malign images and a collection of ∼ 500 benign images, combining the set of benign images used to train the ensemble. These models are trained for 30,000 iterations with α = 0 . 02 for the BCE α loss.

For these non-ensemble models, we use ω = 20 . 0 , which is K = 5 times the guidance weight used in the ensemble case. For censored image generation using ensemble combined recurrence as discussed in Section G, we use ω = 4 . 0 and R = 4 .

L.4 Censored generation samples

Figure 32 shows uncensored, baseline generation. Figures 33 and 34 respectively present images sampled with censored generation without and with recurrence.

HUIAI

HUIMA

A ACMAA

AIR

WHOMTG

HMAIMAN

HMAAN

,HIMaN .

ASHUAN

HUMMA

HUN!

ITTANZWH300009

HUMN HUMAAN

DOUFOOOALE

TIRLIE...

AMONKON

HUIN HIAN

B

VIEAHUAN

回

AME

HOMRONA

ArINo-

M

HUALAILAT

A

nayonyeteakio loho

A ou dadsn Ahe adzeny

windgcilivanewivovy

tondholndsfiosnic wiold

nchhse wilas tho opone,ren.

wngoaidomninaeiosc

Weondrutwoln inleWoushe

tatyhotnoepbnteeeinon

gbmfur vdcbinenerbs)

Hima

IGD

HUMA

I参HFL

9

M Transfer learning ablation

To demonstrate the necessity of transfer learning for relatively more complex tasks, we compare it with training reward model from scratch. We consider the LSUN bedroom task of Section 5.4. We randomly initialize the weights of time-dependent reward model with half-UNet architecture and train it for 40,000 iterations with batch size 128. We use the training loss BCE α with α = 0 . 1 and use the guidance weight of ω = 10 . 0 for sampling.

Weobserve that censoring fails without transfer learning, despite our best efforts to tune the parameters. The reward model successfully interpolates the training data, but its classification performance on the test dataset (which we create separately using additional human feedback data) is poor: it shows 70.63% and 43.23% accuracy respectively for malign and benign test images. If we nevertheless proceed to perform censored generation using this reward model (trained without transfer learning), the proportion of malign images is 15 . 68% ± 5 . 25% (measured using 500 generated images across 5 independent trials). This is no better than 12.6% of the baseline model without censoring.

N Using malign images from secondary source

In this section, we demonstrate the effectiveness of our framework even with malign images from a secondary source, instead of model-generated images. This strategy may be useful, e.g., in cases where the baseline model rarely generates malign images (making it difficult to collect a sufficient number of them) but a user still wishes to impose hard restriction of not generating certain images.

We consider the setup of Section 5.2, and repeat the similar procedure except that we use malign images manually crawled from the Shutterstock webpage 15 ; we utilize Microsoft Windows Snipping Tool to take screenshots of 'church', 'gothic church' and 'cathedral' images with clearly visible watermarks, resize them to 256 × 256 resolution and apply random horizontal flip. We use a fixed set of these 30 secondary malign images (Figure 35) and 30 benign images generated by the baseline model to train each reward model. We use the same hyperparameters as in Section I for reward training and sampling, except the only difference of using the guidance weight ω = 1 . 0 .

cck

ntock

We display the result of censoring using secondary malign images in Table 1 (labeled 'secondary censoring'). We ensemble 5 reward models (trained on the same set of 30 secondary malign images) and apply recurrence with R = 4 . Recall that the baseline model generates 11.4% malign images, while with secondary censoring, the proportion drops to 1.4%. This is slightly worse than 0.76% of our ordinary censoring methodology (Figure 2), but still shows that secondary malign images could be effective alternatives to model-generated images in certain setups.

Peer reviews:
comments: SUMMARY:
The work aims to solve censored diffusion sampling problem, which prevent diffusion model generating malign / bad images. The core approach is to train a classifier and apply classifier-guided diffusion generation.

STRENGTHS:
1. Authors present an interesting finding that the classifier-guided diffusion can effectively reduce the probability of generating malign images.
2. Authors present two approaches to learn the classifier, ensemble-based approaches when number of malign images less than benign images, imitation learning for when malign images are dominates.
3. The proposed methods are lightweight and can be effective with small labeled datasets.

WEAKNESSES:
The core technique employed in the work is not new.  And compared with existing works that are based on text2image diffusion models, authors use relatively simple diffusion models that are either class-condition diffusion models or diffusion models for one category dataset. I am not sure the effectiveness and efficiency of the proposed method can be scaled to a large diffusion model that handles for complicate diffusion models. The classifiers in three experiments are relatively easy and that may explain why it is so effective. It may not be true in large text2image settings.

QUESTIONS:
It would be more convincing that the authors provide more experiments for more complex diffusion models, such as stable diffusion.

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

comments: SUMMARY:
The authors examine the problem of preventing the generation of certain types of images generated by a diffusion model. To achieve this, the authors propose using a reward model trained on human feedback. The authors demonstrate their approach from examples that require minimal human feedback to achieve sufficient censoring performance.

STRENGTHS:
- The authors make a good argument in that 3 minutes of human feedback can be more cost-efficient than retraining models.
- Their discussion and experiments in benign-dominant and malign-dominant scenarios are thorough. 
- Their ablation studies in the experiments give convincing arguments that human feedback works well.

WEAKNESSES:
- A discussion of when to use time-dependent or time-independent would have been nice.
- In the LSUN bedroom experiment, it took 15 minutes so it doesn't always take 3 minutes.

QUESTIONS:
- Line 52: there are two consecutive "the".
- Figure 3(b) text: "Comparsion" to "Comparison"
- How far can this method go? I'm sure there are practioners that want 0% malign images.

LIMITATIONS:
- The authors do not explicitly state their limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents censored diffusion model training by using a reward model trained using human feedback. Towards this, the paper utilizes reward model ensembles (for benign dominant settings) and tools from imitation learning (for malign dominant settings).\n\nSTRENGTHS:\nThe human feedback part is pretty interesting and is a topic of increasing interest. The empirical results appear to be pretty compelling, though, I am not an expert with respect to evaluating this in particular, so I cannot comment on this with high confidence.\n\n==> post rebuttal: I increased the score from a 5 to a 6.\n\nWEAKNESSES:\nThe techniques utilized by the paper are pretty well known and in that sense, this paper's contributions appear incremental. \n\nOne other question that appears to not be addressed is - by performing this paper's procedure iteratively (as multiple rounds), would the probability of producing malign images be brought down to zero? What are the other ways of achieving this?\n\nAnother weakness of the paper is that it is not clear to me if the proposed approach has been compared against other concept removal methods in the literature (which I agree I am not an expert on), but, this is something the authors must either present clarifications on, or, provide more comparisons on.\n\nQUESTIONS:\na. i wonder if the malign dominant settings can be handled by flipping the reward to be something resembling 1-\\prod_{i=1}^K (1-r_{\\psi_k}^{k})?\n\nb. does it help to train every single reward model with some form of hard negative mining? I mean this in the context of algorithm 1.\n\nLIMITATIONS:\nI believe the authors must present a discussion surrounding this in the context of their work - what it seeks to address, what remains to be addressed, and how much to trust the proposed algorithm as to whether it truly achieves what it sets out to. I am not certain about going through an ethics review, and I hope the AC and the authors try to see what is necessary in this direction as it clearly seems like there is potential for fleshing some of these concerns out.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of preventing the generation of unwanted images in diffusion models. It formulates the task of 'censoring' and proposes using reward model trained from human labelling to guide the diffusion model. The method requires no fine-tuning and a few minutes of human feedback, while displaying significant reduction of unwanted images in four experiment setups.\n\nSTRENGTHS:\n- This work identifies and formulates an important problem of misalignment in diffusion probabilistic models, which underwent few studies in the past. Solution to the problem can significant improve the utility of generative models and lead to positive social impact.\n\n- RLHF is an emerging technology which is theoretically sound and empirically useful. The proposed method successfully bring the idea of RLHF into diffusion models while combining the idea of classifier guidance. The method itself is clean and simple to train, and requires few modifications to the sampling process. \n\n- From the experiments, the method seem also quite efficient: (i) it sample efficient: requiring up to 100 malign samples, (ii)  it does not need fine-tuning; (iii) it works under limited human feedback, taking only a few minutes of human labeling. These properties are desirable in practical implementation.\n\nWEAKNESSES:\n- Under the general framework of training reward models and sampling with guidance, the authors use four different methods to train the reward model for the four different tasks in the paper. The methods are quite heuristic. It appears that for any new concept and dataset, it still requires a manual selection of the training algorithms for the reward models to achieve the best effect, which can be costly. However, the paper does not provide a conclusive algorithm or any unifying principles for the selection. In this sense the study seems unfinished.\n\n- There are no large-scale, principled comparisons with baselines. Even if there are limited existing works studying the problem, I think there are still some naive methods such as using natural language prompts, fine-tuning, and post-selection with classifier. It is not straightforward to see why the method proposed in this paper is the best.\n\nQUESTIONS:\n- Why is it better to use time-independent reward model for censoring watermarks and time-dependent reward model for other tasks?  Is there a general reason why time-independent reward models work better for some tasks and not others?\n\n- Any guidance of choosing $\\omega$ ?\n\n- It appears from the paper that the training of reward models need large datasets containing images related to a single unwanted concept (for example fish with human faces). If I do not have such high-quality, single-purpose datasets, maybe only random images with all kinds of undesirable concepts, can the method still work well?\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors combine pre-trained diffusions with classifiers trained on human feedback about which type of images to omit, and use the classifiers to guide diffusion sampling using the Universal Guidance technique. They observe that they are able to filter out several types of malign images on a variety of datasets, notably removing undesirable artifacts on LSUN and faces on Imagenet.\n\nI would recommend this paper for acceptance.\n\nSTRENGTHS:\n- People care about censoring sampled outputs, and don't like re-training large diffusions\n- Together with the appendix, the specifics of how exactly to guide are well explored in this work\n- I like that that while \"bad\" images are hard to quantify with some kind of numerical per-sample score, it is easy for a human to recognize instances of it when they do see it.\n- emphasizing that not much feedback is needed for the cases explored\n\nWEAKNESSES:\nThe main concerns would naturally be:\n- how much feedback is enough\n- are the good results specific to the type of things being censored in the particular datasets chosen\n- for the infrequent malign content case, can a second dataset be used in place of model samples?\n\nSee questions below.\n\nOther small comment: change \"man hours\" to \"human work hours\"\n\nQUESTIONS:\nFor the first concern, it would be great if you could include something showing the number of model-produced malign samples going down as a function of number of malign samples used in classifier training, or as a function of number of total samples used for classifier training, or both. I know this is hard to do since the outputs need to be checked manually. Even doing this with a batch of 128 for a few classifier-training-set-sizes would be good, and see e.g. the malign proportion lowers for several feedback dataset sizes.\n\nFor the second concern, the crossed 7's are distinct enough in MNIST such that a very small classifier could quickly assign zero weight to them. However, the faces on Imagenet are more convincing since the properties to be censored (faces) do appear in several ways that aren't as clearly disentangled from benign properties. It would be useful to share somewhere in the text a case where the censoring did not work despite what looks like an adequate amount of samples. Did this happen for censoring some other of the Imagenet classes? Are there any cases you explore where the choice of malign samples constitute nearly all of the training set?\n\nFeedback on a second dataset rather than model samples: for the case where a diffusion is trained on a dataset A with low frequency of the malign property (but nevertheless we need to guarantee that samples do not ever contain the property), the model samples will feature malign content infrequently. What if you have a second dataset B where the property is common? If the datasets are the same resolution and are diverse enough, and share some concepts this might work and might be a nice way to get past low malign frequency in dataset A? Consider the case where A only has some faces but B is at least half human faces. So samples from the A-trained-model do not produce malign content often and good feedback is hard to collect. How well does a feedback model trained on a second dataset B (which are not model samples) work for censoring the A-trained-diffusion from generating faces?\n\nLIMITATIONS:\nAddressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to censor the sample generation of pre-trained diffusion probabilistic models to better align with human preferences. The authors use minimal human feedback (<3min spent in providing the feedback for basic tasks, and <15min for more complicated tasks that they consider) to train a light-weight reward model, then use the trained reward model to provide either time-dependent or time-independent guidance for the generation process. The authors also use techniques such as ensemble, iterative imitation learning, transfer learning, backward guidance and recurrence to improve censoring performance. Experiments on MNIST, LSU church, ImageNet, and LSUN bedroom showed the effectiveness of their proposed method.\n\nSTRENGTHS:\nThe paper studies an interesting and important problem of aligning diffusion models with human preferences. By training a relatively lightweight reward model to guide the generation process instead of fine-tuning the large pre-trained model, their proposed approach could potentially save a lot of computation and human feedback.\n\nWEAKNESSES:\n- The paper's structure could be more coherent and consistent. The authors introduce classifier guidance in Section 1.1 but leave its relation to censored sampling unclear until Section 4. This disconnect could confuse readers. Similarly, there is a lack of clarity regarding the reward function $r$ and the reward model $r_\\phi$ introduced in Section 2. While initially, it seems that $r_\\phi \\approx r$, the possibility of $r_\\phi$ becoming time-dependent, as mentioned in Line 97 of Section 3, introduces confusion because $r$ is time-independent. This is not resolved until Section 4, when time-dependent guidance is addressed. To improve readability, it would be beneficial for the authors to introduce the underlying notations and mathematical concepts of the entire model—from the diffusion probabilistic models to the reward models and their role in censored sampling—before discussing training and evaluations.\n- In section 3.1, the authors choose $r_\\phi$ as the product of $K=5$ independently trained reward models. However, they did not have enough discussions or experiments to back up these choices, which makes the resulting model seem arbitrary. In particular, since the authors mentioned $r_\\phi(X)\\approx r(X)=P(Y=1|X)$, it's unclear why taking the product--as opposed to taking the average--makes sense because each $r_{\\psi_k}$ is also approximating $r$. Moreover, if we ignore the mathematical implications for now, the authors explained in line 115 that taking the product is essentially asking for unanimous approval, but readers might still question whether other methods, such as increasing the value of $w$ in the \"union\" method, could work just as well.\n- The paper highlights extreme human feedback efficiency by saying that a few minutes of human feedback are sufficient. However, this claim overlooks the complexity of tasks considered in this paper, which are relatively simple compared to tasks where RLHF is typically employed in large-scale models. Therefore, it would be better to directly compare with previous approaches (such as fine-tuning) under the same task. Additionally, the more complex tasks, like censoring distorted bedrooms, actually require more than the estimated 3 minutes of human feedback. Hence, the title could be seen as potentially misleading or exaggerating the method's efficiency.\n\nQUESTIONS:\n- For the training data $X^{(1)},\\cdots,X^{(N)}$ in Algorithms 1 and 2, are they images corrupted by the VP SDE or images generated by the pre-trained model $\\varepsilon_\\theta$? What are the objectives in training each reward model in the ensemble method?\n- In Algorithm 1, for each $k$ the learner randomly select with replacement $N_M$ benign samples. Does this step also need human feedback?\n- Typo in line 204. \"30 malign and 150 malign samples\"\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}