SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. As in Section 5.1, we verify the effectiveness of each component through an ablation study, summarized in Figure 2b. The label names follow the same rules as in Section 5.1. Notably, on average, the "single" models trained with 30 malign and 30 benign samples outperform the "union" models trained with 30 malign and 150 malign samples.
5.3 ImageNet: Tench (fish) without human faces
Although the ImageNet1k dataset contains no explicit human classes, the dataset does contain human faces, and diffusion models have a tendency to memorize them [7]. This creates potential privacy risks through the use of reverse image search engines [3]. A primary example is the ImageNet class "tench" (fish), in which the majority of images are humans holding their catch with their celebrating faces clearly visible and learnable by the diffusion model.
In this experiment, we use a conditional diffusion model $^{4}$  pre-trained on the  $128 \times 128$  ImageNet dataset [12] as baseline and censor the instances of class "tench" containing human faces (but not other human body parts such as hands and arms). In this malign-dominant setup, the baseline model generates about  $68.6\%$  malign images.
We perform 3 rounds of imitation learning with 10 malign and 10 benign images in each round to train a single reward model. The human feedback takes no more than 3 minutes in total. We observe that this is sufficient for reducing the proportion of images with human faces to  $1.0\%$  as shown in Figure 1f and Figure 3. Further details are provided in Appendix J.
Ablation studies. We verify the effectiveness of imitation learning by comparing it with training the reward model at once using the same number of total samples. Specifically, we use 20 malign and 20 benign samples from the baseline generation to train a reward model (labeled “non-imitation (20 malign)” in Figure 3a) and compare the censoring results with round 2 of imitation learning; similarly we compare training at once with 30 malign and 30 benign samples (labeled “non-imitation (30 malign)” and compare with round 3. We consistently attain better results with imitation learning. As in previous experiments, the best precision is attained when backward and recurrence are combined with imitation learning (labeled “30+Univ”).
We additionally compare our censoring method with another approach: rejection sampling, which simply generates samples from the baseline model and rejects samples  $X$  such that  $r_{\psi}(X)$  is less than the given acceptance threshold. Figure 3b shows that rejection sampling yields worse precision compared to the guided generation using the same reward model, even when using the conservative threshold 0.8. We also note that rejection sampling in this setup accepts only  $28.2\%$  and  $25.5\%$  of the generated samples respectively for thresholds 0.5 and 0.8 on average, making it suboptimal for situations where reliable real-time generation is required.
<sup>4</sup>https://github.com/openai/guided-diffusion
(a) Ablation results for imitation learning
(b) Comparison with rejection sampling
Figure 3: Mean proportion of malign tench images (w/ human face) with standard deviation over 5 trials, each measured with 1000 samples. Left: Before censoring, the proportion is  $68.6\%$ . Using imitation learning and universal guidance, it progressively drops to  $17.8\%$ ,  $7.5\%$ ,  $2.2\%$ , and  $1.0\%$ . Non-imitation learning is worse: with 20 and 30 malign images, the proportions are  $10.7\%$  and  $6.8\%$ . Right: With acceptance thresholds 0.5 and 0.8, rejection sampling via reward models from round 1 produces  $32.0\%$  and  $29.8\%$  of malign images, worse than our proposed guidance-based censoring.
Figure 4: Mean proportion of malign (broken) bedroom images with standard deviation over 5 trials, each measured with 500 samples. Before censoring, the malign proportion is  $12.6\%$ . The mean values of each point are:  $8.68\%$ ,  $8.48\%$ ,  $4.56\%$ ,  $1.36\%$ ,  $4.16\%$ , and  $2.30\%$ .
5.4 LSUN bedroom: Censoring broken bedrooms
Generative models often produce images with visual artifacts that are apparent to humans but are difficult to detect and remove via automated pipelines. In this experiment, we use a pre-trained diffusion model trained on  $256 \times 256$  LSUN Bedroom images [12] and censor "broken" images as perceived by humans. In Appendix K, we precisely define the types of images we consider to be broken, thereby minimizing subjectivity. In this benign-dominant setup, the baseline model generates about  $12.6\%$  malign images.
This censoring task is the most difficult one we consider, and we design this setup as a "worst case" on the human work our framework requires. We use 100 malign samples to train a reward-model ensemble. This requires about 900 human feedback labels, which takes about 15 minutes to collect. To enhance human-feedback efficiency, we use a pre-trained ResNet18 model and use transfer learning as discussed in Section 3.3. We observe that this is sufficient for reducing the proportion of malign images to  $1.36\%$  as shown in Figure 1h and Figure 4. Further details are provided in Appendix K.
Ablation studies. We achieve our best results by combining the (time-independent) reward ensemble and backward guidance with recurrence. We verify the effectiveness of each component through an ablation study summarized in Figure 4. We additionally find that rejection sampling, which rejects a sample  $X$  such that  $\frac{1}{K}\sum_{k = 1}^{K}r_{\psi_k}^{(k)}(X)$  is less than a threshold, yields worse precision compared to the guided generation using the ensemble model and has undesirably low average acceptance ratios of  $74.5\%$  and  $55.8\%$  when using threshold values 0.5 and 0.8, respectively.
<sup>5</sup>https://github.com/openai/guided-diffusion
Figure 5: Mean proportion of malign images (with embedded text) with standard deviation over 5 trials, each measured with 500 samples. Before censoring, the malign proportion is  $23.7\%$ . The mean values of each point are:  $7.10\%$ ,  $7.95\%$ ,  $3.86\%$ ,  $1.24\%$ .
5.5 Stable Diffusion: Censoring unexpected embedded texts
Text-to-image diffusion models, despite their remarkable prompt generality and performance, are known to often generate unexpected and unwanted artifacts or contents. In this experiment, we use Stable Diffusion $^{6}$  v1.4 to demonstrate that our methodology is readily applicable to aligning text-to-image models. The baseline Stable Diffusion, when given the prompt "A photo of a human", occasionally produces prominent embedded texts or only display texts without any visible human figures (as in Figure 1i). We set the output resolution to  $512 \times 512$  and censor the instances of this behavior. In this benign-dominant setup, the baseline model generates about  $23.7\%$  malign images.
This censoring problem deals with the most complex-structured model and demonstrates the effectiveness of our methodology under the text-conditional setup. We use 100 malign samples to train a reward-model ensemble. This requires about 600 human feedback labels, which takes no more than 5 minutes to collect. We use a pre-trained ResNet18 model and use transfer learning as discussed in Section 3.3. We observe that this is sufficient for reducing the proportion of malign images to  $1.24\%$  as shown in Figure 1j and Figure 5. Further details are provided in Appendix L.
Ablation studies. We achieve our best results by combining the time-independent reward model ensemble method described in Section 3.1 and the universal guidance components (recurrence) detailed in Appendix G. We verify the effectiveness of each component through an ablation study, summarized in Figure 5. Similarly as in Section 5.2, we observe that the "single" models outperform the "union" models on average.
Note on the guidance weight  $\omega$ . We speculate that the effective scale of the guidance weight  $\omega$  grows (roughly doubles) with 1) significant scale growth in terms of data size and 2) the introduction of new modality (e.g. unconditional or class-conditional  $\rightarrow$  text-conditional model). We use  $\omega = 1.0$  for the simplest task of Section 5.1, while we use  $\omega = 2.0$  for Sections 5.2 and 5.4 where the data size grows to  $256 \times 256$ . For this section where we use text-conditioning, we use  $\omega = 4.0$ .
In this work, we present censored sampling of diffusion models based on minimal human feedback and compute. The procedure is conceptually simple, versatile, and easily executable, and we anticipate our approach to find broad use in aligning diffusion models. In our view, that diffusion models can be controlled with extreme data-efficiency, without fine-tuning of the main model weights, is an interesting observation in its own right (although the concept of guided sampling itself is, of course, not new [40, 12, 32, 35]). We are not aware of analogous results from other generative models such as GANs or language models; this ability to adapt/guide diffusion models with external reward functions seems to be a unique trait, and we believe it offers a promising direction of future work on leveraging human feedback with extreme sample efficiency.
<sup>6</sup>https://github.com/CompVis/stable-diffusion
TY, KM, and EKR were supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.2021-0-01343, Artificial Intelligence Graduate School Program (Seoul National University)] and a grant from KRAFTON AI. AN was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1F1A1059567). We thank Sehyun Kwon and Donghwan Rho for providing valuable feedback.
[38] C. Sahara, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022.
[39] S. Särkkä and A. Solin. Applied Stochastic Differential Equations. Cambridge University Press, 2019.
[40] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. ICML, 2015.
[41] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. *ICLR*, 2021.
[42] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. *ICLR*, 2021.
[43] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. NeurIPS, 2020.
[44] Z. Tang, D. Rybin, and T.-H. Chang. Zeroth-order optimization meets human feedback: Provable learning via ranking oracles. arXiv preprint arXiv:2303.03751, 2023.
[45] S. Um and J. C. Ye. Don't play favorites: Minority guidance for diffusion models. arXiv preprint arXiv:2301.12334, 2023.
[46] A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. NeurIPS, 2021.
[47] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015.
[48] E. Zhang, K. Wang, X. Xu, Z. Wang, and H. Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17591, 2023.
[49] J.-Y. Zhu, P. Krahenbuhl, E. Shechtman, and A. A. Efros. Generative visual manipulation on the natural image manifold. ECCV, 2016.
[50] D. M. Ziegler, N. Stiannon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
A Broader impacts & safety
As our research aims to suppress undesirable behaviors of diffusion models, our methodology carries the risk of being used maliciously to guide the diffusion model toward malicious behavior. Generally, research on alignment carries the risk of being flipped to "align" the model with malicious behavior, and our work is no exception. However, despite this possibility, it is unlikely that our work will be responsible for producing new harmful materials that a baseline model is not already capable of, as we do not consider training new capabilities into diffusion models. In this sense, our work does not pose a greater risk of harm compared to other work on content filtering.
Our methodology accomplishes its main objective, but there are a few limitations we point out. First, although the execution of our methodology requires minimal (few minutes) human feedback, an objective evaluation of our methodology does require a non-trivial amount of human feedback. Indeed, even though we trained our reward models with 10s of human labels, our evaluation used 1000s of human labels. Also, the methodology is built on the assumption of having access to pretrained diffusion models, and it does not consider how to train new capabilities into the base model or improve the quality of generated images.
C Human subject and evaluation
The human feedback used in this work was provided by the authors themselves. We argue that our work does not require external human subjects as the labeling is based on concrete, minimally ambiguous criteria. For the setups of Sections 5.1 ("crossed 7"), 5.2 ("watermarks"), and 5.3 ("tench") the criteria is very clear and objective. For the setup of Section 5.4 ("broken" bedroom images), we describe our decision protocol in Section K. For transparency, we present comprehensive censored generation results in Sections H to K.
We used existing datasets—ImageNet, LSUN, and MNIST—for our study. These are free of harmful or sensitive content, and there is no reason to expect the labeling task to have any adverse effect on the human subjects.
D Prior Works
DPM. The initial diffusion probabilistic models (DPM) considered forward image corruption processes with finite discrete steps and trained neural networks to reverse them [40, 19, 41]. Later, this idea was connected to a continuous-time SDE formulation [42]. As the SDE formalism tends to be more mathematically and notationally elegant, we describe our methods through the SDE formalism, although all actual implementations require using an discretizations.
The generation process of DPMs is controllable through guidance. One approach to guidance is to use a conditional score network, conditioned on class labels or text information [31, 20, 32, 35, 38]. Alternatively, one can use guidance from another external network. Instances include CLIP guidance [32, 35], which performs guidance with a CLIP model pre-trained on image-caption pairs; discriminator guidance [22], which uses a discriminator network to further enforce consistency between generated images and training data; minority guidance [45], which uses perceptual distances to encourage sapling from low-density regions, and using a adversarially robust classifier [21] to better align the sample quality with human perception. In this work, we adapt the ideas of (time-dependent) classifier guidance of [40, 12] and universal guidance [2].
RLHF. Reinforcement learning with human feedback (RLHF) was originally proposed as a methodology for using feedback to train a reward model, when an explicit reward of the reinforcement learning setup is difficult to specify [9, 26]. However, RLHF techniques have been successfully used in natural language processing setups with no apparent connection to reinforcement learning [50, 43, 33]. While the RLHF mechanism in language domains is not fully understood, the success indicates that the general strategy of fine-tuning or adjusting the behavior of a pre-trained model with human feedback and reward models is a promising direction.
Controlling generative models with human feedback. The use of human feedback to fine-tune generative models has not yet received significant attention. The prior work of [24] aims to improve the aesthetic quality of the images produced by generative adversarial networks (GANs) using human feedback. There are methods that allow interactive editing of images produced by GANs (i.e., modifying images based on human feedback) but such methods do not fine-tune or modify the generation procedure of GANs [8, 49].
For DPMs, the prior work of [25] fine-tunes the pre-trained Stable Diffusion [36] model to have better image-text alignment using 27,000 human annotations. In [44], rank-based human feedback was used to improve the generated image of a diffusion model on an image-by-image basis. There has been prior work on removing certain concepts from pre-trained DPMs [16, 48], which involve human evaluations, but these approaches do not use human feedback in their methodologies.
Reward models. Many prior work utilizing human feedback utilize reward models in the form of a binary classifier, also called the Bradley-Terry model [5]. However, the specifics of the deep neural network architecture varies widely. In the original RLHF paper [9], the architecture seems to be simple MLPs and CNNs. In [33], the architecture is the same as the GPT-3 architecture except that the unembedding layer is replaced with a projection layer to output a scalar value. In [50, 43], the reward model is a linear function of the language embedding used in the policy network. In [34], the authors use transformer-based architectures to construct the reward models. Overall, the conclusion is that field has not yet converged to a particular type of reward model architecture that is different from the standard architectures used in related setups. Therefore, we use simple UNet and ResNet18 models for our reward model architectures.
E GUI interface
We collect human feedback using a very minimal graphical user interface (GUI), as shown in the following.
Figure 6: Simple GUI used to collect human feedback for the setup of Section 5.3. Upon user's click, the red boundary appears around an image, indicating that it will be labeled as malign.
F Reward model: Further details
Weighted loss function. We train the reward model using the weighted binary cross entropy loss
$$
B C E _ {\alpha} \left(r _ {\psi} (x; t), y\right) = - \alpha \cdot y \log r _ {\psi} (x; t) - (1 - y) \log \left(1 - r _ {\psi} (x; t)\right). \tag {4}
$$
We use  $\alpha < 1$  to prioritize the model to accurately classify malign images as malignant at the expense of potentially misclassifying some benign images as malign.
Data augmentation. We augment the training dataset with 10 to 20 random variations of each training image using rotation, horizontal flip, crop, and color jitter. We augment the data once and train the reward model to fit this augmented data as opposed to applying a random augmentation every time the data is loaded.
Bootstrap subsampling. As discussed in Section 3.1, we use the reward model ensemble in the benign-dominant setup, where labeled benign images are more plentiful while there is a relatively limited quantity of  $N_{m}$  malign images. The  $K$  reward models of the ensemble utilize the same set of  $N_{m}$  malign images. As for the benign images, we implement a resampling strategy that is inspired by bootstrapping [14, 15, 13]. Each model selects  $N_{m}$  benign images independently with replacement from the pool of labeled benign images.
G Backward guidance and recurrence
We describe backward guidance and recurrence, techniques inspired by the universal guidance of [2].
G.1 Backward guidance
Compute  $\hat{\varepsilon}_{\theta}(\overline{X}_t,t)$  as in (2) or (3) (time-independent or time-dependent guidance) and form
$$
\hat {X} _ {0} ^ {\mathrm {f w d}} = \frac {\overline {{X}} _ {t} - \sqrt {1 - \alpha_ {t}} \hat {\varepsilon} _ {\theta} (\overline {{X}} _ {t} , t)}{\sqrt {\alpha_ {t}}}.
$$
We then take  $\hat{X}_0^{\mathrm{fwd}}$  as a starting point and perform  $B$  steps of gradient ascent with respect to  $\log r_{\psi}(\cdot)$  and obtain  $\hat{X}_0^{\mathrm{bwd}}$ . Finally, we replace  $\hat{\varepsilon}_{\theta}$  by  $\hat{\varepsilon}_{\theta}^{\mathrm{bwd}}$  such that  $\overline{X}_t = \sqrt{\alpha_t}\hat{X}_0^{\mathrm{bwd}} + \sqrt{1 - \alpha_t}\hat{\varepsilon}_{\theta}^{\mathrm{bwd}}(\overline{X}_t,t)$  holds, i.e.,
$$
\hat {\varepsilon} _ {\theta} ^ {\mathrm {b w d}} (\overline {{X}} _ {t}, t) = \frac {1}{\sqrt {1 - \alpha_ {t}}} \left(\overline {{X}} _ {t} - \sqrt {\alpha_ {t}} \hat {X} _ {0} ^ {\mathrm {b w d}}\right).
$$
G.2 Recurrence
Once  $\hat{\varepsilon}_{\theta}^{\mathrm{bwd}}$  is computed, the guided sampling is implemented as a discretized step of the backward SDE
$$
d \overline {{X}} _ {t} = \beta_ {t} \left(\frac {1}{\sqrt {1 - \alpha_ {t}}} \hat {\varepsilon} _ {\theta} ^ {\mathrm {b w d}} (\overline {{X}} _ {t}, t) - \frac {1}{2} \overline {{X}} _ {t}\right) d t + \sqrt {\beta_ {t}} d \overline {{W}} _ {t}.
$$
Say the discretization step-size is  $\Delta t$ , so the update computes  $\overline{X}_{t - \Delta t}$  from  $\overline{X}_t$ . In recurrent generation, we use the notation  $\overline{X}_t^{(1)} = \overline{X}_t$  and  $\overline{X}_{t - \Delta t}^{(1)} = \overline{X}_{t - \Delta t}$  and then obtain  $\overline{X}_t^{(2)}$  by following the forward noise process of the (discretized) VP SDE (1) starting from  $\overline{X}_{t - \Delta t}^{(1)}$  for time  $\Delta t$ . We repeat the process  $R$  times, sequentially generating  $\overline{X}_{t - \Delta t}^{(1)},\overline{X}_{t - \Delta t}^{(2)},\ldots ,\overline{X}_{t - \Delta t}^{(R)}$ .
H MNIST crossed 7: Experiment details and image samples
H.1 Diffusion model
For this experiment, we train our own diffusion model. We use the 5,000 images of the digit "7" from the MNIST training set and rescale them to  $32 \times 32$  resolution. The architecture of the error network  $\varepsilon_{\theta}$  follows the UNet implementation<sup>7</sup> of a prior work [12], featuring a composition of residual blocks with downsampling and upsampling convolutions and global attention layers, and time embedding injected into each residual block. We set the input and output channel size of the initial convolutional layer to 1 and 128, respectively, use channel multipliers  $[1, 2, 2, 2]$  for residual blocks at subsequent resolutions, and use 3 residual blocks for each resolution. We train the diffusion model for 100,000 iterations using the AdamW [27] optimizer with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ , using learning rate  $10^{-4}$ , EMA with rate 0.9999 and batch size 256. We use 1,000 DDPM steps.
H.2 Reward model and training
The time-dependent reward model architecture is a half-UNet model with the upsampling blocks replaced with attention pooling to produce a scalar output. The weights are randomly initialized, i.e., we do not use transfer learning. We augment the training (human feedback) data with random rotation in  $[-20, 20]$  degrees. When using 10 malign and 10 benign feedback data, we use  $\alpha = 0.02$  for the training loss  $BCE_{\alpha}$  and train all reward models for 1,000 iterations using AdamW with learning rate  $3 \times 10^{-4}$ , weight decay 0.05, and batch size 128. When we use 10 malign and 50 benign data for the ablation study, we use  $\alpha = 0.005$  and train for the same number of epochs as used in the training of 10 malign & 10 benign case, while using the same batch size 128.
H.3 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose  $\omega = 1.0$ . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 2a: