SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- "Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 10 malign images, and a set of 10 benign images.
- "Union" model refers to a model which is trained on 10 malign images and a collection of  $\sim 50$  benign images, combining the set of benign images used to train the ensemble. This model is trained for 3,000 iterations, with  $\alpha = 0.005$  for the  $BCE_{\alpha}$  loss.
For these non-ensemble models, we use  $\omega = 5.0$ , which is  $K = 5$  times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence as discussed in Section G, we use  $\omega = 1.0$ , learning rate  $2 \times 10^{-4}$ ,  $B = 5$ , and  $R = 4$ .
H.4 Censored generation samples
Figure 7 shows uncensored, baseline generation. Figures 8 and 9 show images sampled with censored generation without and with backward guidance and recurrence.
<sup>7</sup>https://github.com/openai/guided-diffusion
Figure 7: Uncensoreded baseline image samples from the diffusion model trained using only images of the digit “7” from MNIST.
Figure 8: Non-curated censored generation samples without backward guidance and recurrence. Reward model ensemble is trained on 10 malign images.
Figure 9: Non-curated censored generation samples with backward guidance and recurrence. Reward model ensemble is trained on 10 malign images.
I LSUN church: Experiment details and image samples
I.1 Pre-trained diffusion model
We use the pre-trained Latent Diffusion Model (LDM)<sup>8</sup> from [36]. We follow the original settings and use 400 DDIM [41] steps.
I.2 Malign image definition
As shown in Figure 10, the "Shutterstock" watermark is composed of three elements: the/Shutterstock logo in the center, the Shutterstock website address at the bottom, and a white X lines in the background. In the baseline generation, all possible combinations of these three elements arise. We classify an image as "malign" if it includes either the logo in the center or the website address at the bottom. We do not directly censor the white X lines, as they are often not clearly distinguishable when providing the human feedback. However, we do observe a reduction in the occurrence of the white X lines as they are indirectly censored due to their frequent co-occurrence with the other two elements of the Shutterstock watermark. While the majority of the watermarks are in the Shutterstock format, we did occasionally observe watermarks from other companies as well. We choose to censor only the Shutterstock watermarks as the other types were not sufficiently frequent.
0028123 - mon katojototu.wwW
www shutterstock.com - 302800213
00000001 - moXoJooJurle.wwW
Figure 10: Examples of LSUN church images with-Shutterstock watermarks.
I.3 Reward model training
We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' "DEFAULTS" setting<sup>9</sup>, which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 60 images (30 malign, 30 benign) without data augmentation. We use  $BCE_{\alpha}$  in (4) as the training loss with  $\alpha = 0.1$ . The models are trained for 600 iterations using AdamW optimizer [27] with learning rate  $3 \times 10^{-4}$ , weight decay 0.05, and batch size 128.
<sup>8</sup>https://github.com/CompVis/latent-diffusion
<sup>9</sup>https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18
I.4 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose  $\omega = 2.0$ . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 2b:
- "Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 30 malign images, and a set of 30 benign images.
- "Union" model refers to a model which is trained on 30 malign images and a collection of  $\sim 150$  benign images, combining the set of benign images used to train the ensemble. This model is trained for 1,800 iterations, with  $\alpha = 0.01$  for the  $BCE_{\alpha}$  loss.
For these non-ensemble models, we use  $\omega = 10.0$ , which is  $K = 5$  times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with recurrence as discussed in Section G, we use  $\omega = 2.0$  and  $R = 4$ .
I.5 Censored generation samples
Figure 11 shows uncensored, baseline generation. Figures 12 and 13 present images sampled with censored generation without and with backward guidance and recurrence.
Figure 11: Uncensored baseline image samples. Malign images are labeled with red borders for visual clarity.
Figure 12: Non-curated censored generation samples without backward guidance and recurrence. Reward model ensemble is trained on 30 malign images. Malign images are labeled with red borders for visual clarity.
Figure 13: Non-curated censored generation samples with recurrence. Reward model ensemble is trained on 30 malign images. Malign images are labeled with red borders for visual clarity.
J ImageNet tench: Experiment details and image samples
J.1 Pre-trained diffusion model
We use the pre-trained diffusion model from [12], trained on ImagtNet1k dataset [10]. We use (time-dependent) classifier guidance with gradient scale 0.5 as recommended by [12] and 1,000 DDPM steps for sampling to generate samples from the class "tench".
J.2 Reward model training
We use same half-UNet architecture as in Section H for the time-dependent reward model. The weights are randomly initialized, i.e., we do not use transfer learning. All hyperparameters are set identical to the values used for training the time-dependent classifier for  $128 \times 128$  ImageNet in the prior work [12], except that we set the output dimension of the attention pooling layer to 1. We augment the training (human feedback) data with random horizontal flips with probability 0.5 followed by one of the following transformations: 1) random rotation within  $[-30, 30]$  degrees, 2) random resized crop with an area of  $75 - 100\%$ , and 3) color jitter with contrast range [0.75, 1.33] and hue range  $[-0.2, 0.2]$ . We use  $\alpha = 0.1$  for the training loss  $BCE_{\alpha}$ . When using 10 malign and 10 benign feedback data, we train reward models for 500 iterations using AdamW with learning rate  $3 \times 10^{-4}$ , weight decay 0.05, and batch size 128. For later rounds of imitation learning, we train for the same number of epochs while using the same batch size 128. In other words, we train for 1,000 iterations for round 2 and 1,500 iterations for round 3.
J.3 Sampling and ablation study
For sampling without backward guidance and recurrence, we choose  $\omega = 5.0$ . We compare the censoring performance of a reward model trained with imitation learning with reward models trained without the multi-stage imitation learning in the ablation study. We train the non-imitation learning reward model for the same number of cumulative iterations with the corresponding case of comparison; for example, when training with 30 malign and 30 benign images from the baseline, we compare this with round 3 of imitation learning, so we train for 3,000 iterations, which equals the total sum of 500, 1,000 and 1,500 training iterations used in rounds 1, 2, and 3. For censored image generation via backward guidance and recurrence as discussed in Section G, we use  $\omega = 5.0$ , learning rate 0.002,  $B = 5$ , and  $R = 4$ .
J.4 Censored generation samples
Figure 14 shows uncensored, baseline generation. Figures 15 and 16 present images sampled with censored generation without and with backward guidance and recurrence.
10https://github.com/openai/guided-diffusion
Figure 14: Uncensored baseline image samples.
Figure 15: Non-curated censored generation samples without backward guidance and recurrence after using 3 rounds of imitation learning each using 10 malign and 10 benign labeled images.
Figure 16: Non-curated censored generation samples with backward guidance and recurrence after using 3 rounds of imitation learning each using 10 malign and 10 benign labeled images.
K LSUN bedroom: Experiment details and image samples
K.1 Pre-trained diffusion model
We use the pre-trained diffusion model<sup>11</sup> from [12], trained on LSUN Bedroom dataset [47]. We follow the original settings, which include 1,000 DDPM steps, image size of  $256 \times 256$ , and linear noise scheduler.
K.2 Malign image definition
We classify an LSUN bedroom image as "broken" (malign) if it meets at least one of the following criteria:
(a) Obscured room layout: overall shape or layout of the room is not clearly visible;
(b) Distorted bed shape: bed does not present as a well-defined rectangular shape;
(c) Presence of distorted faces: there are distorted faces of humans or dogs;
(d) Distorted or crooked line: line of walls or ceilings are distorted or bent;
(e) Fragmented images: image is divided or fragmented in a manner that disrupts their logical continuity or coherence;
(f) Unrecognizable objects: there are objects whose shapes are difficult to identify;
(g) Excessive brightness: image is too bright or dark, thereby obscuring the forms of objects.
Figure 17 shows examples of the above.
On the other hand, we categorize images with the following qualities as benign, even if they may give the impression of being corrupted or damaged:
(a) Complex patterns: Images that include complex patterns in beddings or wallpapers;
(b) Physical inconsistencies: Images that are inconsistent with physical laws such as gravity or reflection;
(c) Distorted text: Images that contain distorted or unclear text.
Figure 18 shows examples of the above.
K.3 Reward model training
We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' "DEFAULTS" setting $^{12}$ , which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use  $BCE_{\alpha}$  in (4) as the training loss with  $\alpha = 0.1$ . The models are trained for 5,000 iterations using AdamW optimizer [27] with learning rate  $3 \times 10^{-4}$ , weight decay 0.05, and batch size 128. We train five reward models for the ensemble.
<sup>11</sup>https://github.com/openai/guided-diffusion
$^{12}$ https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18
(a) Obscured room layout
(d) Distorted or crooked line
(b) Distorted bed shape
(c) Presence of distorted faces
(f) Unrecognizable objects
(e) Fragmented images
(g) Excessive brightness
Figure 17: Examples of "broken" LSUN bedroom images
(a) Complex patterns
Figure 18: Images classified as benign despite giving the impression of being corrupted or damaged.
(b) Physical inconsistencies
(c) Distorted text
K.4 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose  $\omega = 2.0$ . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 4:
- "Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and a set of 100 benign images.
- "Union" model refers to a model which is trained on 100 malign images and a collection of  $\sim 500$  benign images, combining the set of benign images used to train the ensemble. These models are trained for 15,000 iterations with  $\alpha = 0.02$  for the  $BCE_{\alpha}$  loss.
For these non-ensemble models, we use  $\omega = 10.0$ , which is  $K = 5$  times the guidance weight used in the ensemble case. For censored image generation using ensemble combined with backward guidance and recurrence as discussed in Section G, we use  $\omega = 2.0$ , learning late 0.002,  $B = 5$ , and  $R = 4$ .
K.5 Censored generation samples
Figure 19 shows uncensored, baseline generation. Figures 20-31 present a total of 1,000 images sampled with censored generation, 500 generated by ensemble reward models without backward guidance and recurrence and 500 with backward guidance and recurrence.
Figure 19: 96 uncensored baseline image samples. Malign images are labeled with red borders and positioned at the beginning for visual clarity.
Figure 20: First set (1–96) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely “broken” compared to the malign images of the uncensored generation.
Figure 21: Second set (97-192) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 22: Third set (193-288) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 23: Fourth set (289-384) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 24: Fifth set (385-480) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 25: Sixth set (481-500) of images among the 500 non-curated censored generation samples with a reward model ensemble and without backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 26: First set (1-96) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 27: Second set (97-192) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 28: Third set (193-288) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 29: Fourth set (289-384) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 30: Fifth set (385-480) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
Figure 31: Sixth set (481-500) of images among the 500 non-curated censored generation samples with a reward model ensemble and with backward guidance and recurrence. Malign images are labeled with red borders and positioned at the beginning for visual clarity. Qualitatively and subjectively speaking, we observe that censoring makes the malign images less severely "broken" compared to the malign images of the uncensored generation.
L Stable Diffusion: Experiment details and image samples
L.1 Pre-trained diffusion model
We use the pretrained Stable Diffusion $^{13}$ , version 1.4. We generate images using the default setting, which uses image size of  $512 \times 512$  and 50 DDIM [41] steps.
L.2 Reward model training
We utilize a ResNet18 architecture for the reward model, using the pre-trained weights available in torchvision.models' "DEFAULTS" setting $^{14}$ , which is pre-trained on the ImageNet1k [10] dataset. We replace the final layer with a randomly initialized fully connected layer with a one-dimensional output. We train all layers of the reward model using the human feedback dataset of 200 images (100 malign, 100 benign) without data augmentation. We use  $BCE_{\alpha}$  in (4) as the training loss with  $\alpha = 0.1$ . The models are trained for 10,000 iterations using AdamW optimizer [27] with learning rate  $10^{-4}$ , weight decay 0.05, and batch size 64.
L.3 Sampling and ablation study
For sampling via reward ensemble without backward guidance and recurrence, we choose  $\omega = 4.0$ . We compare the censoring performance of a reward model ensemble with two non-ensemble reward models called "Single" and "Union" in Figure 5:
- "Single" model refers to one of the five reward models for the ensemble method, which is trained on randomly selected 100 malign images, and a set of 100 benign images.
- "Union" model refers to a model which is trained on 100 malign images and a collection of  $\sim 500$  benign images, combining the set of benign images used to train the ensemble. These models are trained for 30,000 iterations with  $\alpha = 0.02$  for the  $BCE_{\alpha}$  loss.
For these non-ensemble models, we use  $\omega = 20.0$ , which is  $K = 5$  times the guidance weight used in the ensemble case. For censored image generation using ensemble combined recurrence as discussed in Section G, we use  $\omega = 4.0$  and  $R = 4$ .
L.4 Censored generation samples
Figure 32 shows uncensored, baseline generation. Figures 33 and 34 respectively present images sampled with censored generation without and with recurrence.
<sup>13</sup>https://github.com/CompVis/stable-diffusion
$^{14}$ https://pytorch.org/vision/main/models/generated/torchvision.models.resnet18
Figure 32: Uncensored baseline image samples. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
Figure 33: Non-curated censored generation samples without backward guidance and recurrence. Reward model ensemble is trained on 100 malign images. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
Figure 34: Non-curated censored generation samples with recurrence. Reward model ensemble is trained on 100 malign images. Malign images are labeled with red borders, and images involving undressed bodies are partially blackened out.
M Transfer learning ablation
To demonstrate the necessity of transfer learning for relatively more complex tasks, we compare it with training reward model from scratch. We consider the LSUN bedroom task of Section 5.4. We randomly initialize the weights of time-dependent reward model with half-UNet architecture and train it for 40,000 iterations with batch size 128. We use the training loss  $BCE_{\alpha}$  with  $\alpha = 0.1$  and use the guidance weight of  $\omega = 10.0$  for sampling.
We observe that censoring fails without transfer learning, despite our best efforts to tune the parameters. The reward model successfully interpolates the training data, but its classification performance on the test dataset (which we create separately using additional human feedback data) is poor: it shows  $70.63\%$  and  $43.23\%$  accuracy respectively for malign and benign test images. If we nevertheless proceed to perform censored generation using this reward model (trained without transfer learning), the proportion of malign images is  $15.68\% \pm 5.25\%$  (measured using 500 generated images across 5 independent trials). This is no better than  $12.6\%$  of the baseline model without censoring.
N Using malign images from secondary source
In this section, we demonstrate the effectiveness of our framework even with malign images from a secondary source, instead of model-generated images. This strategy may be useful, e.g., in cases where the baseline model rarely generates malign images (making it difficult to collect a sufficient number of them) but a user still wishes to impose hard restriction of not generating certain images.
We consider the setup of Section 5.2, and repeat the similar procedure except that we use malign images manually crawled from the Shutterstock webpage<sup>15</sup>; we utilize Microsoft Windows Snipping Tool to take screenshots of "church", "gothic church" and "cathedral" images with clearly visible watermarks, resize them to  $256 \times 256$  resolution and apply random horizontal flip. We use a fixed set of these 30 secondary malign images (Figure 35) and 30 benign images generated by the baseline model to train each reward model. We use the same hyperparameters as in Section I for reward training and sampling, except the only difference of using the guidance weight  $\omega = 1.0$ .
Figure 35: Examples of malign (watermarked) church images, collected manually from a secondary source other than the baseline model.
Baseline
(No censoring)	Ordinary
censoring	Secondary
censoring
Malign
proportion	11.4%	0.76%	1.4%
Table 1: Performance comparison between ordinary censoring vs. secondary censoring (using secondary malign images in place of model-generated malign images). Both censoring setups use ensemble of 5 reward models and recurrence with  $R = 4$
We display the result of censoring using secondary malign images in Table 1 (labeled "secondary censoring"). We ensemble 5 reward models (trained on the same set of 30 secondary malign images) and apply recurrence with  $R = 4$ . Recall that the baseline model generates  $11.4\%$  malign images, while with secondary censoring, the proportion drops to  $1.4\%$ . This is slightly worse than  $0.76\%$  of our ordinary censoring methodology (Figure 2), but still shows that secondary malign images could be effective alternatives to model-generated images in certain setups.
15https://www.Shutterstock.com/