SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 18: Dataset properties LAION-10k vs Imagenette. Left: Self-similarity Middle: Complexity - Entropy Right: Complexity - Compression
Figure 19: Replication of Figure 2 with Stable Diffusion v1.4 using RoBERTa for Text Similarity instead of CLIP
- Caption Word Repetition (CWR): We select a word from the given caption and insert it into a random location within the caption with a probability of 0.4. This step is performed twice for train time experiments and four times for inference time experiments.
- Random Numbers Addition (RNA): Instead of adding a random word that may alter the semantic meaning, we add a random number from the range  $0, 10^{6}$  with a probability of 0.4. This step is performed twice for train time experiments and four times for inference time experiments.
H.2 CLIP score and FID tables
In Tab. 1, we presented similarity scores for models trained/inferred with different mitigation strategies. In Tab. 4 and Tab. 3, we present the corresponding CLIP scores and FID scores. Gaussian Noise
strategy compromises on the FID score. Random Token/ Random Number addition during inference strategy compromises on CLIP score the most.
Table 3: FID scores of LAION-10k models trained with various mitigation strategies. Complementary to Tab. 1. *collapses to same case.
Train time mitigation	Inference time mitigation
Dup style↓ / Mit strat.→	None	MC	GN	RC	RT	CWR	GNI	RT	CWR	RNA
No Duplication	18.72	16.63	20.63	15.98	17.16	16.73	18.92	18.67	18.14	18.73
Image Duplication	18.44	16.22*	21.62	15.92	15.32	16.03	19.04	19.23	18.38	19.19
Image + Caption Dupl.	16.94	16.22*	18.34	15.63	16.25	16.26	17.36	17.75	16.84	17.35
Table 4: CLIP scores of LAION-10k models trained with various mitigation strategies. Complementary to Tab. 1. *collapses to the same case.
Train time mitigation	Inference time mitigation
Dup style↓ / Mit strat.→	None	MC	GN	RC	RT	CWR	GNI	RT	CWR	RNA
No Duplication	30.52	30.27	29.91	30.64	30.74	30.79	30.32	29.54	30.13	29.74
Image Duplication	30.27	30.03*	29.52	30.54	30.86	30.83	29.96	29.10	29.74	29.22
Image + Caption Dupl.	30.62	30.03*	30.30	30.57	30.76	30.79	30.42	29.69	30.13	29.74
H.3 Extended Qual results
Train time - Multiple Captions
Figure 20: Applying Multiple Captions mitigation strategy on images memorized in SD 2.1
Can we forget images memorized by SD 2.1 with MC strategy? We used different images to show qualitative results for train and test time strategies due to the initial freezing of training data in our project. The image-caption pairs necessary for evaluating test-time mitigation using SD 2.1 were discovered later on.
However, to verify if train test strategies can make models forget the memorized images, we conducted an experiment by training two models, incorporating the additional 5 image-caption pairs (from Table 5 or Fig 8, right side). The first model is trained with ddf=5 and no mitigation, while the second model utilizes ddf=5 with the MultipleCaps (MC) mitigation strategy. The "train-time" results for these 5 images are presented in ??. Encouragingly, with MC strategy, the model was able to forget 3 out of 5 images. However, the model still retained strong memorization for the prompts "Mothers influence on her young hippo" and "Classic Cars for Sale." We believe the need to combine multiple mitigations to address these remaining memorizations effectively.
Inference time results. The original and modified prompts used in creating figure Fig. 8 is shown in Tab. 5. For the generation, we used Stable Diffusion 1.4 with the generator seed=2, with default settings guidance_scale=7.5, num_inference_steps=50.
Train time results. In Fig. 21, we show extended train time qualitative results for all the mitigation strategies. We see "Multiple Captions (MC)" mitigation works all the time. In contrast, other methods might or might not always be effective.
Table 5: In first column, we show a few prompts that induce copying in SD 1.4 model. In the second column, we show the updated prompt after applying Random Token (RT) mitigation strategy. Sometime the random token is semantically related to the prompt and that might impact the Clipscore of the generation a bit more.
Original Prompts	Modified Prompts
Wall View 003
Classic Cars for Sale
Mothers influence on her young hippo
Living in the Light with Ann Graham Lotz
Hopped-Up Gaming: East	Wall disappointment View part senator 003 historian
assen Classic dachsh compositions:–) Cars for Sale
Mothers 45460 influence on her 44791 young 32450 50192 hippo
Living in polaris the Light atrix with Ann Graham ancy Lotz turban
ita Hopped-Up Gaming: cricketer sati poutine East
Figure 21: Qualitative results of various train-time mitigation strategies (a) Training image (b) Generated image without mitigation (c) SD model generation (d) Gen from MC strategy model (e) Gen from GN strategy model (f) Gen from RC strategy model (g) Gen from RT strategy model (h) Gen from CWR strategy model. Please refer to Appendix H.1 for the full forms of the mitigation strategies. MC strategy is consistently better.