SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Image Complexity. Throughout our experiments, we consistently observed a higher level of test-time replication in LAION-10k models compared to Imagenette models, even when trained with the same levels of data duplication. We put forth the hypothesis that this discrepancy arises due to the inherent diversity present in the LAION-10k dataset, not only in terms of images but also in terms of image complexity. To quantify image complexity, we employ two metrics. Histogram entropy measures entropy within the distribution of pixel intensities in an image, and JPEG compressibility (at JPEG quality 90), measures the size of images after compression. We resize and crop all images to the same resolution before computing these metrics.
Figure 7 (right) presents the distributions of entropy scores for LAION-10k and Imagenette. LAION-10k exhibits a wide range of data complexity, while Imagenette, predominantly comprised of complex real-world images, is characterized by higher complexity. Motivated by this observation, we explore the relationship between similarity scores and complexity of the top training point matched to each generation of a diffusion model trained on LAION-10k without any duplication. Remarkably, we
Figure 7: Image complexity vs Memorization We compute complexity scores using histogram entropy (left) and JPEG compressibility (middle) for memorized and non-memorized subpopulations, and compare both datasets (right). Refer to Sec. 6 for details.
discover a statistically significant correlation of  $-0.32 / - 0.29$  (with p-values of  $8e - 98 / 7e - 80$ ) for the entropy and compression metrics, respectively. We provide results for other duplication settings in the supplementary material, where we also find significant correlation. We present density plots illustrating the distributions of both complexity metrics for high similarity points (with scores  $>0.6$ ) versus low similarity in Figure 7 (left, middle). The clear separation between the distributions, observed across both complexity metrics, strongly suggests that diffusion models exhibit a propensity to memorize images when they are "simple".
Takeaway: Choosing the optimal length of training, quantity of training data, and training setup is a trade-off between model quality and memorization. When images do get memorized in the absence of training data duplication, they are likely to be "simple" in structure.
7 Mitigation strategies
We have seen that model conditioning plays a major role in test-time replication, and that replication is often rare when image captions are not duplicated. Armed with this observation, we formulate strategies for mitigating data replication by randomizing conditional information. We study both training time mitigation strategies and inference time mitigation strategies. Training strategies are more effective, whereas inference-time mitigations are easily retrofitted into existing diffusion models. We experiment with randomizing the text input in the following ways:
- Multiple captions (MC). We use BLIP to make 20 captions for each image and we randomly sample all of them (plus the original) when training. This is for train time only.
- Gaussian noise (GN). Add a small amount of Gaussian noise to text embeddings.
- Random caption replacement (RC). Randomly replace the caption of an image with a random sequence of words.
- Random token replacement & addition (RT). Randomly replace tokens/words in the caption with a random word, or add a random word to the caption at a random location.
- Caption word repetition (CWR). Randomly choose a word from the given caption and insert it into a random location in the caption.
- Random numbers addition (RNA). Instead of adding a random word that might change the semantic meaning of the caption, we add a random number from the range  $\{0, 10^6\}$ .
In this section, we present the effectiveness of various mitigation strategies, summarized in Table 1. During train time, we find the multiple-caption strategy to be the most effective, which substantially increases caption diversity among duplicates. This matches our analysis in Fig. 5. At inference time strategies, we find that we can still inject a shift in the text prompt through random token addition/replacement. Such a shift again disrupts the memorized connection between caption and image. We verify based on FID and Clipscore [Hessel et al., 2021] that all mitigations have a minimal effect of model performance.
Images generated from models trained/tested with the best performing strategies can be found in Figure 8. For train mitigations we show our finetuned models, whereas for inference time we show mitigation strategies applied directly to Stable Diffusion. We show four images for each strategy, displaying the memorized image, generation from the regular model and from the mitigated model
Table 1: We present the similarity scores for models trained on LAION-10k with various train and inference time mitigation strategies discussed above. We show the results on no duplication, image duplication (ddf=5) and image & caption duplication (ddf=5) scenarios. For similarity scores, the lower the better. Best numbers with in train or test time are shown in bold. * collapses to the same setup.
Train time mitigation	Inference time mitigation
Dup style↓/Mit strat.→	None	MC	GN	RC	RT	CWR	GNI	RT	CWR	RNA
No Dupl.	0.638	0.420	0.560	0.565	0.638	0.614	0.615	0.524	0.576	0.556
Image Dupl.	0.576	0.470*	0.508	0.586	0.645	0.643	0.553	0.489	0.522	0.497
Image + Caption Dupl.	0.663	0.470*	0.598	0.580	0.669	0.644	0.639	0.555	0.602	0.568
Figure 8: We present qualitative results on the best-performing train and inference time mitigation strategies. In each column, we present a potential training image model copied the generation from, a generation, and a generation on a model trained with a mitigation strategy/ mitigation applied to the original model. We used an image & caption duplication model with ddf=5 for the train time results and Stable Diffusion V1.4 (seed=2) to evaluate the inference-time strategies.
in each column. We fix the generation seed in each column. Overall, we find these mitigations to be quite effective in reducing copying behavior, even in large modern diffusion models. We refer the reader to the appendix regarding the hyperparameters used in the generation of Tab. 1 and the captions used for creating Fig. 8.
8 Recommendations for a safer model
Why do diffusion models memorize? In this work, we find that, in addition to the widely recognized importance of de-duplicating image data, conditioning and caption diversity also play fundamental roles. Moreover, we find in Fig. 5 that memorization can decrease even when duplication increases, as long as captions are sufficiently diverse. These findings help to explain why large text-conditional models like Stable Diffusion generate copies. Finally, we collect our observations into a series of recommendations to mitigate copying.
Before training: Identify large clusters of image data with significant object overlap, as measured by specialized copy detection models such as SSCD, and collapse them as described in Sec. 5.1. Likewise, clusters of caption overlap are potentially problematic, and duplicate captions can be resampled, for example using BLIP. In some cases, these clusters may need to be hand-curated because some concepts, like famous paintings or pictures of the Eiffel Tower, are acceptable duplications that the model may copy. Another avenue is to exclude images with limited complexity, see Sec. 6.
During training: We find training with partial duplications, where captions for duplicate images are resampled, to be most effective in reducing copying behavior, see Sec. 7. It may be worthwhile to use a low bar for duplication detection, as near-duplicates do not have to be removed entirely, only their captions resampled.
After training: Finally, after training, inference-time strategies can further reduce copying behavior, see Sec. 7. Such mitigations could be provided either as a user-facing toggle, allowing users to resample an image when desired, or as rejection sampling if a generated image is detected to be close to a known cluster of duplications or a match in the training set.
While text conditioning has also been identified as a crucial factor in dataset memorization for the diffusion model in our analysis, it is important to acknowledge that other factors may also influence the outcome in complex ways. Therefore, we recommend that readers thoroughly evaluate the
effectiveness of the proposed mitigation strategies within the context of their specific use cases before deploying them in production.
Future Directions: This research represents a first investigation into the influence of captions on memorization within diffusion models. We posit that examinations on broader scales, involving larger datasets and models, will further solidify our findings. Moreover, while our adopted similarity metric serves its purpose, we anticipate and encourage future studies to refine this metric or introduce alternatives grounded in a more robust theoretical framework on memorization.
This work was made possible by the ONR MURI program, DARPA GARD (HR00112020007), the Office of Naval Research (N000142112557), and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation (IIS-2212182), and by the NSF TRAILS Institute (2229885). Vasu Singla was supported in part by the National Science Foundation under grant number IIS-2213335.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. arxiv:2202.07646[cs], February 2022. doi: 10.48550/arXiv.2202.07646. URL http://arxiv.org/abs/2202.07646.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramér, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. arxiv:2301.13188[cs], January 2023. doi: 10.48550/arXiv.2301.13188. URL http://arxiv.org/abs/2301.13188.
David Chess. Some light infringement?, December 2022. URL https://ceoln.wordpress.com/2022/12/16/some-light-infringement/.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Matthew Jagielski, Om Thakkar, Florian Tramér, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring Forgetting of Memorized Training Examples. arxiv:2207.00099[cs], June 2022. doi: 10.48550/arXiv.2207.00099. URL http://arxiv.org/abs/2207.00099.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating Concepts in Text-to-Image Diffusion Models. March 2023. URL https://arxiv.org/abs/2303.13516v2.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do Language Models Plagiarize? arxiv:2203.07618[cs], March 2022. doi: 10.48550/arXiv.2203.07618. URL http://arxiv.org/abs/2203.07618.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. February 2023. URL https://arxiv.org/abs/2302.03668v1.
Text-to-image diffusion models are prone to violating intellectual property rights and causing harm to artists, even unbeknownst to the models' users. Given the rapidly increasing popularity of these models and also the likelihood that users will continue to generate and deploy their images whether or not copying behaviors are addressed, mitigating such behavior can prevent real-world harm. We take the first step towards mitigation by building up an understanding of how and why diffusion models copy their training data. Moreover, we show how to build text-to-image diffusion models which simultaneously produce high-quality images and at the same time do not replicate their training data nearly as often as popular existing models. Another important avenue for preventing copying is constructing detection pipelines so that we can identify copies of training samples before they are deployed, as in Somepalli et al. [2022].
B User Caption Study
B.1 Stable Diffusion User Matches False Positives
Figure 9: False Positive matches for User Captions using SSCD. Images are generated using Stable Diffusion v2.1, and corresponding closest matches from LAION subset are shown together.
LAION Caption: Seamless rainbow vertical stripes color line art vector background  
User Caption: Illustration, a series of vertical stripes in different colors. by Alan Moore distorted
LAION Caption: Coffee beans - Stock Photo
User Caption: cheesy pop art of coffee beans, extreme detail
LAION Caption: mosaic color matrix squares (vector)  
User Caption: large squares of different colors
LAION Caption: File:BL2-The-Sheriff-sketches-2.jpg  
User Caption: fantasy rogue armor, all-metal, official concept art, orthographic views
In Figure 9, we show failure cases for SSCD match from the LAION subset to generated images from user captions. Several false positives are reasonable errors, caused by either high style similarity or simple texture of the generated images. A few of the errors are also caused for animated images as shown in the last row. We also observe a few cases where generated, and LAION images show significant domain shift from real images. This may be due to the training data of SSCD, which mostly consisted of real images Pizzi et al. [2022].
B.2 User and LAION Captions for Images
The captions for the generated images using user prompts, and corresponding LAION matches for Figure 1 are shown below in order -
1. User Caption - "flower made of fire, black background,art by artgerm"
LAION Caption - Colorful Cattleya Orchids" (2020) by JessicaJenney
2. User Caption - "the moon landing underwater"
LAION Caption - "H/O: Buzz Aldrin on the Moon"
Cluster 13 555 images
Figure 10: Clusters discovered from 8M subset of LAION using SSCD
3. User Caption - "rosie the riveter with a catfish head" LAION Caption - "We can do it! Canvas Print"
4. User Caption - "pubg splash art advertisement by masashi kishimoto / hiroya oku, hayao miyazaki, concept art, vibrant colors, 8 k " LAION Caption - "Minor kills brother for not letting him play PUBG game in Thane"
5. User Caption - "the super mario logo" LAION Caption - "Thumb super mario run logo"
6. User Caption - "a film still from joker ( 2019 ) " LAION Caption - "Joaquin Phoenix as THE Joker in Comic Villain Casting"
7. User Caption - "cyberpunk tufting rug" LAION Caption - "Liquid Pattern Print Design 04 Area Rug"
8. User Caption - "pubg splash art advertisement by masashi kishimoto / hiroya oku, hayao miyazaki, concept art, vibrant colors, 8 k" LAION Caption - "Minor kills brother for not letting him play PUBG game in Thane"
C LAION Clusters
In Figure 10, we show a few of the clusters discovered using our proposed approach discussed in Section 5.1. As shown several of the images share high visual similarities with each other. The number of images discovered in each cluster is also shown. Note that the number is very likely an underestimate since we only use an  $8M$  subset of the LAION dataset. A few of the clusters are also obtained due to broken links or similar placeholder images. A few clusters are also obtained for similar bar graphs or line charts. This shows LAION consists of several duplicates, which can be easily discovered using our approach.
D Extended experimental settings
Unless otherwise specified, all the models are trained for 100k iterations with a batch size of 16. We used Adam optimizer for training with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  and weight decay  $= 1e - 2$  We used one RTX-A6000 per model and it took about 24 hours to train. For inference, we used RTX-A5000 and it took approximately 5 hours to create enough generations to compute our metrics.
Figure 11: Similarity scores at different percentile cutoffs at different ddf models.Left: Full duplication Right: Partial duplication
To analyze various factors in this paper, we used approximately 5000 GPU hours for training and another 1000 GPU hours for inference.
E Discussion on similarity metric design
Occasional memorization, even in a small percentage of generations, can pose problems—precisely the pattern we've observed. To capture this behavior effectively, we measure the sim score of the top  $95\%$  of samples. Using the average or median, in contrast, could fail to detect significant memorization in a few instances. Our choice of the  $95\%$  percentile is based on observations by Somepalli et al., who identified around  $2\%$  of memorization. Adjusting for fine-tuning on a smaller dataset, we found  $95\%$  to be a suitable benchmark.
We have performed an ablation analysis on LAION-10k and full duplication across different data duplication factors and the results are presented in Fig. 11 (Left). We see 95, 97, and 99 percentiles follow similar trends. Median scores remain relatively constant, as expected, due to memorization affecting only a small percentage of images, as previously noted by Somepalli et al. Similar trends appear in partial duplication experiments, as evident in Fig. 11 (Right). Hence we proceeded with  $95\%$  value of similarity score distribution as the metric in this paper.
We present how similar or different the images look at different values of SSCD similarity scores in Fig. 12. We also present the similarity score histogram for the model trained with full duplication with  $\mathrm{df} = 10$  in Fig. 11 (Left). We also present a sample Precision-Recall curves on LAION-10k full duplication case in Fig. 11 (Right).
Figure 12: How similar the generation and the matches are at different SSCD scores.
F Why do diffusion models copy? Extended Results
In this section, we explore factors contributing to replication in diffusion models. We present additional results, including some figures from the main paper for completeness.
Model conditioning. As shown in Fig. 4 (Left), higher uniqueness in conditioning captions leads to increased similarity scores, with Blip caption conditioning yielding the highest score. In Fig. 17 (Left), FID scores for fine-tuned models follow the same trends we have seen in the main paper, with the blip caption model exhibiting the lowest FID score.
Image Duplication vs Image + Caption Duplication In Fig. 14, we analyze similarity and FID scores for models trained with varying duplication levels (ddf) on LAION-10k and Imagenette. Both datasets show increased similarity and FID scores with higher ddf. However, the impact of image duplication versus image + caption duplication differs. This disparity can be attributed to the diversity of captions used in image duplication for LAION-10k and the similarity of captions in Imagenette, potentially collapsing into image + caption duplication.
Training Regimen - Length of Training In Fig. 15, we examine similarity and FID scores during the training process on LAION-10k and Imagenette. We consider scenarios with no duplication, image duplication, and image + caption duplication. Longer training consistently leads to higher similarity and FID scores. However, image duplication consistently yields lower similarity scores compared to image + caption duplication on both datasets.
Training Regimen - Text Encoder Training In Fig. 4 (Right), we analyze the impact of text encoder training on similarity and FID scores for models trained with caption conditioning and various types of duplication on Imagenette and LAION-10k. Training the text encoder increases similarity scores while reducing FID scores in both datasets. In Fig. 17 (Right), we observe the effect of text encoder training on similarity scores for models trained with different conditioning types on Imagenette, specifically with image + caption duplication  $(\mathrm{ddf} = 5)$ , and we see similar trends as of Fig. 4 (right) even in the presence of data duplication.
Training Regimen - Image Complexity In Fig. 18, we present distributions of self-similarity, entropy, and compression metrics for LAION-10k and Imagenette. LAION-10k exhibits lower self-similarity, indicating a distinct data distribution. However, complexity distributions are slightly shifted to the left, suggesting LAION-10k is marginally simpler than Imagenette.
Furthermore, we provide comprehensive findings on the correlation between similarity scores and training data complexity metrics in Tab. 2. We extensively evaluate models trained on LAION-10k without duplication, with image duplication, and with image + caption duplication, including cases with  $\text{ddf} = 5$  and  $\text{ddf} = 20$ . Notably, we consistently observe a statistically significant correlation in all scenarios. Intriguingly, the correlation is stronger when image + caption duplication is present compared to cases with only image duplication.
Figure 14: Diffusion models trained with different amounts of duplication and the style of duplication. We show similarity scores and FID scores at different values of ddf for LAION-10k and Imagenette models.
Figure 15: Does training for longer increase memorization?
Figure 16: Effect of text encoder training on Similarity and FID scores.
Table 2: Correlation coefficients and corresponding p-values between the similarity scores and training data complexity metrics for different training scenarios.
Duplication	ddb	Compression	Entropy	p-val (Comp.)	p-val (Entr.)
No duplication	0	-0.2926589	-0.3230121	7.91E-80	8.48E-98
Image duplication	5	-0.2298073	-0.3034324	4.31E-49	5.80E-86
Image + Cap duplication	5	-0.298469	-0.3412334	4.19E-83	1.24E-109
Image duplication	20	-0.0918793	-0.1559931	5.83E-09	3.29E-23
Image + Cap duplication	20	-0.1994873	-0.2394177	3.50E-37	2.98E-53
G Misc. results.
We recreated the main paper Fig 2 using Roberta and presented it Fig. 19. The findings stay the same.
H Mitigation strategies
H.1 Training details
Except for the affect of length of training on similarity scores experiment, we trained all the diffusion models for 100k iterations, including the models trained with a "mitigation" hueristic. The following are the additional hyperparameters relevant to a given strategy.
- Multiple Captions (MC): We sample 20 captions from BLIP for all training images. For LAION-10k, an additional original caption is included, resulting in a total of 21 captions.
- Gaussian Noise (GN): We sample random multivariate Gaussian noise  $\sim \mathcal{N}(0, I)$ , multiply it by a factor of 0.1, and add it to the latent CLIP representations of the caption.
- Random Caption Replacement (RC): With a probability of 0.4, we replace the caption of an image with a random sequence of words.
- Random Token Replacement & Addition (RT): With a probability of 0.1, we either replace tokens/words in the caption with a random word or add a random word at a random location. This step is performed twice for train time experiments and four times for inference time experiments.
Figure 17: Left: FID scores of models trained on Imagenette with different types of conditioning. Right: Similarity scores of models trained on Imagenette with different conditioning and with and without text encoder training. All the models in this plot are trained with image + caption duplication with ddf=5.