SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Growthami Somepalli  $^{1}$ , Vasu Singla  $^{1}$ , Micah Goldblum  $^{2}$ , Jonas Geiping  $^{1}$ , Tom Goldstein  $^{1}$
<sup>1</sup> University of Maryland, College Park {gouthami, vsingla, jgeiping, tomg}@cs.umd.edu
$^{2}$  New York University goldblum@nyu.edu
Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set. Code is available at https://github.com/somepago/DCR.
A major hazard of diffusion models is their ability to produce images that replicate their training data, often without warning to the user [Somepalli et al., 2022, Carlini et al., 2023]. Despite their risk of breaching privacy, data ownership, and copyright laws, diffusion models have been deployed at the commercial scale by subscription-based companies like midjourney, and more recently as offerings within search engines like bing and bard. Currently, a number of ongoing lawsuits [Saveri and Butterick, 2023] are attempting to determine in what sense companies providing image generation systems can be held liable for replications of existing images.
In this work, we take a deep dive into understanding the causes of memorization for modern text-to-image diffusion models. Prior work has largely focused on the role of duplicate images in the training set. While this certainly plays a role, we find that image duplication alone cannot explain much of the replication behavior we see at test time. Our experiments reveal that text conditioning plays a major role in data replication, and in fact, test-time replication can be greatly mitigated by diversifying captions on images, even if the images themselves remain highly duplicated in the training set. Armed with these observations, we propose a number of strategies for mitigating replication by randomizing text conditioning during either train time or test time. Our observations serve as an in-depth guide for both users and builders of diffusion models concerned with copying behavior.
Memorization in generative models. Most insights on the memorization capabilities of generative models are so far empirical, as in studies by Webster et al. [2021] for GANs and a number of studies for generative language models [Carlini et al., 2022, Jagielski et al., 2022, Lee et al., 2022].
Figure 1: The first row shows images generated from real user prompts for Stable Diffusion v2.1. The second row shows images found in the LAION dataset. Please refer to Appendix for the corresponding captions.
Recently, Somepalli et al. [2022] investigated data replication behaviors in modern diffusion models, finding  $0.5 - 2\%$  of generated images to be partial object-level duplicates of training data, findings also mirrored in Carlini et al. [2023]. Yet, what mechanisms lead to memorization in diffusion models, and how they could be inhibited, remains so far uncertain aside from recent theoretical frameworks rigorously studying copyright issues for image duplication in Vyas et al. [2023].
Removing concepts from diffusion models. Mitigations deployed so far in diffusion models have focused on input filtering. For example, Stable Diffusion includes detectors that are trained to detect inappropriate generations. These detectors can also be re-purposed to prevent the generation of known copyrighted data, such as done recently in midjourney, which has banned its users from generating photography by artist Steve McCurry, due to copyright concerns [Chess, 2022]. However, such simple filters can be easily circumvented [Rando et al., 2022, Wen et al., 2023], and these band-aid solutions do not mitigate copying behavior at large. A more promising approach deletes entire concepts from the model as in Schramowski et al. [2023] and Kumari et al. [2023], yet such approaches require a list of all concepts to be erased, and are impractical for protecting datasets with billions of diverse images covering many concepts.
3 How big of a problem is data replication?
Somepalli et al. [2022] and Carlini et al. [2023] have shown that diffusion models can reproduce images from their training data, sometimes to near-perfect accuracy. However, these studies induce replication behavior by prompting the model with image captions that are directly sampled from the LAION dataset – a practice that amplifies the rate of replication for scientific purposes. We study the rate at which replication happens with real-world user-submitted prompts. This gives us a sense of the extent to which replication might be a concern for typical end-users.
We randomly sample 100K user-generated captions from the DiffusionDB [Wang et al., 2022] dataset and generate images using Stable Diffusion 2.1 [Rombach et al., 2021, Stability AI, 2022]. We use SSCD features [Pizzi et al., 2022] to search for the closest matches against these images in a subset of the training dataset. Note that SSCD was found to be one of the best metrics for detecting replication in Somepalli et al. [2022], surpassing the performance of CLIP [Radford et al., 2021]. We compare each generated image against  $\sim 400$  million images, roughly  $40\%$  of the entire dataset.
We find that  $\sim 1200$  images  $(1.2\%)$  have a similarity score above 0.5, indicating these may be duplicates. Note that this is likely an underestimation of the true rate, as our SSCD-based search method is unlikely to find all possible matches. We show several of these duplicates in Figure 1. Sometimes, the captions precisely describe the image content (for example, 'Jake Gyllenhaal'). However, we also discover instances where captions do not mention the content from the generated image. For example, in the first duplicate, the user caption is "flower made of fire, black background, art by artgerm", and the LAION caption is "Colorful Cattleya Orchids (2020) by JessicaJenney". All the user and LAION captions for Figure 1 are in Appendix B.2. Several duplicates identified by SSCD also have high similarity scores due to the simple texture of the generated image. We include these SSCD false positives and other examples in the Appendix.
A thorough study of replication behavior requires training many diffusion models. To keep costs tractable, we focus on experiments in which large pre-trained models are finetuned on smaller
Figure 2: Stable Diffusion v1.4 generates memorized images when either images or captions are duplicated. Highly replicated generations from Stable Diffusion v1.4 and duplicated images from LAION training data are labeled on the plot and shown on the left. Stable Diffusion v2.1 is trained on a de-duplicated dataset, so as expected we see the clusters with high visual image similarity vanish from the right side of the chart. Nonetheless, we still see a number of replicated generations from clusters with high caption similarity.
datasets. This process reflects the training of Stable Diffusion models, which are pre-trained on LAION and then finetuned in several stages on much smaller and more curated datasets, like the LAION Aesthetics split.
Datasets: We use Imagenette<sup>1</sup>, which consists of 10 classes from Imagenet [Deng et al., 2009] as well as two randomly sampled subsets of 10,000 and 100,000 images from LAION-2B [Schuhmann et al., 2022] for our experiments. The LAION subsets, which we denote as LAION-10k and LAION-100k, include captions, while the Imagenette data is uncaptioned. For some experiments, we use BLIP v1 [Li et al., 2022] to generate captions for images when needed. We provide details for LAION subset construction in the Appendix.
Architecture & Training: We use the StableDiffusion-v2.1 checkpoint as a starting point for all experiments. Unless otherwise noted, only the U-Net [Ronneberger et al., 2015] part of the pipeline is finetuned (the text and auto-encoder/decoder components are frozen) as in the original training run, and we finetune for 100000 iterations with a constant learning rate of  $5e - 6$  and 10000 steps of warmup. All models are trained with batch size 16 and image resolution 256. We conduct evaluations on 4000 images generated using the same conditioning used during model training. Refer to Appendix for complete details on the training process.
Metrics: We use the following metrics to study the memorization and generation quality of finetuned models. Frechet Inception Distance (FID) [Heusel et al., 2017] evaluates the quality and diversity of model outputs. FID measures the similarity between the distribution of generated images and the distribution of the training set using features extracted by an Inception-v3 network. A lower FID score indicates better image quality and diversity.
Somepalli et al. [2022] quantify copying in diffusion models using a similarity score derived from the dot product of SSCD representations [Pizzi et al., 2022] of a generated image and its top-1 match in the training data. They observe that generations with similarity scores greater than 0.5 exhibit strong visual similarities with their top-1 image and are likely to be partial object-level copies of training data.
Given a set of generated images, we define its dataset similarity score as the 95-percentile of its image-level similarity score distribution. Note that measuring average similarity scores over the whole set of generated images is uninformative, as we are only interested in the prevalence of replicated images, which is diluted by non-replicated samples. For this reason, we focus only on the similarity of the  $5\%$  of images with the highest scores. Please refer to Appendix for a longer discussion on this metric design.
5 Data Duplication is Not the Whole Story
Existing research hypothesizes that replication at inference time is mainly caused by duplicated data in the training set [Somepalli et al., 2022, Carlini et al., 2023, Webster et al., 2023]. Meanwhile,
1https://github.com/fastai/imagenette
Figure 3: How does data duplication affect memorization? All models are trained with captions. On both datasets, dataset similarity increases proportionally to duplication in training data. FID score are unaffected by light duplication, but increase on higher levels as image diversity reduces.
data replication has been observed in newer models trained on de-duplicated data sets [Nichol, 2022, Mostaque, 2022]. Our goal here is to quantify the extent to which images are duplicated in the LAION dataset, and understand how this impacts replication at inference time. We will see that data duplication plays a role in replication, but it cannot explain much of the replication behavior we see.
5.1 How Much Duplication is Present in LAION?
In order to understand how much data duplication affects Stable Diffusion, we first identify clusters of duplicated images in LAION. We use 8M images from LAION-2B for our duplication analysis. Since duplicates are often duplicated many times, even a small subset of LAION is sufficient to identify them. Note that we do not use a random subset, but instead a consecutive slice of the metadata. Since the metadata is structured to provide URLs from similar domains together, this makes us likely to find large clusters.
We use SSCD [Pizzi et al., 2022] to compute the  $8M \times 8M$  similarity matrix between all images within a slice of metadata. We threshold the similarity scores, only keeping ones above 0.7 to identify images that are nearly identical. Then, we identify each connected component in the sparse similarity matrix and interpret it as representing a duplicated image. We show several of these clusters in the Appendix. We only consider clusters with at least 250 samples to ensure the images are duplicated enough times, which leaves  $\sim 50$  clusters with a total of around 45K images.
To measure how similar captions are within a cluster, we first compute CLIP text features [Radford et al., 2021]. Then, we compute the similarity between captions using the dot product of CLIP text features multiplied by the unigram-Jaccard similarity between the captions of a cluster and finally compute the median value. We use the unigram-Jaccard similarity in addition to CLIP, since it better captures word-level semantics. We select 40 captions from each cluster and feed them to Stable Diffusion. Finally, we determine if the generated images are replicas of their source cluster again using SSCD.
Results are shown in Figure 2. We observe that Stable Diffusion v1.4 shows high pixel-level replication (reflected in SSCD scores  $>0.4$ ) for some duplicated clusters, but only when caption similarity within the cluster is also high. In contrast, since the dataset for Stable Diffusion v2.1 was de-duplicated before training, it shows nearly no test-time replication for our identified clusters [Mostaque, 2022]. Despite being de-duplicated, Stable Diffusion v2.1 still exhibits memorization when scanning through a larger portion of the dataset, as we observed previously in Figure 1, showing that replication can happen even when the duplication rate in the training set is greatly reduced. In both cases, we see that image duplication is not a good predictor of test-time replication.
5.2 Controlled Experiments with Data Duplication
We train diffusion models with various levels of data duplication factor (ddf), which represents the factor by which duplicate samples are more likely to be sampled during training. We train each model for 100k iterations and evaluate similarity and FID scores on 4000 generated samples.
Figure 3 contains results for LAION-10k and Imagenette. We observe that increased duplication in the training data tends to yield increased replication during inference. The relationship between data duplication and similarity scores is not straightforward for LAION-10k. As the duplication factor increases, similarity scores rise again until reaching a data duplication factor ddf of 10, after which they decrease. Regarding FID scores, we find that a certain level of data duplication contributes to
improving the scores for models trained on both datasets, possibly because FID is lowest when the dataset is perfectly memorized (if we infer with all training data captions).
Other Observations from the Literature. Somepalli et al. [2022] found that unconditional diffusion models can exhibit strong replication when datasets are small, despite these training sets containing no duplicated images. Clearly, replication can happen in the absence of duplication. As the training set sizes grow ( $\sim 30\mathrm{k}$ ), the replication behaviors seen in Somepalli et al. [2022] vanish, and dataset similarity drops, even when the number of epochs is kept constant. One might expect this for large enough datasets. However, models trained on the much larger LAION-2B dataset do exhibit replication, even in the case of Stable Diffusion 2.1 (Figure 1) which was trained on (at least partially) de-duplicated data. We will see below that the trend of replication in diffusion models depends strongly on additional factors, related especially to their conditioning and to dataset complexity.
Takeaway: When training a diffusion model, it is crucial to carefully manage data duplication to strike a balance between reducing memorization and ensuring high-fidelity of generated images. However, controlling duplication is not enough to prevent replication of training data.
5.3 The Effect of Model conditioning
To understand the interplay between model conditioning and replication, we consider four types of text conditioning:
- Fixed caption: All data points are assigned the same caption, An image.
- Class captions: Images are assigned a class-wise caption using the template An image of <class-name>.
- Blip/Original captions: Each point is trained on the original caption from the dataset (LAION-10k) or a new caption is generated for each image using BLIP [Li et al., 2022] (Imagenette).
- Random captions: A random sequence of 6 tokens is sampled from the vocabulary used to uniquely caption each image.
By varying caption scenarios, we transition from no diversity in captions ("fixed caption") case to completely unique captions with no meaningful overlap ("random caption") case. We again finetune on the Imagenette dataset, now using these caption types. As a baseline, we consider the pretrained Stable Diffusion model without any finetuning. Figure 4 (left) shows the dataset similarity among the baseline and models finetuned using the different caption types.
We observe that the finetuned models exhibit higher similarity scores compared to the baseline model. Furthermore, the level of model memorization is influenced by the type of text conditioning. The "fixed caption" models exhibit the lowest amount of memorization, while the "blip caption" models exhibit the highest. This indicates that the model is more likely to memorize images when the captions are more unique or have high perplexity. However, the model does not exhibit the highest level of memorization when using "random captions", meaning that captions should also be correlated with image content in order to help the model retrieve an image from its memory maximally.
Training the text encoder. So far, the text encoder has been frozen during finetuning. We can amplify the impact of conditioning on replication by training the text encoder during finetuning. In Fig. 4 (right), we observe a notable increase in similarity scores across all conditioning cases when the text encoder is trained. This increase is particularly prominent in the cases of "blip captioning" and "random captioning". These findings support our hypothesis that the model is more inclined to remember instances when the captions associated with them are highly specific or have high perplexity, or even unique, keys.
Takeaway: Caption specificity is closely tied to data duplication. Highly specific (or high perplexity) captions act like keys into the memory of the diffusion model that retrieve specific data points.
5.3.1 The Impact of Caption vs. Image Duplication
In this section, we control separately for duplication of images and duplication of their captions to better understand how the two interact.
Figure 4: Left: Diffusion models finetuned on Imagenette with different styles of conditioning. FID scores of finetuned models are as follows (in order) 40.6, 47.4, 17.74, 39.8. Right: We show the effects of training the text encoder on similarity scores with different types of conditioning. Dashed lines are 95 percentile similarity scores of training data distribution.
In the case of full duplication, both the image and its caption are replicated multiple times in the training data. On the other hand, partial duplication involves duplicating the image multiple times while using different captions for each duplicate (although the captions may be semantically similar). To study the partial duplication scenario, we generate 20 captions for each image using the BLIP model for both Imagenette and LAION-10k datasets. For the full-duplication case in the LAION-10k experiments, we keep the original caption from the dataset.
We present the results on LAION-10k and Imagenette in Figure 5. We investigate how dataset similarity changes for both full and partial image-caption duplication at varying levels of duplication. Overall, our findings demonstrate that partial duplication consistently leads to lower levels of memorization compared to full duplication scenarios.
In Figure 5 (left), we compare the similarity scores of several models: the pretrained checkpoint, a model finetuned without any data duplication, a model finetuned with full duplication  $(\mathrm{ddf} = 5)$ , and a model finetuned with partial duplication  $(\mathrm{ddf} = 5)$ . We include dashed horizontal lines representing the background self-similarity computed between the dataset and itself. In the Imagenette case, models trained without duplication and with partial duplication exhibit dataset similarity below the baseline value, indicating a lower level of memorization. In contrast, the model trained with full duplication demonstrates higher levels of memorization compared to both the baseline and other cases. In the LAION-10k experiments, the model trained without duplication surpasses the training data similarity baseline. This observation raises the question of whether the memorization is inherited from the pretrained model, considering it is also trained on the larger LAION-2B dataset. However, when we compute the similarity scores on the pretrained checkpoint, we observe significantly lower values, indicating that the observed memorization is acquired during the fine-tuning process.
In Figure 5 (middle, right), we analyze how the similarity scores and FID scores vary at different data duplication factors (ddb) for full and partial data duplication. As the ddf increases, we observe an increase in memorization for models trained with full duplication. However, for partial
Figure 5: Models trained with different levels duplication and duplication settings. Left: Dataset similarity between models trained with no duplication, with partial duplication, and full duplication. Dashed lines show dataset similarity of each training distribution. Middle, Right: Dataset similarity and FID for full duplication vs partial duplication for different data duplication factors.
Figure 6: Does training for longer increase memorization? Left, Middle: Similarity and FID scores of models trained for different training number of iterations with different types of data duplication (ddf=5). Right: Metrics on models trained with different fractions of LAION-100k subset.
duplication, dataset similarity actually decreases with increased duplication. In our previous analogy, we now have multiple captions, i.e. keys for each duplicated image, which inhibits the memorization capabilities of the model. However, this memorization reduction comes at the cost of a moderate increase in FID at higher duplication levels.
Takeaway: Compared to full duplication, partial duplication substantially mitigates copying behavior, even as duplication rates increase.
6 Effect of the Training Regimen
In this section, we examine how the training process and data complexity influence the degree of memorization.
Length of training. Training for longer results in the model seeing each data point more times, and may mimic the impacts of duplication. In Figure 6 (left), we quantify the increase in dataset similarity at different training iterations. Notably, the image-only duplication model consistently exhibits lower similarity scores across epochs, while the image & caption duplication model achieves the highest values, showing that caption diversity for each duplicated image significantly slows down memorization. We show matching FID scores in Figure 6 (middle).
Quantity of training data. Somepalli et al. [2022] demonstrate that diffusion models, specifically DDPM [Ho et al., 2020] models without text conditioning, exhibit a tendency to replicate the training data when trained on small datasets. In this study, we extend this analysis to text-conditioned models. To conduct this investigation, we now randomly sample a subset of 100,000 images from LAION-2B. We then train using different fractions of this dataset, keeping other hyperparameters constant including the number of training iterations. The results of this experiment are found in Figure 6 (right). We observe that increasing the quantity of training data generally leads to lower similarity and FID scores. This aligns with the intuition that a larger and more diverse training set allows the model to generalize better and produce more varied outputs.