Understanding and Mitigating Copying in Diffusion Models
Authors: Gowthami Somepalli
Vasu Singla
Micah Goldblum
Jonas Geiping
Tom Goldstein
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion models
memorization
data replication
model safety
TL;DR: Diffusion models copy due to factors beyond data duplication; like text conditioning. We also propose mitigation strategies to help counter memorization.

Abstract: Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models.  While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set. Code is available at https://github.com/somepago/DCR.

Understanding and Mitigating Copying in Diffusion Models
Gowthami Somepalli |, Vasu Singla Micah Goldblum 2, Jonas Geiping Tom Goldstein
University of Maryland, College Park
New York University
fgowthami, vsingla, jgeiping, tomg/0ca. umd edu
goldblumenyu edu
Abstract
Images generated by diffuuson models like Stable Diffusion increasingly widespread. Recent works and even lawsuits have shown that these models are prone replicating their training data, unbeknownst to the In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set responsible for content replication at inference time, we observe that the text conditioning of the model plays similarly important role. In fact, we in our experiments that data replication often does not happen for unconditional models, while is common in the text-conditional case. Motivated by our findings, techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set. Code available at https:: h/opss://// /ttgttgttb../...///g com/ httr//smmpaaoo////
then propose several
Introduction
major hazard of diffusion models is their ability to produce images that replicate their training data, often without warning to the user [Somepalli et al., 2022, Carlini et al., 2023]. Despite their risk of breaching privacy, data ownership, and copyright laws, diffusion models have been deployed the commercial scale by subscription-based companies like midjourney, and more recently offerings within search engines like bing and bard. Currently, a number of ongoing lawsuits [Saveri and Butterick, 2023] are attempting to determine in what sense companies providing image generation systems can be held liable for replications of existing images.
In this work, we take a deep dive into understanding the causes of memorization for modern text-to- image diffusion models. Prior work has largely focused on the role of duplicate images in the training set. While this certainly plays role, we find that image duplication alone cannot explain much of the replication behavior we see test time. Our experiments reveal that text conditioning plays major role in data replication, and in fact, test-time replication can be greatly mitigated by diversifying captions on images, even the images themselves remain highly duplicated in the training set. Armed with these observations, text conditioning during either train time or test time. Our observations serve as an in-depth guide for both users and builders of diffusion models concerned with copying behavior.
propose number of strategies for mitigating replication by randomizing
2 Related work
Memorization in generative models. Most insights on the memorization capabilities of generative models are so far empirical, as in studies by Webster et al. [2021] for GANs and a number of studies for generative language models [Carlini al., 2022, Jagielski al., 2022, Lee et al., 2022].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Wo Caat!!
We Can Do h!
Figure 1: The first row shows images generated from real user prompts for Stable Diffusion v2.1. The second shows images found in the LAION dataset. Please refer to Appendix for the corresponding captions.
Recently, Somepalli et al. [2022] investigated data replication behaviors in modern diffusion models, finding 0.5-2% of generated images to be partial object-level duplicates of training data, findings also mirrored in Carlini et al. [2023]. Yet, what mechanisms lead to memorization in diffusion models, and how they could be inhibited, remains so far uncertain aside from recent theoretical frameworks rigorously studying copyright issues for image duplication in Vyas et al. [2023].
Removing concepts from diffusion models. Mitigations deployed so far in diffusion models have focused on input filtering. For example, Stable Diffusion includes detectors that are trained to detect inappropriate generations. These detectors can also be re- re-purposed to prevent the generation of known copyrighted data, such as done recently midjourney, which has banned its users from generating photography by artist Steve McCurry, due to copyright concerns [Chess, 2022]. However, such simple filters can be easily circumvented [Rando et al., 2022, Wen et al., 2023], and these band-aid solutions do not mitigate copying behavior at large. A more promising approach deletes entire concepts from the model as Schramowski et al. [2023] and Kumari et al. [2023], yet such approaches require a list of all concepts to be erased, and are impractical for protecting datasets with billions of diverse images covering many concepts.
How big of a problem is data replication?
Somepalli et al. [2022] and Carlini al. [2023] have shown that diffusion models can reproduce images from their training data, sometimes to near- pear-perfect accuracy. However, these studies induce replication behavior by prompting the model with image captions that are directly sampled from the LAION dataset a practice that amplifies the rate of replication for scientific purposes. We study the rate at which replication happens with real-world user-submitted prompts. This gives us sense of the extent to which replication might be concern for typical end-users.
We randomly sample 100K user-generated captions from the DiffusionDB [Wang et al., 2022] dataset and generate images using Stable Diffusion 2.1 [Rombach et al., 2021, Stability AI, 2022]. We use SSCD features [Pizzi et al., 2022] to search for the closest matches against these images in subset of the training dataset. Note that SSCD was found to be one of the best metrics for detecting replication in Somepalli et al. [2022], surpassing the performance of CLIP [Radford al., 2021]. We compare each generated image against ~ 400 million images, roughly 40% of the entire dataset.
We find that 1200 images (1.2%) have a similarity score above 0.5, indicating these may be duplicates. Note that this is likely an underestimation the true rate, as our SSCD-based search method is unlikely find all possible matches. We show several of these duplicates in Figure 1. Sometimes, the captions precisely describe the image content (for example, 'Jake Gyllenhaal " However, also discover instances where captions do not mention the content from the generated image. For example, in the first duplicate, the user caption is "flower made of fire, black background, art by artgerm", and the LAION caption is "Colorful Cattleya Orchids (2020) by JessicaJenney". All the user and LAION captions for Figure are in Appendix B.2. Several duplicates identified by SSCD also have high similarity scores due the simple texture of the generated image. We include these SSCD false positives and other examples in the Appendix.
4 Experimental Setup
thorough study of replication behavior requires training many diffusion models. To keep costs tractable, we focus on experiments in which large pre-trained models are finetuned on smaller

---

Real
Generated
Generation Dupication w.r. Clustets
Generation Duplication wat Clusters
Text Similarit (TTmin, Train)
Text Similrriy (Thmin, Train)
1.0
2
0.8
0.8
0.6
0.6
0.2
0.2
MOR HURRED Up GRMI GRINE NORDED UP BRMING
0.0
0.0
3
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
EERS
Visual Similurity (Train, Gon)
Vieual Simillaity (Train, Gen)
EAST
Stable Diffusion v1.4
Stable Diffusion v2.1
Figure 2: Stable Diffusion v1.4 1.4 generates memorized images when either images or captions are duplicated. Highly replicated generations from Stable Diffusion v1.4 and duplicated images from LAION training data are labeled on the plot and shown on the left. Stable Diffusion v2.1 trained on a de-duplicated datasct, so as expected we see the clusters with high visual image similarity vanish from the right side of the chart. Nonetheless, we still see a number of replicated generations from clusters with high caption similarity.
datasets. This process reflects the training of Stable Diffusion models, which are pre-trained on LAION and then finetuned in several stages on much smaller and more curated datasets, like the LAION Aesthetics split.
Datasets: We use Imagenette', which consists of 10 classes from Imagenet [Deng et al., 2009] as well as two randomly sampled subsets of 10, 000 and 100, 000 images from LAION-21 2B [Schuhmann et al., 2022] for our experiments. The LAION subsets, which we denote as LAION-1OK and LAION- 100k, include captions, while the Imagenette data is uncaptioned. For some experiments, we use BLIP [Li al., 2022] to generate captions for images when needed. We provide details for LAION subset construction in the Appendix.
We use the StableDiffusion-22. checkpoint as a starting point for all experiments. Unless otherwise noted, only the U-Net [Ronneberger et al., 2015] part of the pipeline is finetuned (the text and auto-encoderrddecodee components are frozen) as in the original training run, and we finetune for 100000 iterations with a constant learning rate of warmup. All models are trained with batch size 16 and image resolution 256. We conduct evaluations on 4000 images generated using the same conditioning used during model training. Refer to Appendix for complete details on the training process.
Architecture & Training:
6 and 10000 steps of
We use the following metrics to study the memorization and generation quality of finetuned models. Frechet Inception Distance (FID) [Heusel et al., 2017] evaluates the quality and diversity of model outputs. FID measures the similarity between the distribution of generated images and the distribution of the training set using features extracted by an Inception-v3 network. A lower FID score indicates better image quality and diversity.
Metrics:
Somepalli et al. [2022] quantify copying in diffusion models using a similarity score derived from the dot product of SSCD representations [Pizzi et al., 2022] of a generated image and its top-1 match in the training data. They observe that generations with similarity scores greater than 0.5 exhibit strong visual similarities with their top-1 image and are likely to be partial object-level copies of training data,
Given a set of generated images, we define its dataset similarity score the perceneeet of its image-leve similarity score distribution. Note that measuring average similarity scores over the whole set of generated images is uninformative, as we only interested in the prevalence of replicated images, which is diluted by non-replicated samples. For this reason, we focus only on the similarity of the 5% of images with the highest scores. Please refer Appendix for longer discussion on this metric design.
5 Data Duplication is Not the Whole Story
Existing research hypothesizes that replication at inference time is mainly caused by duplicated data in the training set [Somepalli et al., 2022, Carlini et al., 2023, Webster et al., 2023]. Meanwhile.
3

---

LAION-10k
Imagenette
0.70
Similairty sivrrs
Similairty aixrrs
0.7
20
0.68
18
FID +
00.6-
19
0.06
17
0.5-
18
0.64
0
10
90
10
15
20
Data dipplication factor
Data duplicctioo factor
Figure 3: How does data duplication affect memorization? All models are trained with captions. On both datasets, dataset similarity increases proportionally to duplication in training data. FID score are unaffected by light duplication, but increase on higher levels as image diversity reduces.
data replication has been observed in newer models trained on de-duplicated data sets [Nichol, 2022, Mostaque, 2022]. Our goal here is to quantify the extent to which images are duplicated in the LAION dataset, and understand how this impacts replication at inference time. We will see that data duplication plays a role in replication, but it cannot explain much of the replication behavior we see.
5.1 How Much Duplication is Present in LAION?
In order understand how much data duplication affects Stable Diffusion, we first identify clusters of duplicated images in LAION. We use 8M images from LAION-2B for our duplication analysis. Since duplicates are often duplicated many times, even a small subset of L.AION is sufficient to identify them. Note that we do not use a random subset, but instead consecutive slice of the metadata. Since the metadata is structured to provide URLs from similar domains together, this makes us likely to find large clusters.
We use SSCD [Pizzi al., 2022] to compute the 8M 8M similarity matrix between all images within a slice of metadata. We threshold the similarity scores, only keeping ones above 0.7 to identify images that are nearly identical. Then, we identify each connected component in the sparse similarity matrix and interpret it as representing a duplicated image. We show several of these clusters in the Appendix. We only consider clusters with at least 250 samples to ensure the images are duplicated enough times, which leaves ~ 50 clusters with a total of around 45K images.
To measure how similar captions within cluster, we first compute CLIP text features [Radford et al., 2021]. Then, we compute the similarity between captions using the dot product of CLIP text features multiplied by the unigram-Jaccant similarity between the captions of cluster and finally com- pute the median value. We use the unigram-Jaccan tures word-level semanties. We select 40 captions from each cluster and feed them Stable Diffusion. Finally, we determine if the generated images are replicas of their source cluster again using SSCD.
similarity in addition to CLIP, since it better cap-
Results are shown in Figure 2. We observe that Stable Diffusion v1.4 shows high pixel-level replication (reflected in SSCD scores > 0.4) for some duplicated clusters, but only when caption similarity within the cluster is also high. In contrast, since the dataset for Stable Diffusion v2.1 was de-duplicated before training, it shows nearly no test-time replication for our identified clusters [Mostaque, 2022]. Despite being de-duplicated, Stable Diffusion v2.1 still exhibits memorization when scanning through a larger portion of the dataset, as we observed previously in Figure 1, showing that replication can happen even when the duplication rate in the training set is greatly reduced. In both cases, we see that image duplication is not a good predictor of test-time replication.
5.2 Controlled Experiments with Data Duplication
We train diffusion models with various levels of data duplication factor (ddf), which repre- sents the factor by which duplicate samples are more likely to be sampled during training. We train each model for 100k iterations and evaluate similarity and FID scores on 4000 generated samples.
Figure 3 contains results for LAION- 10k and Imagenette. We observe that increased duplication in the training data tends to yield increased replication during inference. The relationship between data duplication and similarity scores is not straightforward for LAION-10K. As the duplication factor increases, similarity scores rise again until reaching data duplication factor ddf of 10, after which they decrease. Regarding FID scores, we find that a certain level of data duplication contributes to

---

improving the scores for models trained on both datasets, possibly because FID is lowest when the dataset is perfectly memorized (if we infer with all training data captions).
Other Observations from the Literature. Somepalli al. [2022] found that unconditional dif- fusion models can exhibit strong replication when datasets are small, despite these training sets containing no duplicated images. Clearly, replication can happen in the absence of duplicatioo. As the training set sizes grow (~ 30k), the replication behaviors seen in Somepalli et al. [2022] vanish, and dataset similarity drops, even when the number of epochs is kept constant. One might expect this for large enough datasets. However, models trained on the much larger LAION-2B dataset do exhibit replication, even in the case Stable Diffusion 2.1 (Figure 1) which was trained on (at least partially) de- duplicated data. We will see below that the trend of replication in diffusion models depends strongly on additional factors, related especially to their conditioning and to dataset complexity.
Takeaway: When training diffusion model, it is crucial to carefully manage data duplication to strike balance between reducing memorization and ensuring high-fidelity of generated images. However, controlling duplication is not enough to prevent replication of training data.
5.3 The Effect of Model conditioning
To understand the interplay between model conditioning and replication, we consider four types of text conditioning:
Fixed caption: All data points are assigned the same caption, An image.
Class captions: Images are assigned a class-wise caption using the template An image of <class-namee
Blip/Original captions: Each point is trained the original caption from the dataset (LAION- 10k) ) or new caption is generated for each image using BLIP [Li et al., 2022] (Imagenette).
Random captions: random sequence of tokens is sampled from the vocabulary used to uniquely caption each image.
By varying caption scenarios, we transition from no diversity in captions ("fixed caption") case to completely unique captions with meaningful overlap ("random caption") case. We again finetune on the Imagenette dataset, now using these caption types. As baseline, we consider the pretrained Stable Diffusion model without any finetuning. Figure (left) shows the dataset similarity among the baseline and models finetuned using the different caption types.
We observe that the finetuned models exhibit higher similarity scores compared to the baseline model. Furthermore, the level of model memorization is influenced by the type of text conditioning. The "fixed caption" models exhibit the lowest amount of memorization, while the "blip caption" models exhibit the highest. This indicates that the model is more likely to memorize images when the captions more unique or have high perplexity. However, the model does not exhibit the highest level of memorization when using "random captions", meaning that captions should also be correlated with image content in order help the model retrieve image from its memory maximally.
Training the text encoder. So far, the text encoder has been frozen during finetuning. We can amplify the impact of conditioning on replication by training the text encoder during finetuning. In Fig. 4 (right), we observe a notable increase in similarity scores across all conditioning cases when the text encoder is trained. This increase is particularly prominent in the cases of "blip captioning" and "random captioning". These findings support our hypothesis that the model is more inclined to remember instances when the captions associated with them are highly specific or have high perplexity, or even unique, keys.
Caption specificity is closely tied to data duplication. Highly specific (or high perplexity) captions act like keys into the memory of the diffusion model that retrieve specific data points.
Takeaway:
5.3.1
The Impact of Caption vs. Image Duplication
In this section, we control separately for duplication of images and duplication of their captions to better understand how the two interact.
5

---

Bood capt
hlip/og caps
prstninndd SD
uinttured muddel
cluss capt
zandom cupt
Similairty scores
0.4
o
Similairty seoress
0.1-
0.3
0.00000
1.00000
0.2
0.2-
0.1
Roood cupt class capt
blip capt random ragt
Falee
Trme
Text conditioning style (Inagenette))
Text. emcodes trained (Inageettte)
Figure 4: Left: Diffusion models finetuned Imagenette with different styles of conditioning. FID scores of finetuned models are as follows (in order) 40.6, 47.4, 17.74, 39.8. Right: We show the effects of training the text encoder on similarity scores with different types of conditioning. Dashed lines are 95 percentile similarity scores of training data distribution.
In the case of full duplication, both the image and its caption are replicated multiple times in the training data. On the other hand, partial duplication involves duplicating the image multiple times while using different captions for each (uplicateealthhough the captions may be semantically similar). To study the partial duplication scenario, we generate 20 captions for each image using the BLIP model for both Imagenette and LAION-10K datasets. For the full-duplication case in the LAION-1OK experiments, keep the original caption from the dataset.
We present the results on LAION-1OK and Imagenette in Figure 5. similarity changes for both full and partial image-caption duplication at varying levels of duplication. Overall, our findings demonstrate that partial duplication consistently leads to lower levels of memorization compared to full duplication scenarios.
We investigate how dataset
In Figure 5 (left), we compare the similarity scores of several models: the pretrained checkpoint, model finetuned without any data duplication, a model finetuned with full duplication (ddf=5), and model finetuned with partial duplication (ddf=5). We include dashed horizontal lines representing the background self-similarity computed between the dataset and itself. In the Imagenette case, models trained without duplication and with partial duplication exhibit dataset similarity below the baseline value, indicating lower level of memorization. In contrast, the model trained with full duplication demonstrates higher levels memorization compared to both the baseline and other cases. In the LAION-10K experiments, the model trained without duplication surpasses the training data similarity baseline. This observation raises the question of whether the memorization is inherited from the pretrained model, considering is also trained the larger LAION-2D dataset. However, when compute the similarity scores on the pretrained checkpoint, we observe significantly lower values, indicating that the observed memorization is acquired during the fine-tuning process.
In Figure (middle, right), we analyze how the similarity scores and FID scores vary at different data duplicatin factors (ddf) for full and partial data duplication. As the ddf increases, we observe an increase memorization for models trained with full duplication. However, for partial
pretrained SD
dupl.
image
capt. dupl.
image dupl.
LAION-OO
LAION-10K
0.6
0,70-
Similairty antrrs
Inagengtte
Similaitty sccuree
LAION. DB
0.65-
0.2
0.55
LAIONNICC
Imagento
10 15
20
0
10
175
70
Dataset
Data duplication factor
Date duplication factor
Figure 5: Models trained with different levels duplication and duplication settings. Left: Dataset similarity between models trained with duplication, with partial duplication, and full duplication. Dashed lines show dataset similarity of each training distribution. Middle, Right: Dataset similarity and FID for full duplication partial duplication for different data duplication factors.

---

no dupl.
ing+eap dupl.
img dupl.
LA1ON-10k
LAION-1Ok
LAION-100K
0.7
0.65
Sinilaiity scores
3.6
18
15.3
0.35
Similairty
0.3-
15.0
0.30
344
0.45
500000
100000
50060
100000
50
Leagth of training (in iters)
Length of training [in itees)
Amnout of training data (x 10")
Figure 6: Does training for longer increase memorization? Left, Middle: Similarity and FID scores of models trained for different training number of iterations with different types of data duplication (ddf=5). Right: Metries models trained with different fractions of LATON-100K subset.
duplication, dataset similarity actually decreases with increased duplication. In our previous analogy, have multiple captions, i.e. keys for each duplicated image, which inhibits the memorization capabilities of the model. However, this memorization reduction comes at the cost of a moderate increase in FID higher duplication levels.
Takeaway: Compared to full duplication, partial duplication substantially mitigates copying behavior, even as duplication rates increase.
Effect of the Training Regimen
In this section, we examine how the training process and data complexity influence the degree of memorization.
Length of training, Training for longer results in the model seeing each data point more times, and may mimic the impacts duplication. In Figure (left), we quantify the increase in dataset similarity at different training iterations. Notably, the image-only duplication model consistently exhibits lower similarity scores across epochs, while the image & caption duplication model achieves the highest values, showing that caption diversity for each duplicated image significantly slows down memorization. We show matching FID scores Figure (middle).
Quantity of training data. Somepalli et al. [2022] demonstrate that diffusion models, specifically DDPM [Ho et al., 2020] models without text conditioning, exhibit tendency to replicate the training data when trained models. To conduct this investigation, we now randomly sample a subset of 100, 000 images from LAION-2B. We then train using different fractions of this dataset, keeping other hyperparameters constant including the number of training iterations. The results of this experiment are found in Figure 6 (right). We observe that increasing the quantity of training data generally leads to lower similarity and FID scores. This aligns with the intuition that larger and more diverse training set allows the model generalize better and produce more varied outputs.
cox-cconditioned small datasets. In this study, extend this analysis
Image Complexity. Throughout our experiments, we consistently observed higher level of test- time replication LATON-10K models compared to Imagenette models, even when trained with the same levels of data duplication. We put forth the hypothesis that this discrepancy arises due to the inherent diversity present in the LAION-IOK dataset, not only in terms of images but also in terms of image complexity. To quantify image complexity, measures entropy within the distribution of pixel intensities in an image, and JPEG compressibility (at JPEG quality 90), measures the size of images after compression. We resize and crop all images to the same resolution before computing these metrics.
employ two metrics. Histogram entropy
Figure 7 (right) presents the distributions of entropy scores for LAION-10K and Imagenette. LAION- 10k exhibits a wide range of data complexity, while Imagenette, predominantly comprised of complex real-world images, is characterized by higher complexity. Motivated by this observation, we explore the relationship between similarity scores and complexity of the top training point matched to each generation of diffusion model trained on LAION- 10k without any duplication. Remarkably,
7

---

High 8im
High Sim
1.5
Imagenttte LAION-TO
3.5
Low Sim
0.300
Low Sim
Densiny
Density
Density
10--
3.4-
3.060
0.7-
3.055
20
30
Complexity (Entropy)
Complexity (Compression)
Complexity (Entropy)
Figure Image complexity Memorization We compute complexity using histogram entropy (left) JPEG compressibility (middle) memorized and non memooizee subpopulations. and compare both datasets (right). Refer to Sec. 6 for details.
discover a statistically significant correlation of -0.32/ 0.29 (with p-values of 8e 98/7e 80) for the entropy and compression metrics, respectively. We provide results for other duplication settings in the supplementary material, where we also find significant correlation. We present density plots illustrating the distributions of both complexity metrics for high similarity points (with scores 0.6) versus low similarity in Figure 7 (left, middle). The clear separation between the distributions, observed across both complexity metrics, strongly suggests that diffusion models exhibit propensity to memorize images when they are "simple".
Choosing the optimal length of training, quantity of training data, and training setup is trade-off between model quality and memorization. When images do get memorized in the absence of training data duplication, they are likely to be "simple" in structure.
Takeaway:
7 Mitigation strategies
We have seen that model conditioning plays a major role test-time replication, and that replication is often rare when image captions are not duplicated. Armed with this observation, we formulate strategies for mitigating data replication by randomizing conditional information. training time mitigation strategies and inference time mitigation strategies. Training strategies more effective, whereas inference-time mitigations are easily retrofitted into existing diffusion models. We experiment with randomizing the text input in the following ways:
study both
Multiple captions (MC). We use BLIP to make 20 captions for each image and we randomly sample all of them (plus the original) when training. This for train time only.
Gaussian noise (GN). Add small amount of Gaussian noise text embeddings.
Random caption replacement (RC). Randomly replace the caption of an image with random sequence of words.
Random token replacement & addition (RT). Randomly replace tokens/words in the caption with a random word, or add a random word to the caption at a random location.
Caption word repetition (CWR). Randomly choose word from the given caption and insert it into random location in the caption.
Random numbers addition (RNA). Instead of adding a random word that might change the semantic meaning of the caption, we add a random number from the range {0, 10%).
In this section, we present the effectiveness of various mitigation strategies, summarized in Table 1. During train time, we find the multiple-caption strategy to be the most effective, which substantially increases caption diversity among duplicates. This matches time strategies, we find that we can still inject a shift in the text prompt through random token additiion/replacmmnnt Such shift again disrupts the memorized connection between caption and image. We verify based on FID and Clipscore [Hessel et al., 2021] that all mitigations have minimal effect of model performance.
analysis in Fig. 5. At inference
Images generated from models trained/tested with the best performing strategies can be found in Figure 8. For train mitigations we show our finetuned models, whereas for inference time we show mitigation strategies applied directly to Stable Diffusion. We show four images for each strategy, displaying the memorized image, generation from the regular model and from the mitigated model

---

Table We present the similarity scores for models trained on LAION- 10k with various train inference time mitigation strategies discussed above. We show the results on no duplication, image duplication (ddf=5) and image & caption duplication (ddf=5) scenarios. For similarity scores, the lower the hetter. Best numbers with in train test. time are shown in bold. collapses to the same setup.
Train time mitigation
Inference time mitigation
Dup style/ / Mit strat.>>
None
MC
GN
CWR
GNI
RT
CWR
RNA
No Dupl.
0.638
0.420
0.560
0.565
0.638
0.614
0.615
0.524
0.576
0.556
Image Dupl.
0.576
0.470*
0.508
0.586
0.645
0.643
0.553
0.489
0.522
0.497
Image
Caption Dupl.
0.663
0.470
0.598
0,580
0.669
0.644
0.639
0.555
0.602
0.568
Train time
Multiple Captions
Inference time
Random token / number addn.
NOR IDDARO UP BRMING
Train
data
FAST NOR EAST
Gen. w. mitigation Genertion
OOPEEO UP DAMAN
Figure We present qualitative results on best-performing train and inference time mitigation strategies. In each column, we present a potential training image model copied the generation from, a generation, and a generation on model trained with mitigation strategy/ mitigation applied to the original model. We used an image & caption duplication model with ddf-5 for the train time results and Stable Diffusion V1.4 (seed-2) evaluate the inference- time strategies.
in each column. We fix the generation seed in each column. Overall, we find these mitigations to be quite effective in reducing copying behavior, even in large modern diffusion models. We refer the reader to the appendix regarding the hyperparameters used in the generation of Tab. and the captions used for creating Fig. 8.
Recommendations for a safer model
Why do diffusion models memorize? In this work, we find that, addition to the widely recognized importance of de-duplicating image data, conditioning and caption diversity also play fundamental roles. Moreover, we find in Fig. 5 that memorization can decrease even when duplication increases. as long as captions are sufficiently diverse. These findings help to explain why large text-conditional models like Stable Diffusion generate copies. Finally, we collect our observations into a series of recommendations to mitigate copying.
Before training: Identify large clusters of image data with significant object overlap, as measured by specialized copy detection models such SSCD, and collapse them as described in Sec. 5.1. Likewise, clusters of caption overlap are potentially problematic, and duplicate captions can be resampled, for example using BLIP. In some cases, these clusters may need to be hand-curated because some concepts, like famous paintings or pictures of the Eiffel Tower, are acceptable duplications that the model may copy. Another avenue is to exclude images with limited complexity, see Sec. 6.
During training: We find training with partial duplications, where captions for duplicate images are resampled, to most effective reducing copying behavior, see Sec. 7. It may be worthwhile to use low bar for duplication detection, as near-duplicates do not have to be removed entirely, only their captions resampled.
After training: Finally, after training, inference-time strategies can further reduce copying behavior, see Sec. 7. Such mitigations could be provided either as a user-facing toggle, allowing users to resample an image when desired, as rejection sampling if generated image is detected to be close to known cluster of duplications or a match in the training set.
While text conditioning has also been identified as a crucial factor in dataset memorization for the diffusion model in our analysis, it important to acknowledge that other factors may also influence the outcome in complex ways. Therefore, recommend that readers thoroughly evaluate the

---

effectiveness of the proposed mitigation strategies within the context of their specific use cases before deploying them production.
This research represents a first investigation into the influence of captions on memorization within diffusion models. We posit that examinations on broader scales, involving larger datasets and models, will further solidify our findings. Moreover, while our adopted similarity metric serves its purpose, we anticipate and encourage future studies to refine this metric or introduce alternatives grounded in a more robust theoretical framework on memorization.
Future Directions:
Acknowledgements
This work was made possible by the ONR MURI program, DARPA GARD (HR001 12020007), the Office of Naval Research (N000142112557) and the AFOSR MURI program. Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. Further support was provided by the National Science Foundation TRAILS Institute (2229885). Vasu Singla was supported in part by the National Science Foundation under grant number IIS-2213335.
and by the NSF
References
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. arxiv:2202. (122020007666sss/ February 2022.
doi: 10.485500arxxi,2222..76666 URL http: http::////xivvvoo harg/abs/2200 07646.
Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag. Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. arxiv:2301. 13 /8]/ January 2023.
doi: 10.488550/r21...301111111 URL http://arxiv.
org/abs 2301 13188.
David Chess. Some light infringement?, December 2022. URL https:////c wordpress.com with://mmeemittt----eeeee
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint arXiv:2104.08716 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Car- lini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring Forgetting of Memorized Training Examples. arxiv:2207.00 00099(cs). June 2022. doi: axx0v2200.00099/ccss
(0.488550/rxxvv2200 www.:awwwww.222.....m .0099.. URL http: / aaaxiiiivvv..... www.aos/////.oooooootoo.ooo..aet 2207 .00999.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating Concepts in text-to-Image Diffusion Models. March 2023. URL https: / arxxiv org/
abs/2303. 13516v2.
Jooyoung Lec, Thai Le, Jinghui Chen, and Dongwon Lee. Do Language Models Plagiarize? trxiv:2203.07618fcr March 2022. doi: 10.445550/axii.2203.776661 http://a org/ abs/ abs/2203.07618.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vasion-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900 PMLR, 2022.
10

---

Emad Mostaque. Some more from first batch. Lots of optimisation to do, took about an hour of playing about. Prompts here: https://tt Gaooo negative important for 2.0 given how we flatten distribution of latents with dedupe etc Embeddings will make it easier out of the box hutps:/t.coo///xhkk1 v&J, November 2022. https/ts:://tatt
wwt.coo///////. com/Blostaqueestaaus
1596907328548139008.
Alex Nichol. Dall 2 pre-training mitigationss 2022. URL httts:://ppeaai
com/research/ https///pppna...oorrreeaaaa
dall-e-2-pre- teappre-triiin----ee mntiiattions Accessed: 2023-05-05.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A Self-Supervised Descriptor for Image Copy Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 14532-14542, 2022. URL httts://openaccesst thecvf. com/ coontent/CVPR202200 /aiiziii//iiiAA Self www.-Sueevieeeeeerrppeeeeee Strprctpptoo_ooo___nee__o___oop
html.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Javier Rando, Daniel Palcka, David Lindner, Lennart Heim, and Florian Tramèr. Red-Teaming the Stable Diffusion Safety Filter. wwwp::22/000461000 October 2022. doi: 10.48550//rXiv.2210. 04610. URL http: http://axvvvv.oo www.orgg//b///eeeeeeoe 04610.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical
image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
Joseph Saveri and Matthew Butterick, Stable Diffusion Litigation, 2023. URL https:// stablediffusiomlitieee com//
Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. 1ttp::2211.05005//1 April 2023, doi: 10.48550 1.0550. URL http: org/ abs//211. 05105.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.084002 2022.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models. 2rxiv:2212..3886(cs)) December 2022. doi: 00.48550/rXii.2222288888 URL http:://taa:: wwaerivv..o//aaaee org/abs/////eeeoeeoo.../ee /212. 2212.03860. 03860.
Stability AI. Stable Diffusion v2.1 and DreamStudio Update, December 2022. URL https:
//stability att/blog//tabeedffffttie htt.ss/taeddffffiinno/// 1-w.:oo22--reeeee-----eeeee 2022.
Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable Copyright Protection for Generative Models. 1omi:22302.18870/cc stat], February 2023. doi: 4485500/aXXi.2222210997777 URL http://
arxiv. srriv.oorg/bb//20000 10800
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models.
arXiv:2210. 14896 [cs], 2022. URL https: hatps:///arxii www.:og.g///a..oe....oo attpaabb......//////// 14896.
Ryan Webster, Julien Rabin, Loie Simon, and Frederic Jurie. This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces. axxiv:2107.060//8ccs// July 2021. doi: 000448500/rrxi.2107..0
URL http: //t:://rriii /ttp//avxv...aaaaaaa//aaa wrgg/oa//1/0..../// 200b//220700008.
Ryan Webster, Julien Rabin, Loie Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint XIX::2333.127333 2023.
11

---

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Basee Discrete Optimization for Prompt Tuning and Discovery. February 2023. https: http///xxvvvvr/// wttp///abb//...0006....
12

---

A Broader Impact
Texl-to-image diffusion models are prone to violating intellectual property rights and causing harm artists, even unbeknownst the models' users. Given the rapidly increasing popularity of these models and also the likelihood that users will continue to generate and deploy their images whether not copying behaviors addressed, mitigating such behavior can prevent real-world harm. We take the first step towards mitigation by building up an understanding of how and why diffusion models copy their training data. Moreover, we show how to build text-to-image diffusion models which simultancously produce high-quality images and at the same time do not replicate their training data nearly as often popular existing models. Another important avenue for preventing copying constructing detection pipelines so that we can identify copies of training samples before they are deployed, as Somepalli al. [2022].
B
User Caption Study
B.1
Stable Diffusion User Matches False Positives
LAION
Generated
LAION Caption: Seamless rainbow vertical stripes color line art vector background
Illustration. series of vertical stripes in different colors. by Alan Moore distorted
User Caption:
LAION Caption: Coffee beans Stock Photo
User Caption: cheesy pop coffee beans, extreme detail
mosaic color matrix squares (vector) User Caption: large squares of different colors
LAION Caption:
LAION Caption: File:BL2-The-Sherie ttee-Sheriffsskethhs--ppppp
User Caption: fantasy rogue armor, all-metal, official concept art, orthographic views
Figure 9: False Positive matches for User Captions using SSCD. Images are generated using Stable Diffusion 12. and corresponding closest matches from LAION subset shown together.
In Figure 9, we show failure cases for SSCD match from the LAION subset to generated images from user captions. Several false positives are reasonable errors, caused by either high style similarity or simple texture of the generated images. A few of the errors are also caused for animated images as shown the last row. We also observe few cases where generated, and LAION images show significant domain shift from real images. This may be due to the training data of SSCD, which mostly consisted of real images Pizzi et al. [2022].
B.2 User and LAION Captions for Images
The captions for the generated images using user prompts, and corresponding LAION matches for Figure are shown below in order-
1. User Caption "flower made of fire, black background,art by artgerm" LAION Caption "Colorulll Cattleya Orchids"""" (2020) by Jessicalenney""""
2. User Caption "the moon landing underwater
LAION Caption "H/O: Buzz Aldrin the Moon"
13

---

Cluster 555 images
Cluster 466 images
2
Cluster 25 382 images
Cluster 27 345 images
Figure 10: Clusters discovered from 8M subset of LAION using SSCD
3. User Caption "rosie the riveter with catfish head"
LAION Caption "We can do it! Canvas Print"
4. User Caption "pubg splash art advertisement by masashi kishimoto hiroya oku, hayao miyazaki, concept art, vibrant colors, 8 k
LAION Caption "Minor kills brother for not letting him play PUBG game in Thane"
5. User Caption "the super mario logo
LAION Caption "Thumb super mario run logo"
6. User Caption "a film still from joker ( (2019)"
LAION Caption "Joaquin Phoenix THE Joker in Comic Villain Casting"
7. User Caption "cyberpunk tufting rug" LAION Caption "Liquid Pattern Print Design 04 Area Rug"
8. User Caption "pubg splash art advertisement by masashi kishimoto hiroya oku, hayao miyazaki, concept art, vibrant colors, k
LAION Caption "Minor kills brother for not letting him play PUBG game in Thane"
C LAION Clusters
In Figure 10, we show few of the clusters discovered using our proposed approach discussed in Section 5.1. As shown several of the images share high visual similarities with each other. The number of images discovered in each cluster also shown. Note that the number very likely an underestimate since only use 8.M subset of the LAION dataset. A few of the clusters also obtained due to broken links or similar placeholder images. A few clusters are also obtained for similar bar graphs line charts. This shows LAION consists of several duplicates, which can be easily discovered using our approach.
D Extended experimental settings
Unless otherwise specified, all the models are trained for 100k iterations with a batch size of 16. We used Adam optimizer for training with 3 = 0.9 and Bg 0.999 and weight decay= 1e We used one -T0006000 per model and took about 24 hours train. For inference, we used RTX-A5000 and took approximately 5 hours create enough generations to compute our metrics.
14

---

Metric percentile (Img + Cap dupl.)
Metric percentile (Img dupl.)
0.8
Similairty scores
Similairty scores
59
0.6
0.6
90
95
97
0.4-
99
10
15
20
10
15
20
Data duplication factor
Data duplication factor
Figure 11: Similarity scores at different percentile cutoffss at different ddf models.Leftt Full duplication Right: Partial duplication
To analyze various factors in this paper, we used approximately 5000 GPU hours for training and another 1000 GPU hours for inference.
E Discussion on similarity metric design
Occasional memorization, even in a small percentage of generations, can pose problems-precisely the pattern we' we've ve observed. To capture this behavior effectively. we measure the sim score of the top 95% of samples. Using the average or median, in contrast, could fail to detect significant memorization in a few instances. Our choice of the 95% percentile is based on observations by Somepalli al., who identified around 2% of memorization. Adjusting for fine-tuning on smaller dataset, we found 95% to be suitable benchmark.
We have performed ablation analysis duplication factors and the results are presented in Fig. 11 (Left), We see 95, 97, and 99 percentiles follow similar trends. Median scores remain relatively constant, as expected, due to memorization affecting only a small percentage of images, as previously noted by Somepalli et al. Similar trends appear in partial duplication experiments, as evident in Fig. 11 (Right). Hence we proceeded with 95% value of similarity score distribution as the metric in this paper.
10K and full duplication across different data
We present how similar different the images look at different values of SSCD similarity scores in Fig. 12. We also present the similarity score histogram for the model trained with full duplication with ddf= 10 in Fig. 11 (Left). We also present sample Precision-Recall curves on LAION-10K full duplication case in Fig. 11 (Right).
0.97
0,86
0.81
0.75
0.64
0.54
0.45
0.33
0.28
0.13
Gen
adidas
March
adidas
Figure 12: How similar the generation and the matches are different SSCD scores.
F Why do diffusion models copy? Extended Results
In this section, we explore factors contributing to replication in diffusion models. We present additional results, including some figures from the main paper for completeness.
Model conditioning. As shown in Fig. 4 (Left), higher uniqueness in conditioning captions leads to increased similarity scores, with Blip caption conditioning yielding the highest score. In Fig. 17 (Left), FID scores for fine-tuned models follow the same trends we have seen the main paper, with the blip caption model exhibiting the lowest FID score.
15

---

3-
LAION-10K (Pree vs Recall)
0.67
Density
0.73
0.00
Prosision
0.78
0.05
0.64
0.77
0.63
0.0
0.2
0.4
0.6
0.8
1.0
5
10
15
20
Similarity Scores
Data duplication factor
Figure 13
Image Duplication vs Image + Caption Duplication In Fig. 14, we analyze similarity and FID scores for models trained with varying duplication levels (ddf) on LAION-IOK and Imagenette. Both datasets show increased similarity and FID scores with higher ddf. However, the impact of image duplication versus image caption duplication differs. This disparity can be attributed to the diversity of captions used in image duplication for LAION- 10k and the similarity of captions in Imagenette, potentially collapsing into image caption duplication.
Training Regimen Length of Training In Fig. 15, we examine similarity and FID scores during the training process on LAION-10K and Imagenette. We consider scenarios with no duplication, image duplication, and image caption duplication. Longer training consistently leads to higher similarity and FID scores. However, image duplication consistently yields lower similarity scores compared image + caption duplication on both datasets.
Training Regimen Text Encoder Training In Fig. (Right), we analyze the impact of text encoder training on similarity and FID scores for models trained with caption conditioning and various types of duplication similarity scores while reducing FID scores both datasets. In Fig. 17 (Right), we observe the effect of text encoder training on similarity scores for models trained with different conditioning types on Imagenette, specifically with image caption duplication (ddf=5), and we see similar trends as of Fig. 4 (right) even in the presence of data duplication.
Imagenette and LAION-10K Training the text encoder increases
Training Regimen Image Complexity In Fig. 18, present distributions of self-similaaity, entropy, and compression metrics for LAION-I0K and Imagenette. LAION-10K exhibits lower self-similarity, indicating distinct data distribution. However, complexity distributions are slightly shifted to the left, suggesting LAION-IOK is marginally simpler than Imagenette.
Furthermore, provide comprehensive findings the correlation between similarity scores and training data complexity metrics Tab. 2. We extensively evaluate models trained on LAION- 10k without duplication, with image duplication, and with image caption duplication, including cases with ddf=5 and ddf=20. Notably, we consistently observe a statistically significant correlation in all scenarios. Intriguingly, the correlation is stronger when image + caption duplication is present compared to cases with only image duplication.
dupl.
img- dupl.
img dupl.
LATON-JOR
LAfON-Ife
Imagrectte
Imsgrmette
Stnbury sores sooes
3.55
20
Sunlaatt scorcs
3.8-
duplectionn
dupliatiin
dupdcctiion
dupdesttion
Figure 14: Diffusion models trained different amounts duplication and style of duplication. We show similarity scores and FID scores at different values of ddf for LAION- 10k and Imagenette models.
16

---

dupl.
img-cap dupl.
img dupl.
LAOON
Imagemette
Imagemette
Smiliry sCOOS
Smalkity scoess
11.2
8.43
2600 Longih Iasining (lens)
Lasggl
Lasggh Iasining
itens)
Figure 15: Does training for longer increase memorization?
dupl.
img-cap dupl.
img dupl.
Imagenite
Insastatii
Stnbbatt sone
Stmbla sones
enooder
Figure Effect text enender training on Similarity and FID scores.
Table Correlation coefficients and corresponding p-values between the similarity scores and training data complexity metrics for different training scenarios.
Duplication
ddf
Compression
Entropy
p-val (Comp.)
p-val (Entr.)
No duplication
0
-0.2926589
-0.3230121
7.91E-80
8.48H-98
Image duplication
5
-0.2298073
-0.3034324
4.31E-49
5.80E-86
Image Cap duplication
5
-0.298469
-0.3412334
4.19H-83
1.24E-109
Image duplication
20
-0.0918793
-0.1559931
5.83E-09
3.29E-23
Image + Cap duplication
20
-0.1994873
-0.2394177
3.50E-37
2.98E-53
G Misc. results.
We recreated the main paper Fig 2 using Roberta and presented it Fig. 19, The findings stay the same.
H Mitigation strategies
H.1 Training details
Except for the affect of length of training on similairity scores experiment, we trained all the diffusion models for 100k iterations, including the models trained with "mitigation" hueristic, The following are the additional hyperparameters relevent to given strategy.
Multiple Captions (MC): We sample 20 captions from BLIP for all training images. For LAION-10K an additional original caption is included, resulting in a total of 21 captions.
Gaussian Noise (GN): We sample random multivariate Gaussian noise ~ ~N00, T), multiply it by factor of 0.1, and add it to the latent CLIP representations of the caption.
Random Caption Replacement (RC): With a probability of 0.4, we replace the caption of image with random sequence of words.
Random Token Replacement Addition (RT): With a probability 0.I, we either replace tokens/words in the caption with a random word or add a random word at a random location. This step is performed twice for train time experiments and four times for inference time experiments.
17

---

flod capt
hlip/og caps
finetaned model
cluss capt
zandom cupt
10-
0.6
Siiiliirty seorees
20-
0. 00000
10-
OOD 0000
2000
Exed capt class
bipp random capt
Falee
True
Text conditioning style (Imagenette)
Text encoder trained (Imagrnester, ing+rap dupl.)
of models trained Imagenette with different types of conditioning. Right: Similarity scores of models trained Imagenette with different conditioning and with and without encoder training. All the models in this plot are trained with image caption duplication with ddf-5.
Figure 17: Left: FID
Imagrasttee
Imnggrsette
12.5
Imangraette
LAION-LOk
LAION-10k
LATON-1OR
8
Denstt
Deneity
Density
7.6-
5.0
3.5
2.5-
0-
0.0
0.35
0.50
0.75
0.1
Similarity scores (top-1)
Complexity (Entropy)
Complexity (Compression)
Figure 18: Dataset properties LAION-1OK vs Imagenette.
Leff:: Self-similarity Middle: Complexity Entropy
Right: Complexity Compression
Generation Duplication w.r.t Clusters
Text Similarity (Train, Train)
1.0
0.6
0.2
0.0
0.8
Visual Similarity (Train, Gen)
Figure 19: Replication of Figure 2 with Stable Diffusion v1.4 using RoBERTa for Text Similarity instead of CLIP
Caption Word Repetition (CWR): We select a word from the given caption and insert it into random location within the caption with probability 0.4. This step is performed twice for train time experiments and four times for inference time experiments.
Random Numbers Addition (RNA): Instead of adding a random word that may alter the semantic meaning, add random number from the range 0, 10 with probability of 0.4. This step is performed twice for train time experiments and four times for inference time experiments.
H.2 CLIP score and FID tables
In Tab. I, we presented similarity scores for models trained/inferred with different mitigation strategies. In Tab. 4 and Tab. 3, present the corresponding CLIP scores and FID scores. Gaussian Noise
18

---

strategy compromises on the FID score. Random Token/ Random Number addition during inference strategy compromises on CLIP score the most.
Table 3: FID scores LAION-IOK models trained with various mitigation strategies. Complementary Tab. 1.
*collapses to same case.
Train time mitigation
Inference time mitigation
Dup style. / Mit strat.
None
MC
GN
RC
RT
CWR
GNI
RT
CWR
RNA
No Duplication
18.72
16.63
20.63
15.98
17.16
16.73
18.92
18.67
18.14
18.73
Image Duplication
18.44
16.22*
21.62
15.92
15.32
16.03
19.04
19.23
18.38
19.19
Image Caption Dupl.
16.94
16.22*
18.34
15.63
16.25
16.26
17.36
17.75
16.84
17.35
Table CLIP scores of LAION-10K models trained with various mitigation strategies. Complementary to Tab. 1. *collapses to the same case.
Train time mitigation
Inference time mitigation
Dup style./ / Mit strat.>>
None
MC
GN
RC
RT
CWR
GNI
RT
CWR
RNA
No Duplication
30.52
30.27
29.91
30.64
30.74
30.79
30.32
29.54
30.13
29.74
Image Duplication
30.27
30.03*
29.52
30.54
30.83
29.96
29.10
29.74
29.22
Image Caption Dupl.
30.62
30.03*
30.30
30.57
30.76
30.79
30.42
29.69
30.13
29.74
H.3
Extended Qual results
Train time Multiple Captions
NO MODDED UP CIDNING
Train data
JODDY DODDD EoST
Genearion
FAST
Gen wt midiaation
BINNED
CIIRED
EAST
Figure 20: Applying Multiple Captions mitigation strategy on images memorized in SD1
Can we forget images memorized by SD 2.1 with MC strategy? We used different images to show qualitative results for train and test time strategies due the initial freezing of training data in our project. The image-caption pairs necessary for evaluating test-time mitigation using SD 2. were discovered later on.
However, to verify train test strategies can make models forget the memorized images, we conducted experiment by training two models, incorporating the additional 5 image-caption pairs (from Table or Fig 8, right side). The first model is trained with ddf=5 and no mitigation, while the second model utilizes ddf=5 with the MultipleCaps (MC) mitigation strategy. The "train-time" results for these images are presented in ??. Encouragingly, with MC strategy, the model was able to forget out of 5 images. However, the model still retained strong memorization for the prompts "Mothers influence on her young hippo" and "Classic Cars for Sale." " We believe the need to combine multiple mitigations to address these remaining memorizations effectively.
Inference time results. The original and modified prompts used in creating figure Fig. is shown in Tab. 5. For the generation, used Stable Diffusion 1.4 with the generator seed=2, with default settings guidance_ scale=7 55, num num_inffeetteeeeee inferencce num_innferance_step=550
Train time results. In Fig. 21, we show extended train time qualitative results for all the mitigation strategies. We see "Multiple Captions (MC)" mitigation works all the time. In contrast, other methods might might not always be effective.
19

---

Table 5: first column, we show few prompts that induce copying in SD 1.4 model. In the second column, we show updated prompt after applying Random Token (RT) miligation strategy. Sometime the random token is semantically related to the prompt and that might impact the Clipscore of the generation a bit more.
Original Prompts
Modified Prompts
Wall View 003 Classic for Sale
Wall disappointment View part senator 003 historian assen Classic dachsh compositions ;-) Cars Sale Mothers 45460 influence her 44791 young 32450 50192 hippo Living polaris the Light atrix with Ann Grahun ancy Lotz turban ita Hopped- Up Gaming: cricketer sati poutine East
Nothers influence her young hippo Living in the Light with Ann Grahan Lotz Hopped- Upp Gaming: East
20

---

IAD
a
b
(d)
(h)
Figure 21: Qualitative results of various train-time mitigation strategies (a) Training image (b) Generated image without mitigation (c) SD model generation (d) Gen from MC strategy model (e) Gen from GN strategy model (f) Gen from RC strategy model (g) Gen from strategy model (h) Gen from CWR strategy model. Please refer to Appendix H.1 for the full forms of the mitigation strategies. MC strategy is consistently better.
21


Reviews:
[{"review_id": "48N2LNpmJZ", "reviewer": "Reviewer_ciAp", "summary": "The authors conduct a large-scale empirical study about the memorization problem in diffusion models. Their experimental results indicate that text conditioning plays an important role in this problem. Based on these observations, they propose to randomize the text prompts during training to mitigate memorization and copying.", "questions": "- 1. In line177, the conclusion ‘’the model is more likely to memorize images when the captions are more diverse” conflicts with the observations in Section 7. The authors propose several data augmentation methods to make the text prompts as diverse as possible, which successfully mitigates the memorization problem. This indicates that when the captions are more diverse, the model is less likely to memorize images. Please clarify this conflict.\n\n- 2. In line185-186, why do you draw a conclusion that ‘’the model is more inclined to remember instances when the captions associated with them are highly specific, or even unique, keys” according to the experiment phenomenon that the similarity score of random captioning increases after finetuning the text encoder? Wouldn't random captions be even less SPECIFIC keys?\n\n- 3. Please report the FID values in Table1 to verify whether the proposed solutions will hurt the generation quality.\n\n- 4. The mitigation strategies in Section7 are not novel. Even though some of them have positive effects, they are just some data augmentation tricks that should have been used.\n\n- 5. Actually, I find the memorization problem discussed in this paper is related to a common problem in generative models – mode collapse. Especially in the failure cases mentioned in line149-157, the phenomenon is very similar to mode collapse – the generative models perform terribly when they are only able to cover a small portion of the data distribution. As for the memorization problem in Stable Diffusion, I think it is a special case of mode collapse that the text conditioning model collapses because there are not diverse enough text prompts in the training set, which is consistent with the analysis and the proposed mitigation strategies in this paper. Please discuss the difference between memory problems and mode collapse, if you disagree with me, and vice versa, please mention the connection in the related work section.\n\n- 6. Some figures are confusing. For example:\n\n    - a) There is no text prompt in Figure1, so it is hard to understand the error modes. Take the 4-th image as an example: if the text is ''The front page of PUBG”, the output image should be the 4-th image in the lower row and thus I don’t think this is a problem.\n\n    - b) Please use the vector figures in the right two subfigures in Figure2. They look very blurry.\n\n    - c) There is no caption about the left subfigure in Figure2. I don’t know why you put it here.\n\n    - d) There is no legend for the two dash lines in Figure4. And I suggest drawing the FID scores of finetuned models above the chart bars rather than writing them in the caption.", "limitations": "The authors do not discuss the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The memorization problem is valuable for the AI safety community.\n\n2. The authors design a large number of experiments to study this problem.\n\n3. The authors observe that text conditioning plays an important role in this problem, which is a new discovery compared with the existing works.\n", "weaknesses": "1. Some experiment designs cannot verify the arguments. And some results are not consistent with the arguments of the authors. Please refer to Question1-3.\n\n2. The proposed solutions are not novel. Please refer to Question4.\n\n3. Some discussion and references about mode collapse are missing. Please refer to Question5.\n\n4. Some figures in the paper are messy and confusing. Please refer to Question6.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the significance of the memorization problem in the AI safety domain. We sincerely appreciate your acknowledgment of our extensive experimental design.\n\n### [Clarifications on caption diversity] \n\nWe apologize that our language at L177 is slightly confusing. When we said \"... model is more likely to memorize images when the captions are more diverse\", we meant that as the caption gets more specific, the model is more likely to remember that sample. For example, let's say there is an image of a sunset. A less diverse caption would be \"A beautiful sunset\" while a more diverse/specific caption would be \"A sunset in Hawaii shot by Ansel Adams\". Another way to put it is, as the perplexity of caption (wrt model trained on general English text as well models trained with the captions of training dataset) is high, the model is more likely to remember the corresponding image. \n\nEven though with less specific (low perplexity) captions, we observe low memorization, at the same time, we sacrifice the quality of generations (with higher FID scores). In section 7, we suggest using multiple captions instead of one caption for each image to break the one-to-one image-caption correspondence. This ensures that the quality of generations stays the same while reducing memorization.\n\n We will replace the word \"diverse\" with \"specific\" or \"high perplexity\" in L 177-178, clarify the language, and add a small paragraph at the beginning of section 5 defining these keywords.\n\n### [L185-186, Clarifications on caption specificity]\n\nFollowing the previous example, for the same sunset image, if the caption is random, such as \"guatemala muenvica gregation\", this caption is much more specific than the \"A sunset in Hawaii shot by Ansel Adams\" caption and the model is even more likely to remember (as shown in Fig. 4 (Right)). We see empirically that as the captions get more specific (high perplexity) if the text encoder is aligned with the captions, we see the image quality improves or stays the same while memorization goes up. (correlation coeff = -0.63 between FID and SS for Fig. 4, right, when text encoder is trained). \n\nWe will update the draft with an example so that the readers will get a sense of caption specificity in these experiments.\n\n### [FID scores for the mitigation strategies] \nWe have provided the FID scores for mitigation strategies in Table 3 in Appendix. We see that the models trained with the Multiple Captions mitigation strategy **always** had lower FID scores along with lower memorization. Please note that many mitigation strategies resulted in lower FID scores.\n\n### [Mitigation strategies are not novel] \nWe respectfully disagree with this comment. While some of these techniques may have been previously used as augmentations elsewhere, their application in training diffusion models to mitigate memorization has been unexplored until now. Further, the mitigations we select are based on our empirical findings from the initial part of the paper. We believe these findings contribute to the advancement of our understanding of memorization in diffusion models and are in turn validated by practical innovations for those training diffusion models on sensitive data.\n\n### [Memorization vs Mode Collapse]\nThis is a very intriguing observation. From our current perspective, mode collapse and memorization are distinct phenomena in the context of deep learning models, although both can be considered failure modes. In the GAN literature, mode collapse is defined as the generator failing to capture the full data distribution, producing a limited range of similar samples and neglecting distribution diversity. This is a result of joint training of generator, decoder, and encoder.\n\nConversely, memorization aligns closely with overfitting, as indicated in [1,2]. From this viewpoint, the model cannot simultaneously fit the target distribution too precisely (overfitting/memorization) and collapse to only a portion of it (mode collapse). An interesting conjecture is that mode collapse might resemble strong overfitting to specific examples. That said, your proposition regarding mode collapse as a mechanism for memorization is captivating. However, a direct collapse akin to GAN encoder mode collapse isn't feasible in the text-to-image diffusion models we analyze, where the text encoder isn't trained. \n\nYet, determining a clear, falsifiable hypothesis from this connection poses challenges. Could you suggest a specific mode collapse definition isolating this effect? We're open to trying your suggestion. Regardless, we'd be pleased to incorporate this potential link into our related work.\n\n### [Misc improvements]\n\n**Reg. PUBG case.** While it is a legal question whether a model can or cannot generate an image, we follow the same definition of memorization from [3] which follows the most stringent definition of memorization where the model cannot generate anything that a human will perceive as a copy. In Fig. 1, even though the PUBG images are not exact copies, they are close enough to be perceived as copies by humans, and while we cannot speak with legal certainty, using the generated image as a game logo, for example, would imply a sufficient violation of the PUBG logo trademark. \n\n**Reg. figures.** We thank the reviewer for taking the time to suggest improvements to our figures to make the paper more accessible. We will make these changes for the camera-ready version.\n\nThank you for an insightful review with important questions and meaningful suggestions. We hope we addressed your questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional clarifications.\n\n---\n\n[1] Carlini et al. \"The secret sharer\" USENIX Security 19 \n\n[2] Feldman et al. \"Discovering the long tail via influence estimation.\" NeurIPS'20.\n\n[3] Somepalli, et al. \"Diffusion art or digital forgery?\" CVPR'23"}, {"review_id": "BDJ2FMzQPS", "reviewer": "Reviewer_qP9c", "summary": "This paper systematically investigates the memorizing phenomenon of stable diffusion. As they empirically observed, most of the memorizing phenomenon originates from the duplication of images and captions in the training set. Owing to this, the authors propose to deduplicate the captions during the training stage to obviate the memorization of the diffusion model.   ", "questions": "In Section 5.2, the authors explore the effect of the data duplication rate. They vary the rate via increase the sampled probability of duplicated data. My question is by doing so, the underlying ground-truth distribution is changed. Since the model is trained to approximate the ground-truth one. Does it reasonable for the model to generate more data similar to the duplication cluster?\n\nWhat sampler do you use to generate data, DDIM or DDPM? Does the noise in DDPM helps mitigate the memorization phenomenon?", "limitations": "no", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments in this paper are quite complete. In my opinion, most of the components that may be related to the mesmerization phenomenon are considered. ", "weaknesses": "To measure the memorization magnitude, the authors use a similarity score and set the threshold as 0.5, i.e., the score is larger than 0.5 means there is a memorization for a certain generated image. My concern is whether this score and threshold are valid criteria. \n\nFor the stable diffusion trained on a large-scale dataset without duplication i.e., stable diffusion v2.1, the memorization phenomenon seems almost mitigated. Thus, my question is whether increasing the unduplicated data can fix the memorization phenomenon. \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the merit of our work and acknowledging the thoroughness of the experiments. \n\n### [Is SSCD score $> 0.5$ a reasonable proxy for memorization?]\n\nSomepalli et al [1] found if the similarity of SSCD embeddings of 2 images is greater than 0.5, they are most likely to be extremely similar to each other. On the fine-tuned models on Imagenette and LAION-10k, we also found that 0.5 is a reasonable cutoff while 0.7 is stricter threshold to detect potential copies. Please also refer to **Fig. 22 in the response pdf** to get a sense of how much the similarity score changes based on how similar or different the images are.\n\n### [Can increasing the unduplicated data remove memorization?]\n\nWhile most of the instances in which the model memorized due to training data *exact-duplication* disappeared in SD 2.1 compared to SD 1.4, some instances like \"Canvas Wall Art Print\" studied in Somepalli et al [1] still exist. From our experiments, we found that even in the absence of training data duplication, some memorization exists. We observe that if an image has a unique caption (For e.g. \"Pubg\" in Fig 1 or \"Bloodborne\" from Somepalli et al. [1]), the model is a bit more likely to memorize. Also if images are too simple (as we examined in section 6, figure 7), the model can still memorize. These are a few factors we have examined and it is highly possible that there might be other underlying factors that we hope the future works will uncover. \n\n\n\n### [Are similarity scores higher due to changes in underlying distribution?]\n\nWe apologize that we missed mentioning an important detail in the paper. For a given dataset (say LAION-10k), we freeze the inference captions for all the experiments, and all the metrics computed are on generations based on these captions.  While it is true we are changing the training data for each of these models, the frozen captions ensure the computed similarity score metric is comparable across the experiments. The computed score is also a lower bound of memorization in these models. Additionally, this setting is closest to reality since we don't sample the same caption multiple times during inference even if the training data contains duplicates. \n\n### [Impact of sampler]\nWe used the DDPM sampler for training our models. We will add this detail to the experimental section. The effect of the sampler on memorization is indeed an interesting question. We leave this study for future work and we will amend our conclusion to include this question.\n\n\nWe thank the reviewer for a thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n--------\nReferences:\n[1] - Somepalli, Gowthami, et al. \"Diffusion art or digital forgery? investigating data replication in diffusion models.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2023."}, {"review_id": "HjcjYrrFfD", "reviewer": "Reviewer_rsTQ", "summary": "This paper is a deep dive into the phenomenon of memorization in Diffusion models. I find that the paper is very well position given the current state of generative ML and the paper does a great job at asking the right questions about various factors concerning the memorization of images. Some of the key takeaways in the paper are really strong, such as randomizing captions, or partial duplication of images, but some others are not very well substantiated. Finally, the authors also conclude with some guidelines and mitigation strategies for practitioners, making this a refreshing read.\n\nThis paper is hard to review in a strengths/weakness pattern because each section has its strengths and weaknesses. First, this paper tries to do a lot. I think that ends up being a weakness of the paper because the high-level message from the abstract and intro gets swayed away through the body making this a very confusing read. Especially in the sections from 3 to 5.3. It is only after that the main narrative of the work catches on. I understand that the authors want to show a lot of experiments and analyses they did, but many of them are not at all central to the key message of the paper.\n", "questions": "1. \" Since the metadata is structured to provide URLs from similar domains together, this makes us likely to find large clusters.\" Do you have a reference to this? I do not think this is the case from my observations of similar images across various parquets in the LAION dataset\n2. Are the results in 5.2 on unconditional models?\n3. Figure 4: why does pretrained SD have different similarity scores?\n\n", "limitations": "-", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "1. This paper tries to break down various factors that are responsible for the memorization of training data by diffusion models.\n2. The discussions in sections 6 and 7 are really valuable to the community.\n3. The paper asks the right questions and also performs experiments to support their hypothesis (more about this in the weaknesses). But overall, the paper serves as a great starting point for understanding the state of memorization in generative models, giving a peak at some thought-provoking questions, and using some of these hypotheses to actually suggest mitigation strategies.\n4. I am curious about the image complexity argument. This is a refreshing take, but also very different from image classification literature where typically hard images are memorized because the model can easily learn the simple ones to classify. It seems that the take for generative models is the opposite. but again there is a distinction between anecdotal arguments of features versus the argument of compressibility and entropy studies here. A good metric potentially would be the embedding similarity as assessed by a CLIP model.", "weaknesses": "My biggest qualm with this paper is about the metrics. It seems that this paper is based in a delusional world where dataset similarity and FID are the sole indicators of model memorization and model performance. I know that evaluating both generation and memorization are hard problems, but this paper takes these yardsticks way too far, to an extent that following up with some results becomes uncomfortable. And in many sections, I get the sense that hypotheses and conjectures are drawn just to satisfy these so-called oracle notions of memorization and quality.\n\n1. On generation quality: Section 7 severely needs an analsis on the impact of these changes on the quality of generation. Similary, in Figure 4, FID scores are all over the place. are these even good models?\n2. On memorization: In Section 5.1 the authors note that they did not observe any duplication in the clusters they studies. Again telling how the metrics of DS are not informative or reflective of memorization in the first place. I do not understand why checking similarity scores with the top 5% of generations. Further, in Figure 5: lower SS when the image is duplicated. and this happens even in the right figure. But does this actually related to \"memorization\" or just stay till the idea of dataset similarity. For instance, in the de-duplication done for SD2, not all duplicates were \"full duplicates\" if I am correct. So i think the explanation saying random captions helps reduce SS is an over statement.\n3. On section 3 and Figure 1 proper attribution should be given to the original authors who devised the technique\n4. I have not seen people use clip text features to assess how close texts are. I would suggest using some text model such as RoBERTa for these use cases. \n5. \"possibly because FID is lowest when the dataset is perfectly memorized\" I do not see how this logically flows. And in general I did not like that this paper makes too many conjectures to explain every possible phenomenon they see in results. It just feels like the authors are trying to fit a jigsaw no matter what.\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are happy to learn that you perceive our contributions as both innovative and crucial in initiating a deeper understanding of memorization. \n\n### [Generation quality]\nWe provide FID scores in Appendix Table 3 for different mitigation strategies. Irrespective of data duplication, Multiple Captions strategy yields lower FID scores than no mitigation case. We agree with the reviewer that the FID scores in Fig. 4 bounce around a lot.  When we reviewed the generations of these models, they seem to be of reasonable quality (for models trained on only 10k images). We present a few images in **response pdf Fig 21**. It seems that the increases in FID are because of changes in the image distribution (as in the model is not learning all the classes equally well), and not in image quality.\n\n### [Memorization]\n**Are DS similarity scores indicative of memorization?** Kindly refer to **\"Is SSCD good for understanding memorization?\"** section in the global response.\n\n**Not finding clusters in sec 5.1** We do not agree with the reviewer's characterization that we did not find meaningful clusters in SD 2.1.  While we found fewer & smaller clusters (likely because of de-duplication of the training set) in  SD 2.1 than we found in SD 1.4, clusters were still found, and an examination of these clusters revealed copying behavior (e.g., the \"hopped up gaming'' cluster). Note that for SD v2.1, the dataset was explicitly cleaned of duplicate images using the PHash metric  (based on personal correspondence with the authors). \n\nWhile acknowledging the metric's limitations, our focus in section 5.1 is to demonstrate memorization under high text and visual similarity and perform a comparative SD v1.4 vs. SD v2.1 analysis.\n\n**Why top 95pc?** Kindly refer to **\"The choice of percentile in SS metric\"** section in the global response.\n\n**Why SS lower when the image is duplicated?** In partial duplication experiments, we used BLIP captions for duplicated samples and LAION captions for others. During inference, all captions are from LAION. We believe this shift slightly reduced replication. In **response pdf Fig 18**, SS with varied sampling approaches in test time is presented. While the strategy (BLIP captions for duplicated images) in training and testing slightly increases memorization compared to LAION captions, it is still less than image + caption duplication models.\n\nEven in the updated plot, a slight dip at lower ddfs remains. This might be due to the model not memorizing \"easy\" images as non-duplicated easy images are sampled less. We base this hypothesis on manual observations made on the top 100 inferences with high similarity scores.\n\n**..random captions help reduce SS is an overstatement.** While we are unsure which section the reviewer is referring to, we will address this as best as we can. In all our LAION expts, we never used \"truly random\" captions. We always used BLIP captions which describe the contents of the image reasonably well. And training with BLIP caps (along with original caps) consistently results in lower top 95% SS.  Our statements were motivated by this consistent drop in SS.  \n\n### [Use Roberta instead of CLIP]\nWe adopted CLIP for text similarity computation in Fig. 2 due to its use in SD conditioning. As per the reviewer's suggestion, we recreated Fig. 2 (of SD 1.4), using RoBERTa and we present it in **response pdf Fig 24**. The change minimally impacts results and still shows that SD v1.4 generates memorized images when both images and captions are duplicated. The Spearman Correlation b/w RoBERTa and our previous metric is **0.899**,  indicating consistent rankings. We will include the new figures for SD v1.4 and v2.1 using RoBERTA to the appendix. \n\n### [possibly because FID is lowest when the dataset is perfectly memorized - L147-148]\nOur experiments follow Somepalli et al.'s framework, sampling inference captions from training data. We ensured no exact image or caption duplicates in our training data. In this context, if the model perfectly memorizes image-caption pairs and associates them during inference, FID -> 0 as inference images -> training size with non-repeating captions. We intend to revise this phrasing to clarify the connection.\n\n###  [Misc Qs]\n**section 3 and Fig. 1 proper attribution** We are happy to amend section 3 and Fig. 1 that this SSCD-based approach for finding duplicates was first introduced in Somepalli et al. \n\n**Q. reg. metadata:**  Originally, we observed higher similarity within a subset of our selected parquet file compared to other slices, indicating potential metadata structure. Specifically, we used the 2nd parquet file and computed the SSCD similarity b/w the first 800K samples to the next 800K samples and found $\\sim 805K$ edges above our selected threshold. When comparing the first 800K samples from the 2nd parquet file, to the first 800K samples from the 7th parquet file  we found much fewer edges $\\sim 734K$. However, prompted by the reviewer, we tested this across multiple slices and observed the metadata may indeed not be structured. We will update the next draft to reflect the same. Importantly, this observation doesn't impact the analysis in Section 5.1.\n\n**Results in 5.2:** They are blip-caption conditioned models for Imagenette and original-caption conditioned for LAION-10k experiments.\n\n**Clarifications on Fig.4:** In this experiment, even though the training images are the same, the captions used in each of the experiments vary. Since we draw the captions from the training data (to study the worst-case memorization behavior), essentially the captions used in generation vary between the different caption settings. \n\nOverall, we appreciate the reviewer's insights, and these additional experiments have added depth to our analysis. We hope we addressed your questions sufficiently and we would be grateful if you would consider raising your score. Do you have any additional questions we can address?"}, {"review_id": "FOdgYUm4t7", "reviewer": "Reviewer_4RBY", "summary": "The paper tries to explain the memorization and copying of training data in diffusion models. The paper establishes a connection between caption duplications and memorization on top of data duplications analyzed in previous works. In their main experiments, the authors fine-tune various SD models on ImageNette/LAION subsets and show that controlling data duplication does not completely prevent memorization and that highly specific captions amplify copying. They also relate memorization to training parameters and data complexity. Finally, the paper introduces several mitigation strategies based on randomizing conditional information. \n", "questions": " - There are pretty strong differences between Imagenette and Laion-subsets, do you have also looked at other datasets and do they fall between the two datasets shown in the paper or do the results look completely different there? ", "limitations": "In my opinion, one of the largest limitations is that even the best-performing training guidelines/mitigation strategies might not always prevent copying. The paper gives us a dataset similarity result of 0.42/0.47 and claims that in terms of image similarity, 0.50 would be a duplicate. So it is not clear to me that this actually solves copying. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": " - Overall, I find the experiment design in the paper to be good and got a better understanding of the memorization phenomenon from the paper. I also enjoyed the writing style of the paper. \n - I find the connection between image complexity and copying to be reasonable and believe this is the first time this has been analyzed this way. \n - While previous papers were mostly about analyzing the impact of data duplication on memorization, it makes sense to also control the captions and the paper shows that there exists a connection between the two. \n - The problem setting of the paper is very relevant. \n", "weaknesses": " - Pretty much all experiments are based on the dataset similarity score which is defined as the 95-percentile of image-level similarities. As this is very important for the validity of the experiments, I believe that it would make a lot of sense to motivate this design choice further. From previous works, the authors quote that an image similarity larger than 0.5 shows strong similarities. In the worst case, the distribution of image similarities could contain slightly less than 5% near duplicates and slightly more than 95% unrelated images which would result in an arbitrarily small dataset similarity but still a decent memorization quote. Maybe the authors could simply demonstrate that the distribution of image similarities is well-behaved and also quantify the number of outliers, eg at a dataset similarity of 0.2, how many of the remaining images above the 95%-quantile actually achieve an image similarity larger than 0.5. Ideally, it would also be great to shortly demonstrate that results look similar with larger percentiles and that the 95-percentile is not critical for the outcome of the experiments. \n- The range of similarity scores in most of the experiments is actually somewhat limited. I do not have a good feeling about the scale of this metric, but for example, the data duplication factor seems to only influence the score in the range 0.64 to 0.70. As this is the 95% quantile and 0.5 can be assumed to be a duplicate, this means that in all cases we have strong duplicates. I do believe the authors that there is a connection between duplicates/captions and copying, however, either the dataset score is not the right metric for this kind of experiment or duplicates/caption can actually only explain a relatively small part of the problem. Also even with the best-performing train time mitigation strategies, the similarity score is 0.42/0.47 so from my understanding this could mean that we still have about 5% of duplicates. \n- The mitigation strategies are somewhat simplistic. I also believe that it is crucial to give a more detailed analysis of the \"minimal effect\" that the mitigations have on model performance. The images on the left half of Figure 8 work well in the sense that they seem to produce the same object that was also on the original image. However, most of the images on the right side of Figure 8 are not convincing to me. For example, I do not see any interaction between the hippo mother and her child (prompt: \"Mothers influence on her young hippo\"), and from a short Google search I would say that the person in the image is not Ann Graham Lotz (prompt: \"Living in the Light with Ann Graham Lotz\") and I could not find any connection between the last image with mitigation and \"hopped up gaming east\". I think mitigating copying is clearly important, but if this prevents the SD model from following the user's prompt it becomes useless. I think Figure 8 should show the results of the best train and test time mitigation strategies for the same images and I would also like to see the results of the best train time defenses on the images on the right side as apparently they seem to be harder. For some images in Figure 16, I also do not find any of the test time defenses to produce reasonable results. \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough feedback. We greatly appreciate your acknowledgment of our experiment design, improved comprehension of the memorization phenomenon, and writing style, and that you found our approach to controlling captions and image complexity analysis insightful.\n\n### [Design choice of 95 percentile]\n\nWe kindly ask the reviewer to consult the global response for our detailed response to this specific comment.\n\n### [Understanding the range of similarity score metric]\n\nTo understand what it means to go from simscore of 0.64 to 0.7, we looked at the inferences from different perspectives. We computed the number of points with high similarity scores for the model with no duplication and the model with duplication ddf=20. We found that the latter model has **2.5x** the number of points than the former model. It shows that the simscore does indeed reflect an amount of memorization in the model. \n\nWe've further added an additional plot to explain how this metric is scaled. For example, if we duplicated 20%  of the training data. In **response pdf Fig 19**, we compare the results of 5% and 20% data duplication. This extends the range of the metric to 0.75 for the 20% duplication case. \n\n\n### [Is duplication or caption the whole story?]\n\nTo answer your question on whether copies exist in mitigated models, we computed % of images with high SSCD scores (>=0.5) for different models.  We present the numbers against similarity scores in the table below. (MC= multicaps mitigation)\n\n|  Model setting| Sim.Score | % of points > 0.5|\n| --| -- |--|\n|Nodup | 0.638| 17.65 |\n|Nodup + MC| 0.42| 2.17 |\n| Full dup, ddf=5 | 0.663 | 19.55| \n| Full dup, ddf=5 + MC | 0.47 | 3.85| \n\nWhile duplication and caption specificity explain a part of memorization, we believe that other factors, such as image complexity, may also play a role in influencing the extent of memorization. Notably, even with our most effective mitigation strategy, the model memorizes approx 2% of images, even when they were not duplicated during training. Qualitatively, they are simple images (e-commerce visuals) or brand logos (like Adidas).\n\n\nOur primary focus in this study was to address memorization arising from the caption side and successfully mitigate its impact. While we briefly explored the influence of image complexity on memorization, we acknowledge that there are more factors at play. We view these findings as a stepping stone, and we encourage future research to delve deeper into other contributing factors for a comprehensive understanding of memorization in diffusion models. \n\n\n\n### [Reg. mitigation strategies]\n\n**How are mitigation strategies affecting the model performance?** \n\nWe agree with the reviewer that test time strategies can impact the text guidance, we actually studied this impact quantitatively by comparing Clip Scores in Appendix Table 4. We see that the *impact is minimal* under many strategies. We also study how the FID scores are impacted when each of the mitigations is applied in Appendix Table 3. The increase in FID scores are also minimal for many mitigation strategies.\n\nRegarding the example, \"Living in the Light with Ann Graham Lotz”, this image was observed as an undesirable memorization in a recent Google work [1]. This is, of course, subjective, we might still want models that generate people of public interest with high fidelity. Yet, we, as well as [1] are concerned with copyright problems, as the generated image is the exact copyrighted image corresponding to her radio program. Based on our definitions of memorization,  would rather not generate an image in the likeness of \"Ann Graham Lotz\", than generate a copyrighted image. We agree that there is a trade-off here, between fine-grained prompt following and memorization, and our work presents one possible choice. We will extend the discussion of the scope of memorization in our work to better reflect this.\n \n\n[1] - Carlini, Nicholas, et al. \"Extracting training data from diffusion models.\" arXiv preprint arXiv:2301.13188 (2023).\n\n**Why we chose different samples for train, test time results?** \n\nWe used different images to show qualitative results for train and test time strategies due to the initial freezing of training data in our project. The image-caption pairs necessary for evaluating test-time mitigation using SD 2.1 were discovered later on.\n\nIn response to the reviewer's concerns, we trained two models, incorporating the additional 5 image-caption pairs (from Table 5 or Fig 8, right side). The first model is trained with ddf=5 and no mitigation, while the second model utilized ddf=5 with the MultipleCaps (MC) mitigation strategy. The \"train-time\" results for these 5 images are presented in **response pdf Fig. 23**. Encouragingly, with MC strategy, the model was able to forget 3 out of 5 images. However, it still retained strong memorization for the prompts \"Mothers influence on her young hippo” and \"Classic Cars for Sale.\" We believe the need to combine multiple mitigations to address these remaining memorizations effectively. \n\n**Multiple Captions strategy is consistently better.**\n\nWe would like to point out that Fig 16 is all train time strategies. We apologize for being unclear and we will update the caption.  MultipleCaption (Fig 16, col c) mitigation consistently removed memorization in all of the cases. Although the other strategies may not have exhibited the same level of efficiency, we deemed it essential to present both the qualitative and quantitative results for all strategies. Our intent is to inspire and guide future research in this crucial area.\n\n\nOverall, we appreciate the reviewer's insights, and these additional experiments have added depth to our analysis. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address."}, {"review_id": "4QpApARHVY", "reviewer": "Reviewer_MEzn", "summary": "This paper discusses the issue of data replication in text-to-image diffusion models, specifically focusing on the memorization problem. It mentions that these models, such as Stable Diffusion, are known for unintentionally replicating their training data, which has led to recent lawsuits. The commonly held belief is that duplicated images in the training set contribute to content replication during inference. However, the paper's analysis reveals that the text conditioning of the model also plays a significant role in data replication. The experiments show that unconditional models don't replicate data as frequently as text-conditional models. Building on these findings, the paper proposes several techniques to mitigate data replication during inference by randomizing and augmenting image captions in the training set.\n\n\n\n\n", "questions": "- It may be interesting to show some toy cases in 2 dimensional or 1 dimension in which one can simply play around with how the conditioning and replication will influence the vector field (probabilistic flow) of the generation. It might be a good justification of authors' hypothesis.", "limitations": "The observation is based on the heuristic observation, however, it is fine since it is impossible to have a white-box explanation of a neural network. It would be better to have some guidance in theoretical reasoning.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and well-written\n- The experiment setup is clear and sound.\n- The originality of resolving the memorization issue of the diffusion model is interesting.\n- The experiment results are convincing.", "weaknesses": "- Even though this paper is focusing on the empirical results and evidence of memorization of the diffusion model, I am still expecting a high level of theoretical understanding. Specifically, does it holds for the general case regardless of the type of SDE, data distribution etc.?\nI think these questions are also important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the merit of our work. We are pleased to know that you found our experiment setup clear and sound.  \n\n### [Toy cases]\nWe thank the reviewer for this suggestion. In our camera-ready version, we will visualize a toy problem in 2 dimensions where we know the closed-form density function of the underlying data distribution precisely. In this case, we can visualize the deviation of the vector fields generated by diffusion models trained using various protocols, including our interventions, from the gradients of the true log-density. \n\n### [Theoretical understanding]\n\nUnderstanding why diffusion models memorize, and why they even work well in the first place, is an exciting and active area for future theory research. Text-to-image models considered in this work are a form of classifier-free guidance models proposed by Ho et al. [1].  Existing research [1,2,3] has made limited progress towards building a theoretical understanding of diffusion models generally, and memorization remains unexplored as of yet.  Alternatively, [4] provided a framework for studying memorization in discriminative models, and future work may build up corresponding theories for diffusion models. In this work, we present an empirical investigation into factors which contribute to memorization in diffusion models with a focus on providing recommendations to training future diffusion models. We hope subsequent work will bridge the gap between theoretical underpinnings and empirical observations of memorization in diffusion models. We will update the discussion to underscore this as a promising trajectory for future research.\n\n\nThank you again for your thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n-----\n\nReferences: \n\n[1] - Ho, Jonathan, and Tim Salimans. \"Classifier-free diffusion guidance.\" arXiv preprint arXiv:2207.12598 (2022).\n\n[2] - Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020).\n\n[3] - Chen, Sitan, et al. \"Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.\" arXiv preprint arXiv:2209.11215 (2022).\n\n[4] - Feldman, Vitaly, and Chiyuan Zhang. \"What neural networks memorize and why: Discovering the long tail via influence estimation.\" Advances in Neural Information Processing Systems 33 (2020): 2881-2891.\n\n"}, {"review_id": "TE058X0lj3", "reviewer": "Reviewer_R9Bm", "summary": "This paper aligns with a range of recent works which investigate the tendency of diffusion models to memorize  training data and emit this during sampling, when prompted to, and presents strategies to mitigate this for data-processing, training and inference. Other than previous works, which mainly focused their analyses around duplicate images in the training data and model overfitting (i.e. dataset size, portion of train examples seen during training, …) , this work separately analyzes the influence of both these points and further adds to this the influence of captions (and their duplication) in the training data and complexity of the images in the training dataset. The main findings for these are that highly specific captions for every image promotes duplication which is furthermore also more likely for images which are simple (as measure by pixel histogram-entropies and jpeg-compressability). ", "questions": "How diverse are BLIP captions? It would be interesting to do the same analysis than made for LAION captions in Sec. 5.1.", "limitations": "There is not dedicated limitations section, however, the limited scope of the experiments, which focus only on finetuning is mentioned in the paper. I’d recommend to add  a dedicated limitations section and once more state this. Also it would be interesting to see this analysis for other types of models such as AR models or GANS (although I know that no comparably powerful models to SD are publicly available as of now). The fact that this misses could also go into such a section", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The paper separates the main factors of influence during training from each other and bases the findings made for each of these separate categories on rich sets of empirical experiments, making the paper conclusive and interesting\n* The main take-aways are novel and of high importance for the community, especially since the authors center their experiments around fune-tuning a foundational DM, which is what many researchers do nowadays.\n* The key take-aways for every analyzed influencing factor are summarized below the respective sections which makes the paper well structured and raises comprehensibility.", "weaknesses": "* It is totally reasonable that the authors cannot train a foundation model like SD from scratch and therefore focus on finetuning based on these models. However, the exact influence of the pretraining on all the results presented on conclusions taken remains unclear to me after the paper. It would therefore be interesting to also include results of models trained from scratch for all of the presented experiments, since it would provide insights about the generality of the statements made.\n* Related to the above point, I think the title may promise a bit too much, since it would need experiments from scratch (for models such as SD) to choose that title. Maybe sth. Like “Why finetuned diffusion models memorize and …” would be better. What du you think?\n* To me, it is counterintuitive that the model trained with no duplication has a higher dataset similarit score than the one with partial dup. in Fig. 5. If I didn’t miss sth, this is never discussed and is therefore confusing. It might also raise concerns about the significance of the results presented in this part which I think is crucial for the entire paper. Could the authors share their take on this?\n* For a reader not knowing SSCD it may be hard to estimate, how reasonable the chosen newly introduced metrics related to SSCD are, since many of these are based on thresholds. Therefore, it would be great of the authors could add similarity scores to all visual examples showing found duplicates (especially in Fig. 1). \n* In line 91 it is stated that FID would measure image quality and diversity. I don’t agree for diversity since there are many models with low FID scores which actually have very low diversity (many GANs, see e.g. [1,2]). I’d therefore recommend to either additionally include Precision and Recall Metrics [2] in the evaluation or remove diversity here.\n\t\t\n[1] Sauer, A., Chitta, K., Müller, J., & Geiger, A. Projected GANs Converge Faster.\n[2] T Kynkäänniemi, T Karras, S Laine, J Lehtinen, T Aila ,  Improved precision and recall metric for assessing generative models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for your insightful comments and positive feedback on our paper. Your recognition of our rigorous empirical experiments in dissecting training factors, coupled with the acknowledgment of the novelty and significance of our main takeaways, is truly encouraging.\n\n### [Do fine-tuned models tell the whole story?]\n\nWe concur with the reviewer that fine-tuning alone may not provide a complete picture. To assess this, we initially benchmarked the pretrained models on inference captions to evaluate the level of memorization. As shown by the green bars in Figure 4 and Figure 5 (left), similarity scores were found to be minimal, suggesting that fine-tuning might not significantly contribute to additional memorization. \n\nTo further investigate, we attempted to train the U-Net from scratch (while keeping the encoder and decoder modules frozen). However, on this relatively small dataset, the trained models exhibited high FID scores, and the generated results were not satisfactory. Scaling to a much larger dataset for this setting is computationally prohibitive, considering the multitude of models we need to train for this paper.\n\nDespite these challenges, we acknowledge the importance of providing an ablation study. For the camera-ready version, we plan to include additional results using a slightly larger dataset (~1M samples), with either LDM or DDPM backbone (as explored in Somepalli et al.[1]), to offer a more comprehensive analysis.\n\n### [Title update]\nWe agree that the current title might be a bit too general currently while we are studying mainly T2I models, we will definitely update the title and parts of the introduction to reflect that.\n### [How can partial dup be less than no dup? ]\n\nIn experiments where the image is duplicated but the caption is not, we used a set of captions for each training instance of the duplicated images.  To do this, we use BLIP to sample random captions for duplicated images at train time.  At test time, we sample using the original LAION captions.  The test-time sampling uses different captions for duplicated images than the (Blip) captions used to train, and this shift makes the model less likely to replicate the training images.\n\nTo dig deeper into this matter, we've added **response pdf Fig 18** with various sampling approaches in test time. The new curve reflects the same strategy (BLIP captions for duplicated images) employed during both training and testing. We see that this strategy results in slightly more memorization than sampling with LAION captions, but still far less memorization than the case where both image and caption are duplicated during training. \n\nEven with these updated captions, for lower ddfs, the similarity scores are lower compared to no duplication case. We hypothesize this slight dip is caused by the model not memorizing \"easy\" images since some of the easy images are not sampled as much as in the case of no duplication since we train all the models to a fixed number of iterations. We base this hypothesis on manual observations made on the top 100 inferences with high similarity scores.\n\n### [ How does the sim score and different SSCD thresholds look like?]\n\nWe agree with the reviewer that adding SSCD similarities in Fig 1 could help the reader understand the content better, and we will update this in the camera-ready version. However, for the sake of your understanding, these are the SSCD numbers we observed in Figure 1, left to right 0.63, 0.60, 0.57, 0.57, 0.56, 0.58, 0.52.\n\nPlease refer to **Fig 22 in response pdf** to see how much the images differ at different similarity scores. We also observed that similarity scores > 0.5 tend to be very close matches.\n\n### [ L91 FID would measure image quality and diversity]\n\nWe sincerely thank the reviewer for bringing this error to our attention. We will update the draft. In response to your suggestion, we have computed Precision and Recall based on the approach presented in Kynkäänniemi et al. [2]. In **Fig. 20 (right) of the response pdf**, we have included these metrics for the full duplication LAION-10k experiments. We firmly believe that incorporating these metrics will enhance the depth of our analysis. Therefore, we commit to including them for all LAION-10k experiments in the supplementary material of the camera-ready version.\n\n### [How diverse are BLIP captions?]\nWe thank the reviewer for the suggestion. We will add a similar analysis to Sec 5.1 using BLIP captions to the appendix in the final draft.  We perform preliminary exploration to understand the diversity of BLIP captions. We used the CLIP text encoder, to compute the cosine similarity for each image across BLIP captions, and observe median similarity b/w BLIP captions is $0.79 \\pm 0.08$. The similarity of LAION captions to BLIP captions for the same images is $0.64 \\pm 0.07$. Unfortunately, since LAION doesn't provide multiple captions per image we cannot use this metric to measure diversity for real captions.\n\n### [Adding a limitations section]\n\nWe appreciate the reviewer's valuable feedback regarding the scope of our experiments and the need for a dedicated limitations section. It is indeed a great suggestion to provide a comprehensive overview of the scope and assumptions underlying our experiments. We will update the conclusion to include the limitations in the final draft.  \n\nThank you for your thoughtful review. We hope we addressed your concerns/questions sufficiently and we would appreciate it if you would consider raising your score in light of our response. Please let us know if you have additional questions we can address.\n\n---\nReferences:\n\n[1] - Somepalli, Gowthami, et al. \"Diffusion art or digital forgery? investigating data replication in diffusion models.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2023.\n[2] - T Kynkäänniemi, T Karras, S Laine, J Lehtinen, T Aila , Improved precision and recall metric for assessing generative models"}]

Author rebuttal:
We would like to thank **all six reviewers** for their highly constructive feedback! We very much appreciate their assessment that we are "studying an important problem" (ciAp, 4RBY, R9Bm),  with "thorough experimentation" (ciAp, qP9c, rsTQ, 4RBY, MEzn, R9Bm),  "first to introduce text conditioning.." (ciAp, 4RBY, MEzn), with "results are valuable to community" (rsTQ),  "thought-provoking" (rsTQ), and "a great starting point... memorization" (rsTQ,4RBY),  and we raised "interesting image complexity argument" (rsTQ, 4RBY),  and finally it is "well written" (4RBY, MEzn, R9Bm).

The reviewers raised many excellent and thought-provoking questions and we conducted many experiments to address these questions. This interaction definitely added more depth to the paper and helped us understand this phenomenon better. 

### [A brief summary of questions and additional results]
1. **Choice of 95th percentile for similarity metric**: To clarify the rationale behind selecting the 95th percentile over 99 or 50, we added an ablation in Fig 17.
2. **SSCD similarity score for memorization** We added Fig 22 which shows how generation & top matches look like at different SSCD scores.
3. **Range of similarity scores and what it means?** We conducted additional experiments (with 20\% of data duplicated) and presented those results in Fig 19. We see that when the amount of duplicated training data increases, sim score metric's range increases. 
4. **Partial duplication further analysis**  We took a deep dive into the question of why partial duplication has lower SS than the model with no duplication. We found that a slight domain shift between training and testing led to this, and recomputed these results and presented them in Fig 18. The original findings still hold.
4. **Number of copies** - We computed the actual numbers of images with SS > 0.05 and presented the results in a table as part of our response to **reviewer 4RBY**. This additional analysis also shows that there is a significant reduction of such images with the MC mitigation strategy.
5. **Diversity metrics** For completeness we will add precision, and recall metrics to the paper. One such plot is shown in Fig. 20 (right).
6. **How good are ImageNette models** We presented a few generations of text-encoder-frozen IN models trained with different styles of captions in Fig 21. We will add an expanded version of this figure to the appendix.
7. **Qual results for same images for both test and train time mitigation** Because the training data was fixed before we found the memorized prompts in SD 2.1, we chose different examples to show the effect of train/test time mitigation strategies. For the response, we trained a few new models adding these 5 images to the train set. We recreated the train-time qualitative analysis figure as shown in Fig 23.
8. **Roberta instead of CLIP for image cap analysis** We recreated the main paper Fig 2 using Roberta and presented it in Fig 24. The findings stay the same.
9.  Final draft will incorporate these enhancements: limitations paragraph in conclusion clarifying experiment scope and metric. Introduction section will reemphasize paper's memorization scope. Related work will cover mem. vs mode collapse. Address typos, readability, define with examples, improve figures, add missing citations.

### [Paper's definition of memorization]
In this paper, we follow the same definition of memorization as Somepalli et al where a reasonable resemblance (by a human) to a specific entity is considered a copy. Even when the prompt explicitly references a brand logo and the model duly generates that logo, it is unequivocally categorized as a memorization instance.  The legality of whether a generation is fair use or not is beyond the scope of this work.

### [Is SSCD good for understanding memorization?]
We have found that similarity scores above 0.5 empirically are strongly indicative of memorization. We added a figure to **response pdf, Fig 22** that shows representative image pairs with different levels of sim score. One can see that above 0.5, image pairs appear to be almost copies of one another.  Note that all these examples are from the LAION-10k model trained with no duplication. 

### [The choice of percentile in Similarity Score metric]
Occasional memorization, even in a small percentage of generations, can pose problems—precisely the pattern we've observed. To capture this behavior effectively, we measure the sim score of the top 95% of samples. Using the average or median, in contrast, could fail to detect significant memorization in a few instances. Our choice of the 95% percentile is based on observations by Somepalli et al., who identified around 2% of memorization. Adjusting for fine-tuning on a smaller dataset, we found 95% to be a suitable benchmark.

In response to your query, we've performed an ablation analysis on LAION-10k and full duplication across different ddf settings. Results, as displayed in **Fig 17 of our response pdf**, follow same trends across 95, 97, and 99 percentiles. Median scores remain relatively constant, as expected, due to memorization affecting only a small percentage of images, as previously noted by Somepalli et al. Similar trends appear in partial duplication experiments, as evident in Fig 25 (left). We acknowledge that we needed to justify this design choice better, we will add a paragraph on the motivation behind this design choice in the experiments section.

---
In conclusion, we wish to underscore the current relevance and substantial significance of this research. Since the SD lawsuit, addressing memorization became pivotal for companies. Many recent papers offer short-term solutions, but our study delves deeper. We comprehensively examined underlying factors and proposed mitigation strategies based on what we learned from our analysis. Since the paper came out, an industry-leading model provider has reached out for guidance on how to implement our methods in their pipeline. 

Meta-review: This work presents an in-depth empirical study of the memorization problem of text-to-image diffusion models. Reviewers are overall positive about this work, and think this work is timely, novel, important, and the experiments are comprehensive and conclusive. One major concern is the metric (dataset similarity scores) used to measure memorization, which requires further motivation and justification. AC agrees that this work is a welcome practice towards addressing the memorization and copying challenges of modern generative AI, and thus recommends acceptance.
Decision: Accept (poster)