Title: Understanding and Mitigating Copying in Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models

memorization

data replication

model safety
Abstract: Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models.  While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set. Code is available at https://github.com/somepago/DCR.

Understanding and Mitigating Copying in Diffusion Models

Abstract

Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, we observe that the text conditioning of the model plays a similarly important role. In fact, we see in our experiments that data replication often does not happen for unconditional models, while it is common in the text-conditional case. Motivated by our findings, we then propose several techniques for reducing data replication at both training and inference time by randomizing and augmenting image captions in the training set. Code is available at https://github.com/somepago/DCR .

1 Introduction

A major hazard of diffusion models is their ability to produce images that replicate their training data, often without warning to the user [Somepalli et al., 2022, Carlini et al., 2023]. Despite their risk of breaching privacy, data ownership, and copyright laws, diffusion models have been deployed at the commercial scale by subscription-based companies like midjourney , and more recently as offerings within search engines like bing and bard . Currently, a number of ongoing lawsuits [Saveri and Butterick, 2023] are attempting to determine in what sense companies providing image generation systems can be held liable for replications of existing images.

In this work, we take a deep dive into understanding the causes of memorization for modern text-toimage diffusion models. Prior work has largely focused on the role of duplicate images in the training set. While this certainly plays a role, we find that image duplication alone cannot explain much of the replication behavior we see at test time. Our experiments reveal that text conditioning plays a major role in data replication, and in fact, test-time replication can be greatly mitigated by diversifying captions on images, even if the images themselves remain highly duplicated in the training set. Armed with these observations, we propose a number of strategies for mitigating replication by randomizing text conditioning during either train time or test time. Our observations serve as an in-depth guide for both users and builders of diffusion models concerned with copying behavior.

2 Related work

Memorization in generative models. Most insights on the memorization capabilities of generative models are so far empirical, as in studies by Webster et al. [2021] for GANs and a number of studies for generative language models [Carlini et al., 2022, Jagielski et al., 2022, Lee et al., 2022].

We Can Do It!

Recently, Somepalli et al. [2022] investigated data replication behaviors in modern diffusion models, finding 0.5-2% of generated images to be partial object-level duplicates of training data, findings also mirrored in Carlini et al. [2023]. Yet, what mechanisms lead to memorization in diffusion models, and how they could be inhibited, remains so far uncertain aside from recent theoretical frameworks rigorously studying copyright issues for image duplication in Vyas et al. [2023].

Removing concepts from diffusion models. Mitigations deployed so far in diffusion models have focused on input filtering. For example, Stable Diffusion includes detectors that are trained to detect inappropriate generations. These detectors can also be re-purposed to prevent the generation of known copyrighted data, such as done recently in midjourney , which has banned its users from generating photography by artist Steve McCurry, due to copyright concerns [Chess, 2022]. However, such simple filters can be easily circumvented [Rando et al., 2022, Wen et al., 2023], and these band-aid solutions do not mitigate copying behavior at large. A more promising approach deletes entire concepts from the model as in Schramowski et al. [2023] and Kumari et al. [2023], yet such approaches require a list of all concepts to be erased, and are impractical for protecting datasets with billions of diverse images covering many concepts.

3 How big of a problem is data replication?

Somepalli et al. [2022] and Carlini et al. [2023] have shown that diffusion models can reproduce images from their training data, sometimes to near-perfect accuracy. However, these studies induce replication behavior by prompting the model with image captions that are directly sampled from the LAION dataset - a practice that amplifies the rate of replication for scientific purposes. We study the rate at which replication happens with real-world user-submitted prompts. This gives us a sense of the extent to which replication might be a concern for typical end-users.

We randomly sample 100K user-generated captions from the DiffusionDB [Wang et al., 2022] dataset and generate images using Stable Diffusion 2.1 [Rombach et al., 2021, Stability AI, 2022]. We use SSCD features [Pizzi et al., 2022] to search for the closest matches against these images in a subset of the training dataset. Note that SSCD was found to be one of the best metrics for detecting replication in Somepalli et al. [2022], surpassing the performance of CLIP [Radford et al., 2021]. We compare each generated image against ∼ 400 million images, roughly 40% of the entire dataset.

We find that ∼ 1200 images ( 1 . 2% ) have a similarity score above 0.5, indicating these may be duplicates. Note that this is likely an underestimation of the true rate, as our SSCD-based search method is unlikely to find all possible matches. We show several of these duplicates in Figure 1. Sometimes, the captions precisely describe the image content (for example, ' Jake Gyllenhaal '). However, we also discover instances where captions do not mention the content from the generated image. For example, in the first duplicate, the user caption is ' flower made of fire, black background, art by artgerm ', and the LAION caption is ' Colorful Cattleya Orchids (2020) by JessicaJenney '. All the user and LAION captions for Figure 1 are in Appendix B.2. Several duplicates identified by SSCD also have high similarity scores due to the simple texture of the generated image. We include these SSCD false positives and other examples in the Appendix.

4 Experimental Setup

A thorough study of replication behavior requires training many diffusion models. To keep costs tractable, we focus on experiments in which large pre-trained models are finetuned on smaller

Real

Generated

Generation Duplication w.r.t Clusters

Generation Duplication w.r.t Clusters

1.0

2

1

Train

1.0

(Train,

0.8

3

(Train,

0.8

0.6

0.6

Similarity

Similarity

0.4

0.4

2

0.2

0.2

UP

UP

Text

Text

0.0

0.0

3

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

Visual Similarity (Train, Gen)

Visual Similarity (Train, Gen)

ERST

ERST

StableDiffusionv1.4

Stable Diffusion v2.1

datasets. This process reflects the training of Stable Diffusion models, which are pre-trained on LAION and then finetuned in several stages on much smaller and more curated datasets, like the LAION Aesthetics split.

Datasets: We use Imagenette 1 , which consists of 10 classes from Imagenet [Deng et al., 2009] as well as two randomly sampled subsets of 10 , 000 and 100 , 000 images from LAION-2B [Schuhmann et al., 2022] for our experiments. The LAION subsets, which we denote as LAION-10k and LAION100k, include captions, while the Imagenette data is uncaptioned. For some experiments, we use BLIP v1 [Li et al., 2022] to generate captions for images when needed. We provide details for LAION subset construction in the Appendix.

Architecture & Training: We use the StableDiffusion-v2.1 checkpoint as a starting point for all experiments. Unless otherwise noted, only the U-Net [Ronneberger et al., 2015] part of the pipeline is finetuned (the text and auto-encoder/decoder components are frozen) as in the original training run, and we finetune for 100000 iterations with a constant learning rate of 5 e -6 and 10000 steps of warmup. All models are trained with batch size 16 and image resolution 256 . We conduct evaluations on 4000 images generated using the same conditioning used during model training. Refer to Appendix for complete details on the training process.

Metrics: Weuse the following metrics to study the memorization and generation quality of finetuned models. Frechet Inception Distance (FID) [Heusel et al., 2017] evaluates the quality and diversity of model outputs. FID measures the similarity between the distribution of generated images and the distribution of the training set using features extracted by an Inception-v3 network. A lower FID score indicates better image quality and diversity.

Somepalli et al. [2022] quantify copying in diffusion models using a similarity score derived from the dot product of SSCD representations [Pizzi et al., 2022] of a generated image and its top-1 match in the training data. They observe that generations with similarity scores greater than 0 . 5 exhibit strong visual similarities with their top-1 image and are likely to be partial object-level copies of training data.

Given a set of generated images, we define its dataset similarity score as the 95 -percentile of its image-level similarity score distribution. Note that measuring average similarity scores over the whole set of generated images is uninformative, as we are only interested in the prevalence of replicated images, which is diluted by non-replicated samples. For this reason, we focus only on the similarity of the 5% of images with the highest scores. Please refer to Appendix for a longer discussion on this metric design.

5 Data Duplication is Not the Whole Story

Existing research hypothesizes that replication at inference time is mainly caused by duplicated data in the training set [Somepalli et al., 2022, Carlini et al., 2023, Webster et al., 2023]. Meanwhile,

0

5

10

15

20

Data duplication factor

0

.

64

0

.

66

0

.

68

0

.

70

Similairty scores

17

18

FID

LAION-10k

0

5

10

15

20

Data duplication factor

0

.

5

0

.

6

0

.

7

Similairty scores

18

19

20

FID

Imagenette

data replication has been observed in newer models trained on de-duplicated data sets [Nichol, 2022, Mostaque, 2022]. Our goal here is to quantify the extent to which images are duplicated in the LAION dataset, and understand how this impacts replication at inference time. We will see that data duplication plays a role in replication, but it cannot explain much of the replication behavior we see.

5.1 How Much Duplication is Present in LAION?

In order to understand how much data duplication affects Stable Diffusion, we first identify clusters of duplicated images in LAION. We use 8M images from LAION-2B for our duplication analysis. Since duplicates are often duplicated many times, even a small subset of LAION is sufficient to identify them. Note that we do not use a random subset, but instead a consecutive slice of the metadata. Since the metadata is structured to provide URLs from similar domains together, this makes us likely to find large clusters.

We use SSCD [Pizzi et al., 2022] to compute the 8 M × 8 M similarity matrix between all images within a slice of metadata. We threshold the similarity scores, only keeping ones above 0 . 7 to identify images that are nearly identical. Then, we identify each connected component in the sparse similarity matrix and interpret it as representing a duplicated image. We show several of these clusters in the Appendix. We only consider clusters with at least 250 samples to ensure the images are duplicated enough times, which leaves ∼ 50 clusters with a total of around 45 K images.

To measure how similar captions are within a cluster, we first compute CLIP text features [Radford et al., 2021]. Then, we compute the similarity between captions using the dot product of CLIP text features multiplied by the unigram-Jaccard similarity between the captions of a cluster and finally compute the median value. We use the unigram-Jaccard similarity in addition to CLIP, since it better captures word-level semantics. We select 40 captions from each cluster and feed them to Stable Diffusion. Finally, we determine if the generated images are replicas of their source cluster again using SSCD.

Results are shown in Figure 2. We observe that Stable Diffusion v1.4 shows high pixel-level replication (reflected in SSCD scores > 0.4) for some duplicated clusters, but only when caption similarity within the cluster is also high. In contrast, since the dataset for Stable Diffusion v2.1 was de-duplicated before training, it shows nearly no test-time replication for our identified clusters [Mostaque, 2022]. Despite being de-duplicated, Stable Diffusion v2.1 still exhibits memorization when scanning through a larger portion of the dataset, as we observed previously in Figure 1, showing that replication can happen even when the duplication rate in the training set is greatly reduced. In both cases, we see that image duplication is not a good predictor of test-time replication.

5.2 Controlled Experiments with Data Duplication

We train diffusion models with various levels of data duplication factor (ddf) , which represents the factor by which duplicate samples are more likely to be sampled during training. We train each model for 100 k iterations and evaluate similarity and FID scores on 4000 generated samples.

Figure 3 contains results for LAION-10k and Imagenette. We observe that increased duplication in the training data tends to yield increased replication during inference. The relationship between data duplication and similarity scores is not straightforward for LAION-10k . As the duplication factor increases, similarity scores rise again until reaching a data duplication factor ddf of 10, after which they decrease. Regarding FID scores, we find that a certain level of data duplication contributes to

improving the scores for models trained on both datasets, possibly because FID is lowest when the dataset is perfectly memorized (if we infer with all training data captions).

Other Observations from the Literature. Somepalli et al. [2022] found that unconditional diffusion models can exhibit strong replication when datasets are small, despite these training sets containing no duplicated images. Clearly, replication can happen in the absence of duplication . As the training set sizes grow ( ∼ 30 k), the replication behaviors seen in Somepalli et al. [2022] vanish, and dataset similarity drops, even when the number of epochs is kept constant. One might expect this for large enough datasets. However, models trained on the much larger LAION-2B dataset do exhibit replication, even in the case of Stable Diffusion 2.1 (Figure 1) which was trained on (at least partially) de-duplicated data. We will see below that the trend of replication in diffusion models depends strongly on additional factors, related especially to their conditioning and to dataset complexity .

Takeaway: When training a diffusion model, it is crucial to carefully manage data duplication to strike a balance between reducing memorization and ensuring high-fidelity of generated images. However, controlling duplication is not enough to prevent replication of training data.

5.3 The Effect of Model conditioning

To understand the interplay between model conditioning and replication, we consider four types of text conditioning:

Fixed caption: All data points are assigned the same caption, An image .

Class captions: Images are assigned a class-wise caption using the template An image of <class-name> .

Blip/Original captions: Each point is trained on the original caption from the dataset (LAION10k ) or a new caption is generated for each image using BLIP [Li et al., 2022] (Imagenette).

Random captions: A random sequence of 6 tokens is sampled from the vocabulary used to uniquely caption each image.

By varying caption scenarios, we transition from no diversity in captions ('fixed caption') case to completely unique captions with no meaningful overlap ('random caption') case. We again finetune on the Imagenette dataset, now using these caption types. As a baseline, we consider the pretrained Stable Diffusion model without any finetuning. Figure 4 (left) shows the dataset similarity among the baseline and models finetuned using the different caption types.

We observe that the finetuned models exhibit higher similarity scores compared to the baseline model. Furthermore, the level of model memorization is influenced by the type of text conditioning. The 'fixed caption' models exhibit the lowest amount of memorization, while the 'blip caption' models exhibit the highest. This indicates that the model is more likely to memorize images when the captions are more unique or have high perplexity. However, the model does not exhibit the highest level of memorization when using 'random captions', meaning that captions should also be correlated with image content in order to help the model retrieve an image from its memory maximally.

Training the text encoder. So far, the text encoder has been frozen during finetuning. We can amplify the impact of conditioning on replication by training the text encoder during finetuning. In Fig. 4 (right), we observe a notable increase in similarity scores across all conditioning cases when the text encoder is trained. This increase is particularly prominent in the cases of 'blip captioning' and 'random captioning'. These findings support our hypothesis that the model is more inclined to remember instances when the captions associated with them are highly specific or have high perplexity, or even unique, keys.

Takeaway: Caption specificity is closely tied to data duplication. Highly specific (or high perplexity) captions act like keys into the memory of the diffusion model that retrieve specific data points.

5.3.1 The Impact of Caption vs. Image Duplication

In this section, we control separately for duplication of images and duplication of their captions to better understand how the two interact.

fixed capt

class capt

blip capt

random capt

Text conditioning style (Imagenette)

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Similairty scores

pretrained SD

finetuned model

False

True

0

.

0

0

.

2

0

.

4

0

.

6

Similairty scores

fixed capt

class capt

blip/og capt

random capt

In the case of full duplication , both the image and its caption are replicated multiple times in the training data. On the other hand, partial duplication involves duplicating the image multiple times while using different captions for each duplicate(although the captions may be semantically similar). To study the partial duplication scenario, we generate 20 captions for each image using the BLIP model for both Imagenette and LAION-10k datasets. For the full-duplication case in the LAION-10k experiments, we keep the original caption from the dataset.

We present the results on LAION-10k and Imagenette in Figure 5. We investigate how dataset similarity changes for both full and partial image-caption duplication at varying levels of duplication. Overall, our findings demonstrate that partial duplication consistently leads to lower levels of memorization compared to full duplication scenarios.

In Figure 5 (left), we compare the similarity scores of several models: the pretrained checkpoint, a model finetuned without any data duplication, a model finetuned with full duplication ( ddf=5 ), and a model finetuned with partial duplication ( ddf=5 ). We include dashed horizontal lines representing the background self-similarity computed between the dataset and itself. In the Imagenette case, models trained without duplication and with partial duplication exhibit dataset similarity below the baseline value, indicating a lower level of memorization. In contrast, the model trained with full duplication demonstrates higher levels of memorization compared to both the baseline and other cases. In the LAION-10k experiments, the model trained without duplication surpasses the training data similarity baseline. This observation raises the question of whether the memorization is inherited from the pretrained model, considering it is also trained on the larger LAION-2B dataset. However, when we compute the similarity scores on the pretrained checkpoint, we observe significantly lower values, indicating that the observed memorization is acquired during the fine-tuning process.

In Figure 5 (middle, right), we analyze how the similarity scores and FID scores vary at different data duplication factors (ddf) for full and partial data duplication. As the ddf increases, we observe an increase in memorization for models trained with full duplication. However, for partial

pretrained SD

no dupl.

image + capt. dupl.

image dupl.

LAION-10k

Imagenette

Dataset

0

.

0

0

.

2

0

.

4

0

.

6

Similairty scores

Imagenette

LAION-10k

0

5

10

15

20

Data duplication factor

0

.

55

0

.

60

0

.

65

0

.

70

Similairty scores

LAION-10k

0

5

10

15

20

Data duplication factor

18

20

FID

LAION-10k

no dupl.

img+cap dupl.

img dupl.

50000

100000

Length of training (in iters)

0

.

4

0

.

5

0

.

6

0

.

7

Similairty scores

LAION-10k

50000

100000

Length of training (in iters)

17

18

19

20

FID

LAION-10k

25

50

75

100

Amount of training data (x 10

3

)

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

Similairty scores

14

.

5

15

.

0

15

.

5

16

.

0

FID

LAION-100k

duplication, dataset similarity actually decreases with increased duplication. In our previous analogy, we now have multiple captions, i.e. keys for each duplicated image, which inhibits the memorization capabilities of the model. However, this memorization reduction comes at the cost of a moderate increase in FID at higher duplication levels.

Takeaway: Compared to full duplication, partial duplication substantially mitigates copying behavior, even as duplication rates increase.

6 Effect of the Training Regimen

In this section, we examine how the training process and data complexity influence the degree of memorization.

Length of training. Training for longer results in the model seeing each data point more times, and may mimic the impacts of duplication. In Figure 6 (left), we quantify the increase in dataset similarity at different training iterations. Notably, the image-only duplication model consistently exhibits lower similarity scores across epochs, while the image & caption duplication model achieves the highest values, showing that caption diversity for each duplicated image significantly slows down memorization. We show matching FID scores in Figure 6 (middle).

Quantity of training data. Somepalli et al. [2022] demonstrate that diffusion models, specifically DDPM [Ho et al., 2020] models without text conditioning, exhibit a tendency to replicate the training data when trained on small datasets. In this study, we extend this analysis to text-conditioned models. To conduct this investigation, we now randomly sample a subset of 100 , 000 images from LAION-2B. We then train using different fractions of this dataset, keeping other hyperparameters constant including the number of training iterations. The results of this experiment are found in Figure 6 (right). We observe that increasing the quantity of training data generally leads to lower similarity and FID scores. This aligns with the intuition that a larger and more diverse training set allows the model to generalize better and produce more varied outputs.

Image Complexity. Throughout our experiments, we consistently observed a higher level of testtime replication in LAION-10k models compared to Imagenette models, even when trained with the same levels of data duplication. We put forth the hypothesis that this discrepancy arises due to the inherent diversity present in the LAION-10k dataset, not only in terms of images but also in terms of image complexity. To quantify image complexity, we employ two metrics. Histogram entropy measures entropy within the distribution of pixel intensities in an image, and JPEG compressibility (at JPEG quality 90 ), measures the size of images after compression. We resize and crop all images to the same resolution before computing these metrics.

Figure 7 (right) presents the distributions of entropy scores for LAION-10k and Imagenette. LAION10k exhibits a wide range of data complexity, while Imagenette, predominantly comprised of complex real-world images, is characterized by higher complexity. Motivated by this observation, we explore the relationship between similarity scores and complexity of the top training point matched to each generation of a diffusion model trained on LAION-10k without any duplication. Remarkably, we

0

2

4

6

Complexity (Entropy)

0

.

0

0

.

2

0

.

4

0

.

6

Density

High Sim

Low Sim

0

10

20

30

Complexity (Compression)

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

Density

High Sim

Low Sim

0

2

4

6

Complexity (Entropy)

0

.

0

0

.

5

1

.

0

1

.

5

Density

Imagenette

LAION-10k

discover a statistically significant correlation of -0 . 32 / -0 . 29 (with p-values of 8 e -98 / 7 e -80 ) for the entropy and compression metrics, respectively. We provide results for other duplication settings in the supplementary material, where we also find significant correlation. We present density plots illustrating the distributions of both complexity metrics for high similarity points (with scores > 0 . 6 ) versus low similarity in Figure 7 (left, middle). The clear separation between the distributions, observed across both complexity metrics, strongly suggests that diffusion models exhibit a propensity to memorize images when they are 'simple'.

Takeaway: Choosing the optimal length of training, quantity of training data, and training setup is a trade-off between model quality and memorization. When images do get memorized in the absence of training data duplication, they are likely to be 'simple' in structure.

7 Mitigation strategies

We have seen that model conditioning plays a major role in test-time replication, and that replication is often rare when image captions are not duplicated. Armed with this observation, we formulate strategies for mitigating data replication by randomizing conditional information. We study both training time mitigation strategies and inference time mitigation strategies. Training strategies are more effective, whereas inference-time mitigations are easily retrofitted into existing diffusion models. We experiment with randomizing the text input in the following ways:

Multiple captions (MC). We use BLIP to make 20 captions for each image and we randomly sample all of them (plus the original) when training. This is for train time only.

Gaussian noise (GN). Add a small amount of Gaussian noise to text embeddings.

Random caption replacement (RC). Randomly replace the caption of an image with a random sequence of words.

Random token replacement & addition (RT). Randomly replace tokens/words in the caption with a random word, or add a random word to the caption at a random location.

Caption word repetition (CWR). Randomly choose a word from the given caption and insert it into a random location in the caption.

Random numbers addition (RNA). Instead of adding a random word that might change the semantic meaning of the caption, we add a random number from the range { 0 , 10 6 } .

In this section, we present the effectiveness of various mitigation strategies, summarized in Table 1. During train time, we find the multiple-caption strategy to be the most effective, which substantially increases caption diversity among duplicates. This matches our analysis in Fig. 5. At inference time strategies, we find that we can still inject a shift in the text prompt through random token addition/replacement. Such a shift again disrupts the memorized connection between caption and image. We verify based on FID and Clipscore [Hessel et al., 2021] that all mitigations have a minimal effect of model performance.

Images generated from models trained/tested with the best performing strategies can be found in Figure 8. For train mitigations we show our finetuned models, whereas for inference time we show mitigation strategies applied directly to Stable Diffusion. We show four images for each strategy, displaying the memorized image, generation from the regular model and from the mitigated model

Train time - Multiple Captions

Inference time - Random token / number addn.

Train

data

Gen. w.

mitigation

Generation

UP

EAST

UP

ERS

in each column. We fix the generation seed in each column. Overall, we find these mitigations to be quite effective in reducing copying behavior, even in large modern diffusion models. We refer the reader to the appendix regarding the hyperparameters used in the generation of Tab. 1 and the captions used for creating Fig. 8.

8 Recommendations for a safer model

Why do diffusion models memorize? In this work, we find that, in addition to the widely recognized importance of de-duplicating image data, conditioning and caption diversity also play fundamental roles. Moreover, we find in Fig. 5 that memorization can decrease even when duplication increases, as long as captions are sufficiently diverse. These findings help to explain why large text-conditional models like Stable Diffusion generate copies. Finally, we collect our observations into a series of recommendations to mitigate copying.

Before training: Identify large clusters of image data with significant object overlap, as measured by specialized copy detection models such as SSCD, and collapse them as described in Sec. 5.1. Likewise, clusters of caption overlap are potentially problematic, and duplicate captions can be resampled, for example using BLIP. In some cases, these clusters may need to be hand-curated because some concepts, like famous paintings or pictures of the Eiffel Tower, are acceptable duplications that the model may copy. Another avenue is to exclude images with limited complexity, see Sec. 6.

During training: We find training with partial duplications, where captions for duplicate images are resampled, to be most effective in reducing copying behavior, see Sec. 7. It may be worthwhile to use a low bar for duplication detection, as near-duplicates do not have to be removed entirely, only their captions resampled.

After training: Finally, after training, inference-time strategies can further reduce copying behavior, see Sec. 7. Such mitigations could be provided either as a user-facing toggle, allowing users to resample an image when desired, or as rejection sampling if a generated image is detected to be close to a known cluster of duplications or a match in the training set.

While text conditioning has also been identified as a crucial factor in dataset memorization for the diffusion model in our analysis, it is important to acknowledge that other factors may also influence the outcome in complex ways. Therefore, we recommend that readers thoroughly evaluate the

effectiveness of the proposed mitigation strategies within the context of their specific use cases before deploying them in production.

Future Directions: This research represents a first investigation into the influence of captions on memorization within diffusion models. We posit that examinations on broader scales, involving larger datasets and models, will further solidify our findings. Moreover, while our adopted similarity metric serves its purpose, we anticipate and encourage future studies to refine this metric or introduce alternatives grounded in a more robust theoretical framework on memorization.

References

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. arxiv:2202.07646[cs] , February 2022. doi: 10.48550/arXiv.2202.07646. URL http://arxiv.org/abs/2202.07646 .

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. arxiv:2301.13188[cs] , January 2023. doi: 10.48550/arXiv.2301.13188. URL http://arxiv. org/abs/2301.13188 .

David Chess. Some light infringement?, December 2022. URL https://ceoln.wordpress.com/ 2022/12/16/some-light-infringement/ .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A referencefree evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring Forgetting of Memorized Training Examples. arxiv:2207.00099[cs] , June 2022. doi: 10.48550/arXiv.2207.00099. URL http://arxiv.org/abs/2207.00099 .

Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating Concepts in Text-to-Image Diffusion Models. March 2023. URL https://arxiv.org/ abs/2303.13516v2 .

Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do Language Models Plagiarize? arxiv:2203.07618[cs] , March 2022. doi: 10.48550/arXiv.2203.07618. URL http://arxiv.org/ abs/2203.07618 .

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Emad Mostaque. Some more from first batch. Lots of optimisation to do, took about an hour of playing about. Prompts here: https://t.co/4soGak9op0 negative important for 2.0 given how we flatten distribution of latents with dedupe etc Embeddings will make it easier out of the box https://t.co/sZxhrk1v8J, November 2022. URL https://twitter.com/EMostaque/status/ 1596907328548139008 .

Alex Nichol. Dall·e 2 pre-training mitigations, 2022. URL https://openai.com/research/ dall-e-2-pre-training-mitigations . Accessed: 2023-05-05.

Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A Self-Supervised Descriptor for Image Copy Detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14532-14542, 2022. URL https://openaccess.thecvf.com/content/CVPR2022/html/Pizzi_A_ Self-Supervised_Descriptor_for_Image_Copy_Detection_CVPR_2022_paper.html .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.

Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-Teaming the Stable Diffusion Safety Filter. arxiv:2210.04610[cs] , October 2022. doi: 10.48550/arXiv.2210. 04610. URL http://arxiv.org/abs/2210.04610 .

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models, 2021.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , pages 234-241. Springer, 2015.

Joseph Saveri and Matthew Butterick. Stable Diffusion Litigation, 2023. URL https:// stablediffusionlitigation.com/ .

Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. arxiv:2211.05105[cs] , April 2023. doi: 10.48550/arXiv.2211.05105. URL http://arxiv.org/abs/2211.05105 .

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models. arxiv:2212.03860[cs] , December 2022. doi: 10.48550/arXiv.2212.03860. URL http://arxiv.org/abs/2212.03860 .

Stability AI. Stable Diffusion v2.1 and DreamStudio Update, December 2022. URL https: //stability.ai/blog/stablediffusion2-1-release7-dec-2022 .

Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable Copyright Protection for Generative Models. arxiv:2302.10870[cs, stat] , February 2023. doi: 10.48550/arXiv.2302.10870. URL http:// arxiv.org/abs/2302.10870 .

Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210.14896 [cs] , 2022. URL https://arxiv.org/abs/2210.14896 .

Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. This Person (Probably) Exists. Identity Membership Attacks Against GAN Generated Faces. arxiv:2107.06018[cs] , July 2021. doi: 10.48550/arXiv.2107.06018. URL http://arxiv.org/abs/2107.06018 .

Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.12733 , 2023.

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. February 2023. URL https://arxiv.org/abs/2302.03668v1 .

A Broader Impact

Text-to-image diffusion models are prone to violating intellectual property rights and causing harm to artists, even unbeknownst to the models' users. Given the rapidly increasing popularity of these models and also the likelihood that users will continue to generate and deploy their images whether or not copying behaviors are addressed, mitigating such behavior can prevent real-world harm. We take the first step towards mitigation by building up an understanding of how and why diffusion models copy their training data. Moreover, we show how to build text-to-image diffusion models which simultaneously produce high-quality images and at the same time do not replicate their training data nearly as often as popular existing models. Another important avenue for preventing copying is constructing detection pipelines so that we can identify copies of training samples before they are deployed, as in Somepalli et al. [2022].

B User Caption Study

B.1 Stable Diffusion User Matches False Positives

LAloN Caption: mosaic color matrix squares (vector)

User Caption:large squaresof different colors

LAION

Generated

LAloN Caption:Seamless rainbow vertical stripes color line art vector background

User Caption: Illustration. a series of vertical stripes in different colors. byAlanMooredistorted

LAloN Caption:Coffee beans-StockPhoto

User Caption: cheesy pop art of coffee beans, extreme detail

LAloN Caption: mosaic color matrix squares (vector)

User Caption:large squaresof different colors

LAlON Caption: File:BL2-The-Sheriff-sketches-2.jpg User Caption: fantasy rogue armor, all-metal, official concept art, orthographic views

In Figure 9, we show failure cases for SSCD match from the LAION subset to generated images from user captions. Several false positives are reasonable errors, caused by either high style similarity or simple texture of the generated images. A few of the errors are also caused for animated images as shown in the last row. We also observe a few cases where generated, and LAION images show significant domain shift from real images. This may be due to the training data of SSCD, which mostly consisted of real images Pizzi et al. [2022].

B.2 User and LAION Captions for Images

The captions for the generated images using user prompts, and corresponding LAION matches for Figure 1 are shown below in order -

User Caption "flower made of fire, black background,art by artgerm" LAION Caption '"""""""Colorful Cattleya Orchids"""" (2020) by JessicaJenney"""'

User Caption "the moon landing underwater " LAION Caption "H/O: Buzz Aldrin on the Moon"

Cluster 13

555 images

Cluster 20

466 images

Cluster 25

382 images

Cluster 27

345images

User Caption "rosie the riveter with a catfish head" LAION Caption "We can do it! Canvas Print"

User Caption "pubg splash art advertisement by masashi kishimoto / hiroya oku, hayao miyazaki, concept art, vibrant colors, 8 k "

"Minor kills brother for not letting him play PUBG game in Thane"

User Caption "the super mario logo " LAION Caption "Thumb super mario run logo"

User Caption "a film still from joker ( 2 0 1 9 ) " LAION Caption "Joaquin Phoenix as THE Joker in Comic Villain Casting"

User Caption "cyberpunk tufting rug "

"Liquid Pattern Print Design 04 Area Rug"

User Caption "pubg splash art advertisement by masashi kishimoto / hiroya oku, hayao miyazaki, concept art, vibrant colors, 8 k " LAION Caption "Minor kills brother for not letting him play PUBG game in Thane"

C LAION Clusters

In Figure 10, we show a few of the clusters discovered using our proposed approach discussed in Section 5.1. As shown several of the images share high visual similarities with each other. The number of images discovered in each cluster is also shown. Note that the number is very likely an underestimate since we only use an 8 M subset of the LAION dataset. A few of the clusters are also obtained due to broken links or similar placeholder images. A few clusters are also obtained for similar bar graphs or line charts. This shows LAION consists of several duplicates, which can be easily discovered using our approach.

D Extended experimental settings

Unless otherwise specified, all the models are trained for 100 k iterations with a batch size of 16 . We used Adam optimizer for training with β 1 = 0 . 9 and β 2 = 0 . 999 and weight decay = 1 e -2 We used one RTX-A6000 per model and it took about 24 hours to train. For inference, we used RTX-A5000 and it took approximately 5 hours to create enough generations to compute our metrics.

0

5

10

15

20

Data duplication factor

0

.

4

0

.

6

0

.

8

Similairty scores

Metric percentile (Img + Cap dupl.)

percentile

50

90

95

97

99

0

5

10

15

20

Data duplication factor

0

.

2

0

.

4

0

.

6

0

.

8

Similairty scores

Metric percentile (Img dupl.)

percentile

50

90

95

97

99

To analyze various factors in this paper, we used approximately 5000 GPU hours for training and another 1000 GPU hours for inference.

E Discussion on similarity metric design

Occasional memorization, even in a small percentage of generations, can pose problems-precisely the pattern we've observed. To capture this behavior effectively, we measure the sim score of the top 95% of samples. Using the average or median, in contrast, could fail to detect significant memorization in a few instances. Our choice of the 95% percentile is based on observations by Somepalli et al., who identified around 2% of memorization. Adjusting for fine-tuning on a smaller dataset, we found 95% to be a suitable benchmark.

We have performed an ablation analysis on LAION-10k and full duplication across different data duplication factors and the results are presented in Fig. 11 (Left). We see 95, 97, and 99 percentiles follow similar trends. Median scores remain relatively constant, as expected, due to memorization affecting only a small percentage of images, as previously noted by Somepalli et al. Similar trends appear in partial duplication experiments, as evident in Fig. 11 (Right). Hence we proceeded with 95% value of similarity score distribution as the metric in this paper.

We present how similar or different the images look at different values of SSCD similarity scores in Fig. 12. We also present the similarity score histogram for the model trained with full duplication with ddf = 10 in Fig. 11 (Left). We also present a sample Precision-Recall curves on LAION-10k full duplication case in Fig. 11 (Right).

Match

0.97

0.86

0.81

0.75

0.64

0.54

0.45

0.33

0.28

0.13

adidas

adidas

F Why do diffusion models copy? Extended Results

In this section, we explore factors contributing to replication in diffusion models. We present additional results, including some figures from the main paper for completeness.

Model conditioning. As shown in Fig. 4 (Left), higher uniqueness in conditioning captions leads to increased similarity scores, with Blip caption conditioning yielding the highest score. In Fig. 17 (Left), FID scores for fine-tuned models follow the same trends we have seen in the main paper, with the blip caption model exhibiting the lowest FID score.

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Similarity Scores

0

1

2

3

Density

0

5

10

15

20

Data duplication factor

0

.

77

0

.

78

0

.

79

Precision

0

.

63

0

.

64

0

.

65

0

.

66

0

.

67

Recall

LAION-10k (Prec vs Recall)

Image Duplication vs Image + Caption Duplication In Fig. 14, we analyze similarity and FID scores for models trained with varying duplication levels ( ddf ) on LAION-10k and Imagenette. Both datasets show increased similarity and FID scores with higher ddf . However, the impact of image duplication versus image + caption duplication differs. This disparity can be attributed to the diversity of captions used in image duplication for LAION-10k and the similarity of captions in Imagenette, potentially collapsing into image + caption duplication.

Training Regimen - Length of Training In Fig. 15, we examine similarity and FID scores during the training process on LAION-10k and Imagenette. We consider scenarios with no duplication, image duplication, and image + caption duplication. Longer training consistently leads to higher similarity and FID scores. However, image duplication consistently yields lower similarity scores compared to image + caption duplication on both datasets.

Training Regimen - Text Encoder Training In Fig. 4 (Right), we analyze the impact of text encoder training on similarity and FID scores for models trained with caption conditioning and various types of duplication on Imagenette and LAION-10k . Training the text encoder increases similarity scores while reducing FID scores in both datasets. In Fig. 17 (Right), we observe the effect of text encoder training on similarity scores for models trained with different conditioning types on Imagenette, specifically with image + caption duplication ( ddf=5 ), and we see similar trends as of Fig. 4 (right) even in the presence of data duplication.

Training Regimen - Image Complexity In Fig. 18, we present distributions of self-similarity, entropy, and compression metrics for LAION-10k and Imagenette. LAION-10k exhibits lower self-similarity, indicating a distinct data distribution. However, complexity distributions are slightly shifted to the left, suggesting LAION-10k is marginally simpler than Imagenette.

Furthermore, we provide comprehensive findings on the correlation between similarity scores and training data complexity metrics in Tab. 2. We extensively evaluate models trained on LAION-10k without duplication, with image duplication, and with image + caption duplication, including cases with ddf=5 and ddf=20 . Notably, we consistently observe a statistically significant correlation in all scenarios. Intriguingly, the correlation is stronger when image + caption duplication is present compared to cases with only image duplication.

no dupl.

img+cap dupl.

img dupl.

0

5

10

15

20

Data duplication factor

0

.

55

0

.

60

0

.

65

0

.

70

Similairty scores

LAION-10k

0

5

10

15

20

Data duplication factor

18

20

FID

LAION-10k

0

5

10

15

20

Data duplication factor

0

.

5

0

.

6

0

.

7

0

.

8

Similairty scores

Imagenette

0

5

10

15

20

Data duplication factor

18

20

22

FID

Imagenette

0

.

Similairty scores

0

0

0

.

.

.

7

6

5

4

LAION-10k

50000

100000

Length of training (in iters)

50000

100000

Length of training (in iters)

50000

100000

Length of training (in iters)

Imagenette

50000

100000

Length of training (in iters)

no dupl.

img+cap dupl.

img dupl.

False

True

Text encoder trained

0

.

0

0

.

2

0

.

4

0

.

6

Similairty scores

LAION-10k

False

True

Text encoder trained

0

5

10

15

FID

LAION-10k

False

True

Text encoder trained

0

.

0

0

.

2

0

.

4

0

.

6

Similairty scores

Imagenette

False

True

Text encoder trained

0

5

10

15

FID

Imagenette

G Misc. results.

We recreated the main paper Fig 2 using Roberta and presented it Fig. 19. The findings stay the same.

H Mitigation strategies

H.1 Training details

Except for the affect of length of training on similairity scores experiment, we trained all the diffusion models for 100 k iterations, including the models trained with a 'mitigation" hueristic. The following are the additional hyperparameters relevent to a given strategy.

Multiple Captions (MC): We sample 20 captions from BLIP for all training images. For LAION-10k , an additional original caption is included, resulting in a total of 21 captions.

Gaussian Noise (GN): We sample random multivariate Gaussian noise ∼ N (0 , I ) , multiply it by a factor of 0.1, and add it to the latent CLIP representations of the caption.

Random Caption Replacement (RC): With a probability of 0.4, we replace the caption of an image with a random sequence of words.

Random Token Replacement & Addition (RT): With a probability of 0.1, we either replace tokens/words in the caption with a random word or add a random word at a random location. This step is performed twice for train time experiments and four times for inference time experiments.

no dupl.

LAION-10k

20

19

FID

18

17

img+cap dupl.

Imagenette

0

.

55

Similairty scores

0

0

0

.

.

.

50

45

40

img dupl.

22

20

FID

18

fixed capt

class capt

blip capt

random capt

Text conditioning style (Imagenette)

0

10

20

30

40

FID

↓

finetuned model

False

True

Text encoder trained (Imagenette, img+cap dupl.)

0

.

0

0

.

2

0

.

4

0

.

6

Similairty scores

fixed capt

class capt

blip/og capt

random capt

0

.

25

0

.

50

0

.

75

1

.

00

Similarity scores (top-1)

0

2

4

6

8

Density

Imagenette

LAION-10k

0

2

4

6

Complexity (Entropy)

0

.

0

0

.

5

1

.

0

1

.

5

Density

Imagenette

LAION-10k

0

.

0

0

.

1

0

.

2

Complexity (Compression)

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

Density

Imagenette

LAION-10k

0.0

0.2

0.4

0.6

0.8

Visual Similarity (Train, Gen)

0.0

0.2

0.4

0.6

0.8

1.0

Text Similarity (Train, Train)

Generation Duplication w.r.t Clusters

Caption Word Repetition (CWR): We select a word from the given caption and insert it into a random location within the caption with a probability of 0.4. This step is performed twice for train time experiments and four times for inference time experiments.

Random Numbers Addition (RNA): Instead of adding a random word that may alter the semantic meaning, we add a random number from the range 0 , 10 6 with a probability of 0.4. This step is performed twice for train time experiments and four times for inference time experiments.

H.2 CLIP score and FID tables

In Tab. 1, we presented similarity scores for models trained/inferred with different mitigation strategies. In Tab. 4 and Tab. 3, we present the corresponding CLIP scores and FID scores. Gaussian Noise

strategy compromises on the FID score. Random Token/ Random Number addition during inference strategy compromises on CLIP score the most.

H.3 Extended Qual results

Train time - Multiple Captions

Train

data

Gen. w.

mitigation

Generation

Can we forget images memorized by SD 2.1 with MC strategy? We used different images to show qualitative results for train and test time strategies due to the initial freezing of training data in our project. The image-caption pairs necessary for evaluating test-time mitigation using SD 2.1 were discovered later on.

However, to verify if train test strategies can make models forget the memorized images, we conducted an experiment by training two models, incorporating the additional 5 image-caption pairs (from Table 5 or Fig 8, right side). The first model is trained with ddf=5 and no mitigation, while the second model utilizes ddf=5 with the MultipleCaps (MC) mitigation strategy. The "train-time" results for these 5 images are presented in ?? . Encouragingly, with MC strategy, the model was able to forget 3 out of 5 images. However, the model still retained strong memorization for the prompts 'Mothers influence on her young hippo' and 'Classic Cars for Sale." We believe the need to combine multiple mitigations to address these remaining memorizations effectively.

Inference time results. The original and modified prompts used in creating figure Fig. 8 is shown in Tab. 5. For the generation, we used Stable Diffusion 1.4 with the generator seed=2 , with default settings guidance_scale=7.5 , num_inference_steps=50 .

Train time results. In Fig. 21, we show extended train time qualitative results for all the mitigation strategies. We see 'Multiple Captions (MC)" mitigation works all the time. In contrast, other methods might or might not always be effective.

(a)

(g)

(b)

(c)

(d)

(e)

(f)

(h)

米米米米米米米米烤

米米

YYYYY

YYYYY

YYYYY

YYYYY

ninininin

YYYYY

YYYYY

ninininin

YnY

YYYYY

YYYYA

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aligns with a range of recent works which investigate the tendency of diffusion models to memorize  training data and emit this during sampling, when prompted to, and presents strategies to mitigate this for data-processing, training and inference. Other than previous works, which mainly focused their analyses around duplicate images in the training data and model overfitting (i.e. dataset size, portion of train examples seen during training, …) , this work separately analyzes the influence of both these points and further adds to this the influence of captions (and their duplication) in the training data and complexity of the images in the training dataset. The main findings for these are that highly specific captions for every image promotes duplication which is furthermore also more likely for images which are simple (as measure by pixel histogram-entropies and jpeg-compressability).\n\nSTRENGTHS:\n* The paper separates the main factors of influence during training from each other and bases the findings made for each of these separate categories on rich sets of empirical experiments, making the paper conclusive and interesting\n* The main take-aways are novel and of high importance for the community, especially since the authors center their experiments around fune-tuning a foundational DM, which is what many researchers do nowadays.\n* The key take-aways for every analyzed influencing factor are summarized below the respective sections which makes the paper well structured and raises comprehensibility.\n\nWEAKNESSES:\n* It is totally reasonable that the authors cannot train a foundation model like SD from scratch and therefore focus on finetuning based on these models. However, the exact influence of the pretraining on all the results presented on conclusions taken remains unclear to me after the paper. It would therefore be interesting to also include results of models trained from scratch for all of the presented experiments, since it would provide insights about the generality of the statements made.\n* Related to the above point, I think the title may promise a bit too much, since it would need experiments from scratch (for models such as SD) to choose that title. Maybe sth. Like “Why finetuned diffusion models memorize and …” would be better. What du you think?\n* To me, it is counterintuitive that the model trained with no duplication has a higher dataset similarit score than the one with partial dup. in Fig. 5. If I didn’t miss sth, this is never discussed and is therefore confusing. It might also raise concerns about the significance of the results presented in this part which I think is crucial for the entire paper. Could the authors share their take on this?\n* For a reader not knowing SSCD it may be hard to estimate, how reasonable the chosen newly introduced metrics related to SSCD are, since many of these are based on thresholds. Therefore, it would be great of the authors could add similarity scores to all visual examples showing found duplicates (especially in Fig. 1). \n* In line 91 it is stated that FID would measure image quality and diversity. I don’t agree for diversity since there are many models with low FID scores which actually have very low diversity (many GANs, see e.g. [1,2]). I’d therefore recommend to either additionally include Precision and Recall Metrics [2] in the evaluation or remove diversity here.\n\t\t\n[1] Sauer, A., Chitta, K., Müller, J., & Geiger, A. Projected GANs Converge Faster.\n[2] T Kynkäänniemi, T Karras, S Laine, J Lehtinen, T Aila ,  Improved precision and recall metric for assessing generative models\n\nQUESTIONS:\nHow diverse are BLIP captions? It would be interesting to do the same analysis than made for LAION captions in Sec. 5.1.\n\nLIMITATIONS:\nThere is not dedicated limitations section, however, the limited scope of the experiments, which focus only on finetuning is mentioned in the paper. I’d recommend to add  a dedicated limitations section and once more state this. Also it would be interesting to see this analysis for other types of models such as AR models or GANS (although I know that no comparably powerful models to SD are publicly available as of now). The fact that this misses could also go into such a section", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the issue of data replication in text-to-image diffusion models, specifically focusing on the memorization problem. It mentions that these models, such as Stable Diffusion, are known for unintentionally replicating their training data, which has led to recent lawsuits. The commonly held belief is that duplicated images in the training set contribute to content replication during inference. However, the paper's analysis reveals that the text conditioning of the model also plays a significant role in data replication. The experiments show that unconditional models don't replicate data as frequently as text-conditional models. Building on these findings, the paper proposes several techniques to mitigate data replication during inference by randomizing and augmenting image captions in the training set.\n\nSTRENGTHS:\n- The paper is well-organized and well-written\n- The experiment setup is clear and sound.\n- The originality of resolving the memorization issue of the diffusion model is interesting.\n- The experiment results are convincing.\n\nWEAKNESSES:\n- Even though this paper is focusing on the empirical results and evidence of memorization of the diffusion model, I am still expecting a high level of theoretical understanding. Specifically, does it holds for the general case regardless of the type of SDE, data distribution etc.?\nI think these questions are also important.\n\nQUESTIONS:\n- It may be interesting to show some toy cases in 2 dimensional or 1 dimension in which one can simply play around with how the conditioning and replication will influence the vector field (probabilistic flow) of the generation. It might be a good justification of authors' hypothesis.\n\nLIMITATIONS:\nThe observation is based on the heuristic observation, however, it is fine since it is impossible to have a white-box explanation of a neural network. It would be better to have some guidance in theoretical reasoning.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tries to explain the memorization and copying of training data in diffusion models. The paper establishes a connection between caption duplications and memorization on top of data duplications analyzed in previous works. In their main experiments, the authors fine-tune various SD models on ImageNette/LAION subsets and show that controlling data duplication does not completely prevent memorization and that highly specific captions amplify copying. They also relate memorization to training parameters and data complexity. Finally, the paper introduces several mitigation strategies based on randomizing conditional information.\n\nSTRENGTHS:\n- Overall, I find the experiment design in the paper to be good and got a better understanding of the memorization phenomenon from the paper. I also enjoyed the writing style of the paper. \n - I find the connection between image complexity and copying to be reasonable and believe this is the first time this has been analyzed this way. \n - While previous papers were mostly about analyzing the impact of data duplication on memorization, it makes sense to also control the captions and the paper shows that there exists a connection between the two. \n - The problem setting of the paper is very relevant.\n\nWEAKNESSES:\n- Pretty much all experiments are based on the dataset similarity score which is defined as the 95-percentile of image-level similarities. As this is very important for the validity of the experiments, I believe that it would make a lot of sense to motivate this design choice further. From previous works, the authors quote that an image similarity larger than 0.5 shows strong similarities. In the worst case, the distribution of image similarities could contain slightly less than 5% near duplicates and slightly more than 95% unrelated images which would result in an arbitrarily small dataset similarity but still a decent memorization quote. Maybe the authors could simply demonstrate that the distribution of image similarities is well-behaved and also quantify the number of outliers, eg at a dataset similarity of 0.2, how many of the remaining images above the 95%-quantile actually achieve an image similarity larger than 0.5. Ideally, it would also be great to shortly demonstrate that results look similar with larger percentiles and that the 95-percentile is not critical for the outcome of the experiments. \n- The range of similarity scores in most of the experiments is actually somewhat limited. I do not have a good feeling about the scale of this metric, but for example, the data duplication factor seems to only influence the score in the range 0.64 to 0.70. As this is the 95% quantile and 0.5 can be assumed to be a duplicate, this means that in all cases we have strong duplicates. I do believe the authors that there is a connection between duplicates/captions and copying, however, either the dataset score is not the right metric for this kind of experiment or duplicates/caption can actually only explain a relatively small part of the problem. Also even with the best-performing train time mitigation strategies, the similarity score is 0.42/0.47 so from my understanding this could mean that we still have about 5% of duplicates. \n- The mitigation strategies are somewhat simplistic. I also believe that it is crucial to give a more detailed analysis of the \"minimal effect\" that the mitigations have on model performance. The images on the left half of Figure 8 work well in the sense that they seem to produce the same object that was also on the original image. However, most of the images on the right side of Figure 8 are not convincing to me. For example, I do not see any interaction between the hippo mother and her child (prompt: \"Mothers influence on her young hippo\"), and from a short Google search I would say that the person in the image is not Ann Graham Lotz (prompt: \"Living in the Light with Ann Graham Lotz\") and I could not find any connection between the last image with mitigation and \"hopped up gaming east\". I think mitigating copying is clearly important, but if this prevents the SD model from following the user's prompt it becomes useless. I think Figure 8 should show the results of the best train and test time mitigation strategies for the same images and I would also like to see the results of the best train time defenses on the images on the right side as apparently they seem to be harder. For some images in Figure 16, I also do not find any of the test time defenses to produce reasonable results.\n\nQUESTIONS:\n- There are pretty strong differences between Imagenette and Laion-subsets, do you have also looked at other datasets and do they fall between the two datasets shown in the paper or do the results look completely different there?\n\nLIMITATIONS:\nIn my opinion, one of the largest limitations is that even the best-performing training guidelines/mitigation strategies might not always prevent copying. The paper gives us a dataset similarity result of 0.42/0.47 and claims that in terms of image similarity, 0.50 would be a duplicate. So it is not clear to me that this actually solves copying.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a deep dive into the phenomenon of memorization in Diffusion models. I find that the paper is very well position given the current state of generative ML and the paper does a great job at asking the right questions about various factors concerning the memorization of images. Some of the key takeaways in the paper are really strong, such as randomizing captions, or partial duplication of images, but some others are not very well substantiated. Finally, the authors also conclude with some guidelines and mitigation strategies for practitioners, making this a refreshing read.\n\nThis paper is hard to review in a strengths/weakness pattern because each section has its strengths and weaknesses. First, this paper tries to do a lot. I think that ends up being a weakness of the paper because the high-level message from the abstract and intro gets swayed away through the body making this a very confusing read. Especially in the sections from 3 to 5.3. It is only after that the main narrative of the work catches on. I understand that the authors want to show a lot of experiments and analyses they did, but many of them are not at all central to the key message of the paper.\n\nSTRENGTHS:\n1. This paper tries to break down various factors that are responsible for the memorization of training data by diffusion models.\n2. The discussions in sections 6 and 7 are really valuable to the community.\n3. The paper asks the right questions and also performs experiments to support their hypothesis (more about this in the weaknesses). But overall, the paper serves as a great starting point for understanding the state of memorization in generative models, giving a peak at some thought-provoking questions, and using some of these hypotheses to actually suggest mitigation strategies.\n4. I am curious about the image complexity argument. This is a refreshing take, but also very different from image classification literature where typically hard images are memorized because the model can easily learn the simple ones to classify. It seems that the take for generative models is the opposite. but again there is a distinction between anecdotal arguments of features versus the argument of compressibility and entropy studies here. A good metric potentially would be the embedding similarity as assessed by a CLIP model.\n\nWEAKNESSES:\nMy biggest qualm with this paper is about the metrics. It seems that this paper is based in a delusional world where dataset similarity and FID are the sole indicators of model memorization and model performance. I know that evaluating both generation and memorization are hard problems, but this paper takes these yardsticks way too far, to an extent that following up with some results becomes uncomfortable. And in many sections, I get the sense that hypotheses and conjectures are drawn just to satisfy these so-called oracle notions of memorization and quality.\n\n1. On generation quality: Section 7 severely needs an analsis on the impact of these changes on the quality of generation. Similary, in Figure 4, FID scores are all over the place. are these even good models?\n2. On memorization: In Section 5.1 the authors note that they did not observe any duplication in the clusters they studies. Again telling how the metrics of DS are not informative or reflective of memorization in the first place. I do not understand why checking similarity scores with the top 5% of generations. Further, in Figure 5: lower SS when the image is duplicated. and this happens even in the right figure. But does this actually related to \"memorization\" or just stay till the idea of dataset similarity. For instance, in the de-duplication done for SD2, not all duplicates were \"full duplicates\" if I am correct. So i think the explanation saying random captions helps reduce SS is an over statement.\n3. On section 3 and Figure 1 proper attribution should be given to the original authors who devised the technique\n4. I have not seen people use clip text features to assess how close texts are. I would suggest using some text model such as RoBERTa for these use cases. \n5. \"possibly because FID is lowest when the dataset is perfectly memorized\" I do not see how this logically flows. And in general I did not like that this paper makes too many conjectures to explain every possible phenomenon they see in results. It just feels like the authors are trying to fit a jigsaw no matter what.\n\nQUESTIONS:\n1. \" Since the metadata is structured to provide URLs from similar domains together, this makes us likely to find large clusters.\" Do you have a reference to this? I do not think this is the case from my observations of similar images across various parquets in the LAION dataset\n2. Are the results in 5.2 on unconditional models?\n3. Figure 4: why does pretrained SD have different similarity scores?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

comments: SUMMARY:
This paper systematically investigates the memorizing phenomenon of stable diffusion. As they empirically observed, most of the memorizing phenomenon originates from the duplication of images and captions in the training set. Owing to this, the authors propose to deduplicate the captions during the training stage to obviate the memorization of the diffusion model.

STRENGTHS:
The experiments in this paper are quite complete. In my opinion, most of the components that may be related to the mesmerization phenomenon are considered.

WEAKNESSES:
To measure the memorization magnitude, the authors use a similarity score and set the threshold as 0.5, i.e., the score is larger than 0.5 means there is a memorization for a certain generated image. My concern is whether this score and threshold are valid criteria. 

For the stable diffusion trained on a large-scale dataset without duplication i.e., stable diffusion v2.1, the memorization phenomenon seems almost mitigated. Thus, my question is whether increasing the unduplicated data can fix the memorization phenomenon.

QUESTIONS:
In Section 5.2, the authors explore the effect of the data duplication rate. They vary the rate via increase the sampled probability of duplicated data. My question is by doing so, the underlying ground-truth distribution is changed. Since the model is trained to approximate the ground-truth one. Does it reasonable for the model to generate more data similar to the duplication cluster?

What sampler do you use to generate data, DDIM or DDPM? Does the noise in DDPM helps mitigate the memorization phenomenon?

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors conduct a large-scale empirical study about the memorization problem in diffusion models. Their experimental results indicate that text conditioning plays an important role in this problem. Based on these observations, they propose to randomize the text prompts during training to mitigate memorization and copying.\n\nSTRENGTHS:\n1. The memorization problem is valuable for the AI safety community.\n\n2. The authors design a large number of experiments to study this problem.\n\n3. The authors observe that text conditioning plays an important role in this problem, which is a new discovery compared with the existing works.\n\nWEAKNESSES:\n1. Some experiment designs cannot verify the arguments. And some results are not consistent with the arguments of the authors. Please refer to Question1-3.\n\n2. The proposed solutions are not novel. Please refer to Question4.\n\n3. Some discussion and references about mode collapse are missing. Please refer to Question5.\n\n4. Some figures in the paper are messy and confusing. Please refer to Question6.\n\nQUESTIONS:\n- 1. In line177, the conclusion ‘’the model is more likely to memorize images when the captions are more diverse” conflicts with the observations in Section 7. The authors propose several data augmentation methods to make the text prompts as diverse as possible, which successfully mitigates the memorization problem. This indicates that when the captions are more diverse, the model is less likely to memorize images. Please clarify this conflict.\n\n- 2. In line185-186, why do you draw a conclusion that ‘’the model is more inclined to remember instances when the captions associated with them are highly specific, or even unique, keys” according to the experiment phenomenon that the similarity score of random captioning increases after finetuning the text encoder? Wouldn't random captions be even less SPECIFIC keys?\n\n- 3. Please report the FID values in Table1 to verify whether the proposed solutions will hurt the generation quality.\n\n- 4. The mitigation strategies in Section7 are not novel. Even though some of them have positive effects, they are just some data augmentation tricks that should have been used.\n\n- 5. Actually, I find the memorization problem discussed in this paper is related to a common problem in generative models – mode collapse. Especially in the failure cases mentioned in line149-157, the phenomenon is very similar to mode collapse – the generative models perform terribly when they are only able to cover a small portion of the data distribution. As for the memorization problem in Stable Diffusion, I think it is a special case of mode collapse that the text conditioning model collapses because there are not diverse enough text prompts in the training set, which is consistent with the analysis and the proposed mitigation strategies in this paper. Please discuss the difference between memory problems and mode collapse, if you disagree with me, and vice versa, please mention the connection in the related work section.\n\n- 6. Some figures are confusing. For example:\n\n    - a) There is no text prompt in Figure1, so it is hard to understand the error modes. Take the 4-th image as an example: if the text is ''The front page of PUBG”, the output image should be the 4-th image in the lower row and thus I don’t think this is a problem.\n\n    - b) Please use the vector figures in the right two subfigures in Figure2. They look very blurry.\n\n    - c) There is no caption about the left subfigure in Figure2. I don’t know why you put it here.\n\n    - d) There is no legend for the two dash lines in Figure4. And I suggest drawing the FID scores of finetuned models above the chart bars rather than writing them in the caption.\n\nLIMITATIONS:\nThe authors do not discuss the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}