Detecting, Explaining, and Mitigating Memorization in Diffusion Models
Authors: Yuxin Wen
Yuchen Liu
Chen Chen
Lingjuan Lyu
Venue: ICLR 2024 oral (2024)
Keywords: Diffusion Model
Memorization
Primary area: generative models

Abstract: Recent breakthroughs in diffusion models have exhibited exceptional image-generation capabilities. However, studies show that some outputs are merely replications of training data. Such replications present potential legal challenges for model owners, especially when the generated content contains proprietary information. In this work, we introduce a straightforward yet effective method for detecting memorized prompts by inspecting the magnitude of text-conditional predictions. Our proposed method seamlessly integrates without disrupting sampling algorithms, and delivers high accuracy even at the first generation step, with a single generation per prompt. Building on our detection strategy, we unveil an explainable approach that shows the contribution of individual words or tokens to memorization. This offers an interactive medium for users to adjust their prompts. Moreover, we propose two strategies i.e., to mitigate memorization by leveraging the magnitude of text-conditional predictions, either through minimization during inference or filtering during training. These proposed strategies effectively counteract memorization while maintaining high-generation quality. Code is available at https://github.com/YuxinWenRick/diffusion_memorization.

Published as a conference paper at ICLR 2024
DETECTING, EXPLAINING, AND MITIGATING MEMO RIZATION IN DIFFUSION MODELS
Yuxin Wen';, Yuchen Liu Chen Chen², Lingjuan Lyu University of Maryland, ?Zhejiang University, $Sony Al ywenšund. edu, yuchen. 1iu. ju edu. en [ChenA. Chen, singjuant Lv)esony.com com
ABSTRACT
Recent breakthroughs in diffusion models have exhibited exceptional image- generation capabilities. However, studies show that some outputs are merely replications of training data. Such replications present potential legal chal- lenges for model owners, especially when the generated content contains pro- prietary information. In this work, we introduce straightforward yet effec- tive method for detecting memorized prompts by inspecting the magnitude text-conditional predictions. Our proposed method seamlessly integrates with- out disrupting sampling algorithms, and delivers high accuracy even at the first generation step, with single generation per prompt. Building tion strategy, we unveil an explainable approach that shows the contribution of individual words or tokens memorization. This offers an interactive medium for users to adjust their prompts. Moreover, propose two strategies i.e., to mitigate memorization by leveraging the magnitude of text-conditional predie- tions, either through minimization during inference or filtering during training. These proposed strategies effectively counteract memorization while maintain- ing high-generation quality. Code is available at https://githi:: com/ YuxinWenRick/ di httf//iffffoooooo wtw.iuunmmmooooitttttttt...
detec-
fusion www.mmmmmmattt...
INTRODUCTION
Recent advancements in diffusion models have revolutionized image generation, with modern text- to-image diffusion models, such as Stable Diffusion and Midjourney, demonstrating unprecedented capabilities in generating diverse, stylistically varied images. However, a growing body of research (Somepalli et al., 2022; Carlini et al., 2023; Somepalli et al., 2023b) reveals concerning trend: some of these "novel" creations are, in fact, near-exact reproductions of images from their training datasets, as depicted in the top row of Fig. 1. Some of the creations appear to borrow elements from these training images, as illustrated in the second row of Fig. This issue of unintended memorization poses serious concern for model owners and users, especially when the training data contains sensitive or copyrighted material. poignant real-life example of this Midjourney, which felt obliged to ban prompts with the substring "Afghan" to avoid generating images reminiscent of the renowned copyrighted photograph of the Afghan girl. Yet, as Wen et al. (2023a) note, merely banning the term "Afghan" does not prevent the model from recreating those images. In light of these issues, the development of techniques to detect and address such inadvertent memorizations has become crucial.
To address this, we first introduce novel method for detecting memorized prompts. We've observed that for such prompts. the text condition consistently guides the generation towards the memorized solution, regardless of the initializations. This phenomenon suggests significant text guidance during the denoising process. As result, our detection method prioritizes the magnitude of fext-conditional predictions as its cardinal metric. Distinctively, memorized prompts tend to exhibit larger magni- tudes than mon-memorized ones, as showcased in Fig. Unlike previous methods that query large training datasets with generated images (Somepalli et al., 2022) or assess density across multiple
Work done during an internship at Sony AL.
"orresponning author.

---

Published as a conference paper at ICLR 2024
Magnitude of Text-Conditional Noise Prediction
Training Image
Generated Image
20
15.
1000
750
500 Time-stop
250
"Living in the Light with Ann Graham Lotz"
15
Metric
10:
1000
750
500
250
Time-step
"Plattville Green Area Rug by Andover Mills"
20
15.
Metric
10
1000
750
250
Time step
"Beautiful Bernese mountain dog (Berner Sennenhund) lies snow in park
Figure I: Memorization vs. non-memorization generation. We display the magnitude of text- conditional noise prediction at each time-step, as described Section 3.3, for four generations distinctly (with 4 different random seeds) for each prompt. As illustrated in the first two rows, the metric typically indicates higher value when memorization occurs. On the other hand, the normal generations, represented in the third row, consistently exhibit significantly lower metric values.
generations (Carlini et al., 2023), our strategy offers precise detection without adding any extra work to the existing generation framework and even without requiring multiple generations. In terms of efficacy, our method achieves AUC of 0.960 and TPR FPR of 0. 0.7 in under 2 seconds. In contrast, the baseline method (Carlini et al., 2023) demands over 39 seconds, even though it registers an AUC of 0.934 and TPR @ 1FFP of 0.523. Such efficiency empowers model owners to halt generations early and initiate corrective measures promptly upon detecting a memorized prompt.
discoveries, devise strategy highlight the influence of each token in driv- ing memorization, aiming to pinpoint the specific trigger tokens responsible for it. Following the intuition that removing these trigger tokens should neutralize memorization, we anticipate cor- responding reduction in the magnitudes of text-conditional predictions. Thus, by evaluating the gradient change for every token when minimizing text-conditional prediction magnitudes, cern the relative significance of every token to memorization. Armed with this tool, model owners can provide constructive feedback to users, guiding them to identify, modify, or omit these pivotal trigger tokens, effectively reducing the propensity for memorization. In contrast to earlier work like (Somepalli et al., 2023b), where trigger tokens were discerned manually from training data or by experimenting with various token combinations, approach stands out for its automated nature and computational efficiency.
Building
dis-
Lastly, we introduce mitigation strategies to address memorization concerns. Catering to both in- ference and training phases, we present model owners with a choice of two distinct tactics. For inference, suggest using perturbed prompt embedding, achieved by minimizing the magni- tude of text-conditional predictions. During training, potential memorized image-text pairs can be screened based the magnitude of text-conditional predictions. Our straightforward approaches
2

---

Published as a conference paper at ICLR 2024
ensure more consistent alignment between prompts and generations, and they effectively reduce the memorization effect when benchmarked against baseline mitigation strategies.
RELATED WORK
Membership Inference. The membership inference attack (Shokri et al., 2017) aims determine if a particular data point was used in the training set of model. Traditional studies on membership inference (Shokri al., 2017; Yeom et al., 2018; Carlini et al., 2022; Wen et al., 2022) have predom- inantly focused on classifiers. An attacker can utilize losses or confidence scores metric. This is because data points from the training set typically exhibit lower losses or higher confidence scores than the unseen data points during inference due to overfitting. In parallel development, recent works (Matsumoto et al., 2023; Duan et al., 2023; Wang et al., 2024; 2023) have extended member- ship inference diffusion models. These methodologies involve introducing noise to target image and subsequently verifying if the predicted noise aligns closely with the induced noise.
Training Data Extraction. Somepalli et al. (2022) demonstrate that diffusion models memorize a subset of their training data, often producing the training image verbatim. Building on this fact, Carlini et al. (2023) introduce black-box data extraction attack designed for diffusion models. This approach involves generating multitude of images and subsequently applying a membership inference attack to assess generation density. Notably, they observe that memorized prompts tend to produce nearly identical images across different seeds, leading to high density. This strategy bears resemblance the pipeline used by Carlini al. (2021), who successfully extract training data from large language models with over billion parameters. Additionally, they discover that larger models are more susceptible to data extraction attacks compared to their smaller counterparts.
Diffusion Memorization Mitigation. Recent research by Daras et al. (2023) presents a method for training diffusion models using corrupted data. In their study, they demonstrate that their proposed training algorithm aids in preventing the model from overfitting to the training data. Their approach involves introducing additional corruption prior to the noising step and subsequently calculating the loss on the original input image. In separate study, Somepalli et al. (2023b) delve into various mitigation strategies, with a focus on altering the text conditions. As a notable example, by inserting random tokens into the prompt or integrating random perturbations into the prompt embedding, they alleviate the memorization concern while preserving high-quality generation output.
3 DETECT MEMORIZATION EFFICIENTLY
3.1 PRELIMINARY
We begin by defining the essential notation associated with diffusion models (Ho et al., 2020; Song & Ermon, 2020; Dhariwal & Nichol, 2021). For a data point drawn from the real data distribution g(xo). forward diffusion process comprises a fixed Markov chain spanning T steps, where each step introduces predetermined amount of Gaussian noise. Specifically:
q( = N(1;; / 3rI,, B,1), for t E T",
(1)
where B, E (0, is the scheduled variance at step t. The closed-form for this sampling is
It = / 20 + vI āze,
(2)
where, ā [1 -.).
In the reverse diffusion process, a Gaussian vector Zy ~ NO, 1) is denoised to map to an image To (xx). At each denoising step, trained noise-predicts eg anticipates the noise ca that was added to Zn- Based on Eq. (2), the estimation of xn can be formulated as:
It /I (2z)(z:)
#
(3)
Vār.
Then, we can predict f;-1 as:
It-1 Vor-15 + V1 01-160(51).
(4)
3

---

Published as a conference paper at ICLR 2024
2
Memorization Non-Memaarizaton Percentile
AUC 993,
TPRig 1MFFR--0.97
LAION
COCD
1.0
Magnitude
23
Density
0.8
22
0.6
21
0.4
20
0.2
0.0
1000
800
400
200
10a
10a
Time-step
Magnitude
(a) Lineplot over generation steps
(b) Histogram over different datasets
Figure 2: Statistics of the magnitude of text-conditional noise predictions.
Text-conditional diffusion models, such as Stable Diffusion (Rombach et al., 2022), employ classifier-free diffusion guidance (Rombach al., 2022) steer the sampling process. Given text prompt p. its embedding Ep fp) is computed using a pre-trained CLIP text encoder f(-) (Radford et al., 2021; Cherti al., 2023). In the reverse process, the conditional sampling adheres to Eq. (3) and Eq. (4), but the predicted noise c0(x:) is changed to:
E0 e0) + s(ea ep) eo (Tt: eg
text-conditional prediction
where, eg represents the prompt embedding of an empty string, and determines the guidance strength, controlling the alignment of the generation to the prompt. We refer to the term co(2,, ep) cI( Cg) the text-conditional noise prediction for future reference.
3.2 MOTIVATION
When provided with the same text prompt but different initializations, diffusion models can generate a diverse set of images. Conversely, when given different text prompts but the same initialization, the resulting images often display semantic similarities. These similarities include analogous layouts and color themes, as demonstrated in Appendix Fig. 6. Such phenomenon might arise when the final generation remains closely tied to its initialization, and the textual guidance not particularly dominant. This observation is consistent with findings from Wen et al. (2023b), suggesting that one can trace the origin to the initial seed even without knowing the text condition.
Interestingly, when it comes to memorized prompts, the initialization appears to be irrelevant. The generated images consistently converge to specific memorized visualization. This behavior implies that the model might be overfitting both the prompt and certain denoising trajectory, which leads to the memorized image. Consequently, the final image deviates substantially from its initial state.
These insights provide foundation for straightforward detection strategy: scrutinizing the magni- tude of text-conditional noise predictions. A smaller magnitude signals that the final image is closely aligned with its initialization, hinting that it is likely not memorized image. On the other hand, a larger magnitude could indicate potential memorization. The correlation between magnitude and memorization is depicted in Fig. 2(a).
3.3 AN EFFECTIVE DETECTION METHOО
Following the intuition above, we introduce straightforward yet effective detection method tered on the magnitude of text-conditional noise predictions. For a prompt embedding p and sampling step of T, the detection metric is defined
T
[ee(xe, ep) ea cx( E0)
T
4

---

Published as a conference paper at ICLR 2024
Table 1:
Memorization detection results with AUC, TPR %FPR, and the running time of the method in seconds. In this table, "n" represents the number of generations per prompt.
Method
1st Step
First 10 Steps
Last Step
AUC TTT@T Time in Seconds.
Density 6, n=4
0.520 0.012/0.810
0.652 0.225 5.314
0.659 0.288/ / 9.904
Density,, b, n=16
0,506 0.000 / 3.570
0.656 0.175 24.78
0.676 0.271 / 40.66
Density e, n=32
0.510/ 0.000 8.009
0.664 / 0.175 / 59.43
0.681 / 0.266 / 81.44
Density SSCD n=4
0.537 / 0.019 0.809
0.405 0.005 5.421
0.878 / 0.525 / 9.892
Density SSCD= n=16 16
0.515 0.000 / 3.186
0.375 0.000 21.02
0.934 0.523 / 39.55
Densitysscin n=32
0.506 0.000 6.341
0.370 0.000 / 42.12
0.940 0.530 / 79.47
Ours, n=1
0.960 0.760 / 0.199
0.989 / 0.944 / 1.866
0.989 / 0.934 9.584
Ours, n=4
0.990/ / 0.912 0.794
0.998 0.982 / 7.471
0.996 / 0.978 / 37.27
Ours, n=32
0.996 / 0.954 1.606
0.999 0. 14.96
0.998 / 0.986 / 74.75
Memorization then identified if the detection metric falls beneath a tunable threshold ?.
In practice, we also find that even the detection metric of a single generation can provide a strong signal of memorization. Consequently, our method remains effective and reliable with the number of generations restricted to 1. In contrast, earlier studies, such as those examining generation density over a large number of generations (Carlini et al., 2023), require the simultaneous generation of multiple images, with some cases necessitating over a hundred generations. This might impose an extra computational burden on the service provider by generating more images than the user requested. Moreover, another method presented in (Somepalli et al., 2023a) identifies memorized prompts by directly comparing the generated images with the original training data. Unlike this method, our approach allows third party to use the detection method without needing access the large training dataset, thereby protecting training data privacy.
Another distinet advantage of our approach is its adaptability in calculating the detection metric. Strong detection does not mandate collecting the metric from all sampling steps. Based on our empirical findings, even when the metric is collated solely from the first step, reliable detection remains attainable. This efficiency enables model owners to identify memorized prompts promptly. By stopping generation early, they can then opt for post-processing, like declining the generated output or reinitializing the generation with corrective strategies in place.
3.4 EXPERIMENTS
Experimental Setup. To evaluate our detection method, we use 500 memorized prompts identified in Webster (2023) for Stable Diffusion v1 (Rombach al., 2022), where the SSCD similarity score (Pizzi et al., 2022) between the memorized and the generated images exceeds 0.7. The memorized prompts gathered in Webster (2023) include three types of memorization: 1) matching verbatim: where the images generated from the memorized prompt are an exact pixel-by-pixel match with the original paired training image; 2) retrieval verbatim: the generated images perfectly align with some training images, albeit paired with different prompts; 3) template verbatim: generated images bear a partial resemblance to the training image, though variations in colors or styles might observed.
Additionally, we use another 2, 000 prompts, evenly distributed from sources LAION (Schuhmann et al., 2022), COCO (Lin et al., 2014), Lexica.art (Santana, 2022), and randomly generated strings. For this set of prompts, DDIM (Song et al., 2020) with 50 inference steps.
assume they not memorized by the model. All generations employ
In our comparison, we use the detection method from Carlini et al. (2023) as baseline. This method determines memorization by analyzing generation density, computed using the pairwise & distance between oon-ovellap tiles. While Carlini et al. (2023) utilizes the & distance in pixel space, introduce an additional baseline that calculates the distance in the SSCD feature space (Pizzi et al., 2022). This adjustment is inspired by Somepalli al. (2022), who underscore the effectiveness of
5

---

Published as a conference paper at ICLR 2024
SSCD. As deep augmentations, like color shift - a critical advantage when the training image is only partially memorized.
distance metric, SSCD offers enhanced resilience to particular
We use the area under the curve (AUC) of the receiver operating characteristic (ROC) curve and the True Positive Rate at the False Positive Rate of 1% (TPR&1%FPR) as metrics. Meanwhile, report the running time in seconds with batch size of 4 on a single NVIDIA RTX A6000.
Results. In Fig. 2(b), we display a density plot comparing the detection metrics for the memorized prompts against the non-memorized ones, calculated over 50 steps with 4 generations per prompt. The distribution of memorized prompts is bimodal. This dichotomy stems from the fact that the tem- plate verbatim scenario often exhibits a slightly smaller metric than the matching verbatim scenario, given that the memorization occurs only partially.
In Table 1, we highlight the balance between the precision and efficiency of our proposed method. Our method is able to achieve very strong detection performance. When generating 32 images and using the metrics from the first 10 steps, method is able to achieve TPR@1 %FPR of 0.988. Remarkably, even when operating with a single generation, our method achieve TPR@1 %FPR of 0.760 from the very first step within merely 0.2 seconds. This feature provides significant advantage in terms of time and computational resource savings, allowing model operators the flexibility to terminate generation early if necessary. In contrast, the baseline methods show noticeably reduced detection capability. In particular, the baseline methods can only achieve relatively high detection accuracy when generating more than 16 images and relying on the image generation from the final step, where it requires at least 40 seconds. Yet, in real-world applications, service providers like Midjourney or DAIL-E (Ramesh et al., 2022) typically generate mere 4 images concurrently for each prompt.
AUC of 0.999 and
Interestingly, our method surpasses the baselines speed even when using the metric with equiva- lent generations and steps. This superiority emerges since our method doesn't 't need to decode latent noise into image space and perform subsequent calculations.
4 MITIGATE MEMORIZATION
4.1 A STRAIGHTFORWARI METHOD TO DETECT TRIGGER TOKENS
As observed by Somepalli al. (2022), certain words or tokens in memorized prompts play a sig- nificant influence on the generation process. Even when only these specific "trigger tokens" are present in the prompt, the memorization effect remains evident. One potential approach to identify these trigger tokens involves probing with various n-gram combinations to discern which combina- tions induce memorization. However, this heuristic becomes notably inefficient, particularly when the prompt contains vast number of tokens. Our earlier observations offer more streamlined method for discerning the significance of each token in relation to memorization: by checking the magnitude of the change applied to each token while minimizing the magnitude of text-conditional noise prediction. A token undergoing substantial change suggests its crucial role in steering the prediction; conversely, a token with minimal change is less important.
Given a prompt embedding e of prompty with N tokens, we form the objective of the minimization problem as:
C(2,, e) = [|ca(z,, e) - co ep)
(5)
We then determine the significance score for each token at position i E [0,
as:
1 SS. T
Σ Ver
In Fig. 3, we display generations with top-2 significant tokens highlighted. The green arrow in the figure emphasizes that altering these significant tokens can substantially diminish the memorization effect. Some trigger tokens, including symbols seemingly trivial words, challenging to iden- tify manually. Consequently, this insight offers model owners practical tool: advising users to rephrase exclude the trigger tokens before initiating another generation. In contrast, indicated

---

Published as a conference paper at ICLR 2024
Training Image
Original Generation
User Modification
After User Modification
Rewrite goats
sheep Delete
Delete "proudly, high"
2. Rewrite "the Rocky
Mountains" to "Blue Ridge Mountains'
"A pair of mountain goats stand proudly, high in the Rocky Mountains.
1. Rewrite Wave to Swell Rewrite Kanagawa to Tokyo
1. Delete Katsushika
Hokusai"
2. Delete "Great"
"A painting of the Great Wave off Kanagawa by Katsushika Hokusai"
Figure 3: By modifying the trigger tokens, memorization can be effectively mitigated. The signifi- cance score for each token is illustrated in histogram in Appendix Fig. 7. The two most significant tokens are highlighted in red and blue. A green arrow indicates modifications made to the top-2 tokens, while white arrow represents changes to less significant tokens.
by the white arrow, alterations to the less significant tokens fail to effectively counter the memo- rization effect, even when extensive changes are made. Even by renaming the play or removing the artist's name, memorization remains evident.
4.2 AN EFFECTIVE INFERENCE-TIME MITIGATION METHOD
direct approach to mitigation without any supervision to adjust the prompt embedding by min- imizing Eq. (5). Optimizing over all time steps is computationally intensive. However, observe that minimizing the loss at the initial time step indirectly results in smaller magnitudes in subse- quent time steps, effectively mitigating memorization. Thus, perturbed prompt embedding, e*, is obtained at t 0 by minimizing Eq. (5). We also apply early stopping once the loss reaches a target value, harget. to keep the embedding close the original meaning.
4.3 AN EFFECTIVE TRAINING-TIME MITIGATION METHOD
During the training of diffusion models, memorization often arises with duplicate data points due to overfitting (Carlini et al., 2023; Somepalli et al., 2023b). Building on our earlier observation, if the model overfits closely memorizes data point, the magnitude of the text-conditional noise

---

that the model retains its capacity for Published as a conference paper at ICLR 2024
Before Hine-luning
Wo Mitigation
Random Taken
0.5
Similarity Score
0.4
Simllanty
0.3
0.2
0.2
0.1
0.1
0.25
0.26
0.27
0.28
0.29
0.30
0.294
0.296
0.298
CLIP Score
CLIP Scoret
(a) Inference-time mitigation
(b) Training-time mitigation
Figure 4: Mitigation results. lower similarity score suggests reduced memorization, whereas a higher clip score denotes better alignment between the generation and the prompt.
prediction might exceed typical values. Therefore, a straightforward mitigation method is to exclude the sample from the current mini-batch if this magnitude surpasses predetermined threshold, т, thereby not computing the loss on that sample. Given that the model has previously seen the sample during training, this exclusion is unlikely to significantly impact model performance.
However, this method introduces additional computational cost during training. To compute the text-conditional noise prediction, an extra forward pass for 6 (3,,, eg) ) is needed. During typical training. only e ep) is computed. Empirically, this results in an approximately 10% increase in training time.
4.4 EXPERIMENTS
Experimental Setup. evaluate the effectiveness of mitigation strategies, we adopt a setup sim- ilar to that in Somepalli al. (2023b). Specifically, 200 LAION data points, each duplicated 200 times, to serve as memorized prompts. In addition, introduce 120, 000 distinct LAION data points to generalization.
fine-tune the Stable Diffusion model using
For performance metrics, compute the SSCD similarity score (Pizzi et al., 2022; Somepalli al., 2023b) to gauge the degree of memorization by comparing the generation to the original image. Additionally, the CLIP score (Radford et al., 202.) used to quantify the alignment between the generation and its corresponding prompt. Our experiments encompass distinct fine-tuned models, each embedded with different memorized prompts, and the results are averages over 5 runs with different random seeds.
In our evaluation of the proposed method, we test 5 distinct target losses larget ranging from 1 to 5, for inference-time mitigation. We use Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.05 and at most steps. Simultaneously, we investigate 5 different thresholds т, spanning from to 6, for training-time mitigation. For comparison, use the most effective method from (Somepalli al., 2023b), random token addition (RTA), as the baseline, which inserts 1, 2, 4, 6, or 8 random tokens to the prompt.
Results. In Fig. 4(a), we present the results of our inference-time mitigation, while Fig. 4(b) de- tails the outcomes for training-time mitigation. Our proposed techniques successfully mitigate the memorization effect and, importantly, offer more favorable CLIP score trade-off compared to RTA. Higher target losses harget or thresholds tend to enhance the model's alignment with the prompt but result in less pronounced mitigation effect. In practice, model owners select the op- timal hyperparameter based on their desired balance between mitigation efficacy and generation alignment.
8

---

Published as a conference paper at ICLR 2024
Training Image
W/o Mitig.
RTA (Inference-time)
Ours (Inference-time)
RTA TTrining-time:
Ours (Training-lime)
THE STANDARD, LONDON INTERIOR: The central London hotel,
oppesato St. Pancras Station, 'colourful grayness London"
OTA Woman"
"aale"
Figure 5: Mitigation results with different mitigation strategies during inference and training phase.
In Fig. 5, we display a selection of qualitative results, setting furget = 3 and T 4 for our method, while adding 4 random tokens as a baseline strategy. Our proposed strategy is effective in mitigat- ing the memorization effect while also ensuring that the generated content aligns closely with the prompt. In contrast, the baseline method often struggles to counteract memorization and occasion- ally produces images with undesirable additions.
5 LIMITATIONS AND FUTURE WORK
Our detection strategy employs tunable threshold to detect memorized prompts. This requires model owners to first compute metrics over non-memorized prompts and then select an empirical threshold based detection approach lack interpretability. In the future, the development terpretable p p-values could significantly assist model owners by providing confidence score that quantifies the likelihood memorization, thereby augmenting the transparency and trustworthiness of the detection process.
predetermined false positive rate. However, the outcomes generated by this method producing in-
Our proposed mitigation strategies effectively tackle the memorization problem, albeit with minor computational overheads for model owners. The inference-time mitigation approach requires addi- tional GPU memory due to the optimization process, and in practice, it takes at most 6 seconds with our setup. On the other hand, the training-time mitigation extends the training period by roughly 10%. For context, while naive fine-tuning in our tests takes approximately 9 hours, our mitigation strategy extends this about 10 hours. However, we argue that these added costs are reasonable given the significance of protecting training data privacy and intellectual property. Furthermore, when combined with our detection method, model owners need only deploy the inference-time mit- igation when memorized prompt is detected, thereby minimizing its use.
CONCLUSION
In this paper, we introduced a new approach to detect memorization in diffusion models by lever- aging the magnitude text-conditional noise predictions. Remarkably, our approach attains high precision even when using limited number of generations per prompt and limited number of sampling steps. Furthermore, we provide an explanatory tool to indicate the significance score of individual tokens in relation to memorization. To conclude, and training- time mitigation strategies. These not only effectively address the memorization concern but also maintain the superior generative performance of the model.
paper presents both inference-time

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
We have detailed all essential hyperparameters used in our experiment setup in the main body. Our experiments were conducted using widely available computing resources and open-source software. All referenced models and datasets in this paper are publicly accessible. Furthermore, we have included the code necessary to reproduce our results in the supplementary material.
REFERENCES
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raf- fel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 21), pp. 2633-2650. USENIX Association, August 2021. ISBN 978-1- 939133-24-3. URL https:ttt///tc /wwww usenix org/ /wwpo/eeeeneeeeseeeeeeeeeeeeeeee
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Mem- bership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914 4. IEEF, 2022.
Nicholas Carlini. Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Schwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting Training Data from Diffusion Models. arxiv:2301. / /8/css) January 2023. doi: 10.485550/2Xx..2301.1111 URL URRp:/t////axxx abs org/abs// 2301 13188.
org/
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive gegguage-imagee learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2818-2829, 2023.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alexandros G Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data. arXiv preprint arXiv:2305.19256. 2023.
Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. arxiv:2105505223//s,, stat], June 2021. doi: 10.48550/aXXi.210500533. URL http:// arxiv.org//aa...aaa......a.....t htt.:/gg/ab//////aaaeee//// 05233.
Jinhao Duan, Fei Kong, Shiqi Wang, Xiaoshuang Shi, and Kaidi Xu. Are diffusion models vulnera- ble to membership inference attacks? arXiv preprint arXiv:2302.01316, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso- ciates, Inc., 2020. URL https: httpt:::::/pproeeeedng.o (c55ofee8554ff005577778 atttaaaaa...........aooatttt.... htt.l..................
neurips... cccppapprr/222//aas.. Abstract
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6600 2014.
Tsung-Yt Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13. pp. 740-755. Springer, 2014.
Tomoya Matsumoto, Takayuki Miura, and Naoto Yanai. Membership inference attacks against diffusion models. arXiv preprint arXiv:2302. 23222232...32222,.. 2023.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14532-14542, 2022.
10

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image syuthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10633 2022.
Gustavo Santana. cember 2022.
Datasets at Hugging Face, De- URL https: http///tgggggggggagaee/c/eeeeee co
Stable-Di www-D--fuiiooo-ooooopeee
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. 20/7 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion Art Digital Forgery? Investigating Data Replication in Diffusion Models. doi: 10.4855Warxi.2222..33666 URL http://
arxiv:2212.03860/css/ December 2022. arxiv. org/ abs /2212.22222222 02222.008600
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion digital forgery? investigating data replication in diffuuion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 6048-6058, June 2023a.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Under- standing and mitigating copying in diffusion models. arXiv preprint arXiv:2305.2008. 2023b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02502. 2020.
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. arXiv:2006.00 0901 [es, stat], June 2020. URL http: //tpxivaa.iii.i//aaa www.exiwwwooggg/beee 2006 09011.
Zhenting Wang, Chen Chen, Lingjuan Lyu, Dimitris Metaxas, and Shiqing Ma. Diagnosis: De- tecting unauthorized data usages in text-to-image diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Zhenting Wang, Chen Chen, Yi Zeng, Lingjuan Lyu, and Shiqing Ma. Where did i come from? origin attribution of ai-generated images. Advances in Neural Information Processing Systems, 36, 2024.
Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint arXiv:2305.08669, 2023.
Yuxin Wen, Arpit Bansal, Hamid Kazemi, Eitan Borgnia, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Canary in coalmine: Better membership inference with ensembled adversarial queries. arXiv preprint arXiv:2210. 10750, 2022.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-basee discrete optimization for prompt tuning and discovery. arXiv preprint 2302 03360. 2023a.

---

Published as a conference paper at ICLR 2024
Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fin- gerprints for diffusion images that are invisible and robust. arXiv preprint arXiv:2305.20030, 2023b.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 JEEE 31st computer security foundations symposium (CSF), pp. 268 282. IEEE, 2018.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
Normal Prompt
Memorized Prompt
Seed 0
CCENTIST
Seed 1
THE
HAPPY CIENTIGT
THE
Seed 2
HAPPY CIENTIST
Seed 3
THE
HAPPY
Figure Different seeds vs. different prompts.
13

---

Published as a conference paper at ICLR 2024
Training Image
Original Generation
User Modification
After User Modification
1. Rewrite goats
sheep
2. Delete
1. Delete "proudly, high" Rewrite "the Rocky
Mountains" "Blue Ridge Mountains
"A pair of mountain goats stand proudly, high in the Rocky Mountains.
pair
of
mourtain
geots
stand
proudty
high
1
the
rodky
mountains
Rewrite Wave Swell
2. Rewrite Kanagawa
Tokyo
1. Delete "by Katsushika
Hokusai"
"Greal"
"A painting of the Great Wave off Kanagawa by Katsushika Hokusai"
painting
great
wave
agama
Figure 7:
By modifying the trigger tokens, memorization can be effectively mitigated. The signifi- cance score for each token is illustrated in histogram in Appendix Fig. 7. The two most significant tokens are highlighted in red and blue. A green arrow indicates modifications made the top-2 tokens, while white arrow represents changes to less significant tokens.
14

---

Published as a conference paper at ICLR 2024
Magnitude of Text-Conditional
Training Image
Generated Image
Noise Prediction
20
15
Metric
10
5
1000
750
500 Time-step
250
"Prince Reunites With Warner Brothers, Plans New Album"
The No Linit' Business
3o6SPinit Business Woman
joo Pimuts
20
Business Woman
15.
Metric
10:
Woman Podcast With KOin
No Pamiit
Vo Pinuts
5
Business Woman
Business
Woman
1.00
750
500 Time-step
250
"The No Limits Business Woman Podcast"
CH
20
15
THE
CIENTIST
CIENTIST
Metric
10
HAPPY SCIENTIST ETPES HONTE PROEECTINTS FEOES
THE
SCEENTIST
9
1000
750
500 Time-step
250
"The Happy Scientist"
METELIX X
LXFELIX LRFELXX
LXFLIXX
20
15
Metric
10
LXFFLIX
5
1000
750
500
250
Time-step
"Netflix Strikes Deal with AT&T for Faster Streaming"
20
15
Metric
10
9
750
500
250
Time-step
"Pencil pleat curtains in collection Avinon, fabrie: 129-66"
Figure 8: Memorization generation. We display the magnitude of text-conditional noise prediction at each time-step, as described in Section 3.3, for all four generations distinctly (with different random seeds) for each prompt. The metric typically indicates higher value when memorization occurs.
15

---

Published as a conference paper at ICLR 2024
Magnitude of Text-Conditional
Training Image
Generated Image
Noise Prediction
20
15
Metric
10
5-
1000
750
500 Time-step
250
"Watercolour painting of Xian fortifications. Sian city wall, China aquarelle illustration."
20
15
Metric
10
1
750
500 Time-step
250
"painting #ballet #painting"
20
15
Metric
10
5
0
1000
750
500
250
Time-step
"Naruto: Ultimate Ninja Storm PS3"
20
15
Metric
10-
1000
750
500
250
Time-step
"the marq vietnams ultimate luxury residential destination to be launched"
20
15
Metric
10
5
1
750
500
250
Time-step
"Croque madame sandwich, delish food"
Figure 9: Non non-memorization generation. We display the magnitude of text-conditional noise predic- tion at each time-step, described in Section 3.3, for all four generations distinctly (with 4 different random seeds) for each prompt. The non-memorized generations consistently exhibit significantly lower metric values.
16


Reviews:
[{"review_id": "PRTXtQyipv", "reviewer": "Reviewer_5xmi", "summary": "This paper builds up on previous work by Somepalli et al. on detecting and mitigating memorization in diffusion models. The authors use a very neat observation that in the case of text-guided diffusion models, the actual text prompt is very important for the final generated image. In particular, as seen by Wen et al., in the cases when the text prompt is important, irrespective of the initialization, the model would typically converge to the same image. However, in other cases, the initialization can change the model generation a lot. Using this insight, they detect memorization by evaluating the impact of the text on the model generation. The work is then further solidified by using this both as a detection and mitigation measure, surpassing past works not only by efficiency but also by performance.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This work builds on a very simple and clever observation that the impact of text prompt on the generation by a diffusion model can be used for detecting if a particular generated image was memorized. \n2. The method is extremely fast and can even detect memorization with a single step. Further, it is much better than past works, both l2, and SSCD metrics in terms of the AUC and the true positive rate at 1% false positive rate. \n3. The proposed mitigation strategies at inference time are very interesting and much more performant than the previously discussed baseline of random token addition. In particular, the method and the insight naturally offer a way of understanding which tokens were responsible for memorization and can be removed appropriately. \n4. In the case of training time mitigation, the authors see significant improvement in the model performance as opposed to when you're doing random token addition. \n5. Overall, this paper is a very enjoyable read and a strong work in the field of memorization and especially when considering diffusion models.", "weaknesses": "1. This work can be written more clearly, especially the section of the introduction was not very well written. I found that section 3.2 motivation was particularly helpful in setting the pace for this work. \n2. In terms of the experimental setting, I do believe that performing experiments to see how the memorization ratio changes with repetitions in the data set might be a great way to further solidify if the method works. In particular, this could follow directly from the setup of Somepalli et al. \n3. I would love to see more images of memorized inputs and further discussion beyond the two images shown in the paper right now to get a better sense of the performance of this method.\n4. Most pertinently, I see that the mitigation strategies lead to a significant drop in CLIP score. In particular, if you were to look at the region on the plot between the model initialization before fine-tuning and the final fine-tuned model, it is evident that, especially when you contrast with Figure B where all the points are between 0.29 and 0.3, the inference time mitigation leads to a significant drop in CLIP score. It is unclear if this method is actually useful in that regard. It suggests that we are unable to reach the same performance as that of a model that was never fine-tuned. I am curious what the authors feel about this particular observation. In particular, a model that was not fine-tuned had a higher CLIP score on the prompts, but the method using mitigation achieved a much lower CLIP score. I am not able to position these results with the overall setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ttkajpP8K", "reviewer": "Reviewer_Mwfj", "summary": "The paper addresses recent breakthroughs in diffusion models, particularly focusing on their image generation capabilities. It highlights a significant issue where some outputs from these models are mere replications of training data, posing legal challenges, especially when the content includes proprietary information. The authors propose a method for detecting memorized prompts by examining the magnitude of text-conditional predictions. This method integrates seamlessly into existing sampling algorithms and provides high accuracy from the first generation step with a single generation per prompt.", "questions": "1. Exploring Applications for Memorization Detection: I'm interested in understanding the practical uses of detecting memorized images. What are some key scenarios or fields where identifying such images is particularly important or beneficial?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces a straightforward yet effective technique for detecting memorized prompts, which is a significant contribution to enhancing the reliability of diffusion models.\n2. Mitigation Strategies: The paper proposes two practical strategies for mitigating memorization - minimization during inference and filtering during training. These strategies effectively balance counteracting memorization while maintaining high generation quality.", "weaknesses": "1. Clarifying the Concept of Memorization: Could you provide a clear definition of what constitutes memorization in this context? Does it require an exact match between the generated and training images? For instance, if there's a slight variation, such as a difference of 10 pixels from the original image in the training dataset, would that still be considered memorization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eLO65h2irv", "reviewer": "Reviewer_VQfz", "summary": "## Summary\n\n\nPaper studies the problem of memorization in diffusion models. These models sometimes simply reproduce images from their training set which could present legal challenges for model owners\n\t- One real life examples of this in Midjourney, which had to ban prompts with the substring \"Afghan\" to avoid generating images reminiscent of the renowned copyrighted photograph of the \"Afghan girl\"\n\t- \n- In this work, the authors introduce a metric to detect such memorized prompts based on the magnitude of \"text-conditional predictions\". Memorized prompts tend to have a higher magnitude than non-memorized prompts.\n\t- Extending this the authors also devise a strategy to highlight the influence of each token in the prompt in driving memorization. This is done by evaluating the change in gradient for every token when minimizing the \"text-conditional prediction\" magnitude. This gives the relative important of each token to memorization. \n\t- This can be used to provide feedback to prompt designers to omit, modify these pivotal trigger tokens in their prompt.\n\t- Stable diffusion uses classifier-free guidance to steer the sampling diffusion process. During the reverse diffusion process the noise part of the original equation is modified to minimize distance from  the embedding of the text computed using a pre-trained CLIP encoder. This difference term is referred to as the \"text-conditional noise prediction\". The metric proposed in the paper is defined as the L2 norm of \"text-conditional noise\" term divided by the number of sampling steps \n\t\t- A smaller magnitude for this term signifies the final image is closely aligned its initialization\nBaseline & Dataset:\n- The authors use the 500 memorized prompts from Webster 2023 for stable diffusion v1 where SSCD similarity score between memorized and generated images exceeds 0.7\n- The detection method from Carlini 2023 is used as baseline\n- They also use an additional baseline where instead of using L2 distance like Carlini 2023 they replace it with distance in the SSCD feature space.", "questions": "## Questions/Clarifications\n\n- How is the metric computed with multiple generations in Table 1?\n- In Table 1, what does the column First 10 steps indicate, is it the AUC, TPR values calculated with the average of metric values for the first 10 steps of the diffusion process?\n- What is meant by the following sentence in the \"An effective inference-time mitigation method\"\n\t- \"Thus, a perturbed prompt embedding e* is obtained as t=0 by minimizing Eq (5)\" - Is this minimization done via gradient descent, what is the data on which this minimization is performed?\n\t- Was this done on the 200 LAION data points? If yes, what is Figure 4(a) and 4(b) plotted over?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Strengths/Weaknesses\n\n- The two advantages of using the proposed metric are\n\t- It doesn't need access to training data which some of the previous methods do\n\t- Even if the metric is collated solely from first step, reliable detection is possible.\n- Results indicate that the method obtains a high detection score with an AUC of 0.999 with small latency.", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "auUDATdvUG", "reviewer": "Reviewer_hCP2", "summary": "The paper discusses memorization issue of diffusion models used in generating images from text prompts. Text-conditional diffusion models employ a pre-trained text encoder, which controls the alignment of image generation to a given prompt. Observations reveal that diffusion models generate diverse images for the same text prompt but different initializations. However, with different prompts but identical initializations, the images display similarities. When the model uses memorized prompts, the output tends to be consistent, hinting at overfitting. One of the significant findings is the relationship between the magnitude of text-conditional noise predictions and the chances of an image being memorized. This paper proposed a method, allowing early detection of memorized prompts, potentially saving computational resources. The effectiveness of this method is further discussed in an experimental setup, where it surpasses other baseline methods in speed and accuracy. Additionally, the concept of \\emph{trigger tokens} is introduced. These are specific words or tokens in a prompt that have a significant impact on the generation process. A technique is provided to identify these tokens, allowing users to modify them to mitigate memorization. In summary, the paper explores the intricacies of diffusion models in generating images from text prompts and provides methods to detect and counter memorized image generation.", "questions": "1. Is there any way to quatify memorization in the diffusion models that the future method could use to benchmark? It might be good to have a discussion in this direction.\n\n2. Are certain tokens more susceptible to triggering memorization than others? How were these trigger tokens identified, and is there a taxonomy or classification for them?\n\n3. Were there any adversarial tests done to ascertain if an attacker could still exploit the memorization tendencies, even after applying the proposed mitigation strategies?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method to efficiently detect memorization in diffusion models by scrutinizing the magnitude of text-conditional noise predictions. This method is both computationally efficient and does not require multiple generations or access to the original training data, ensuring data privacy and reducing computational overhead.\n\n2. This work offers an automated approach to identify specific \"trigger tokens\" in memorized prompts that have a significant influence on the generation process. Instead of manual identification or experimentation with various token combinations, which can be cumbersome and inefficient, the paper's method assesses the change applied to each token while minimizing the magnitude of text-conditional noise prediction. This innovative approach provides model owners with a practical tool to advise users on how to modify or omit these trigger tokens, which can significantly mitigate the effects of memorization.\n\n3. The authors introduce mitigation methods that cater to both inference and training phases. For inference, a perturbed prompt embedding is suggested, achieved by minimizing the magnitude of text-conditional predictions. During training, potentially memorized image-text pairs can be screened out based on the magnitude of text-conditional predictions. These methods not only address the concerns of memorization but also ensure a more consistent alignment between prompts and generations. The experiments conducted, as per the paper's context, seem to support the efficacy of these strategies when benchmarked against baseline mitigation methods.", "weaknesses": "1. While the mitigation strategies aim to reduce memorization, it's unclear what impact they might have on the overall performance of the model. Often, there's a trade-off between reducing a particular behavior and maintaining high performance. If these mitigation strategies significantly impair the model's utility, it might deter their adoption.\n\n2. As stated in the paper, a weakness of the proposed method is the lack of interpretability in the detection strategy of memorized prompts. The current approach requires the model owners to select an empirical threshold based on a predetermined false positive rate, but the outcomes generated lack clear interpretability. This lack of clarity can make it difficult for model owners to fully understand and trust the detection process. The authors acknowledge that developing a method that produces interpretable p-values could significantly assist model owners by providing a confidence score quantifying the likelihood of memorization. \n\n3. Advising users on modifying or omitting trigger tokens might be effective in theory, but in practice, it could be cumbersome. Users might need to understand what these tokens are, why they need to modify them, and how they affect the output. This could make the user experience less intuitive, especially for those unfamiliar with the inner workings of AI models.\n\n4. The paper assumes that all prompts can be modified or that users will be willing to modify them. In real-world scenarios, some prompts might be non-negotiable, and changing them might not be an option.\n\n5. While the paper suggests that the method is computationally efficient, implementing the strategies during the training and inference phases might still introduce computational or operational overheads for model owners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)