Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models
Authors: Dominik Hintersdorf
Lukas Struppek
Kristian Kersting
Adam Dziedzic
Franziska Boenisch
Venue: NeurIPS 2024 poster (2024)
Keywords: memorization
diffusion models
Primary area: diffusion_based_models
TL;DR: We propose the first method to identify neurons in text-to-image diffusion models responsible for memorization.

Abstract: Diffusion models (DMs) produce very detailed and high-quality images. Their power results from extensive training on large amounts of data - usually scraped from the internet without proper attribution or consent from content creators.  Unfortunately, this practice raises privacy and intellectual property concerns, as DMs can memorize and later reproduce their potentially sensitive or copyrighted training images at inference time. Prior efforts prevent this issue by either changing the input to the diffusion process, thereby preventing the DM from generating memorized samples during inference, or removing the memorized data from training altogether. While those are viable solutions when the DM is developed and deployed in a secure and constantly monitored environment, they hold the risk of adversaries circumventing the safeguards and are not effective when the DM itself is publicly released. To solve the problem, we introduce NeMo, the first method to localize memorization of individual data samples down to the level of neurons in DMs' cross-attention layers. Through our experiments, we make the intriguing finding that in many cases, single neurons are responsible for memorizing particular training samples. By deactivating these memorization neurons, we can avoid the replication of training data at inference time, increase the diversity in the generated outputs, and mitigate the leakage of private and copyrighted data. In this way, our NeMo contributes to a more responsible deployment of DMs.

Finding NeMo Localizing Neurons Responsible For Memorization in Diffusion Models
Dominik Hintersdorf Lukas Struppek 1,2
Kristian Kersting 1,2,3,4 Adam Dziedzic Franziska Boenisch
GGerman Research Center for Artificial Intelligence (DFKI)
CComputer Science Department, Technical University of Darmstadt
Hessian Center for AI (Hessian.Al) *Centre for Cognitive Science, Technical University of Darmstadt $CISPA Helmholtz Center for Information Security
Abstract
Diffusion models (DMs) produce very detailed and high-quality images. Their power results from extensive training on large amounts of data-usually scraped from the internet without proper attribution or consent from content creators. Un- fortunately, this practice raises privacy and intellectual property concerns, as DMs can memorize and later reproduce their potentially sensitive or copyrighted training images at inference time. Prior efforts prevent this issue by either changing the input to the diffusion process, thereby preventing the DM from generating mem- orized samples during inference, or removing the memorized data from training altogether. While those are viable solutions when the DM is developed and de- ployed in a secure and constantly monitored environment, they hold the risk of adversaries circumventing the safeguards and are not effective when the DM itself is publicly released. To solve the problem, we introduce NEMo, the first method localize memorization of individual data samples down to the level of neurons in DMs' cross-attention layers. Through our experiments, we make the intriguing finding that in many cases, single neurons are responsible for memorizing particular training samples. By deactivating these memorization neurons, we can avoid the replication of training data at inference time, increase the diversity in the generated outputs, and mitigate the leakage of private and copyrighted data. In this way, our NEMo contributes to a more responsible deployment of DMs.
Introduction
In recent years, diffusion models (DMs) have made remarkable advances in image generation. In particular, text-to-image DMs, such as Stable Diffusion 332. DALL-E (30], or Deep Floyd (38 enable the generation of complex images given a textual input prompt. Yet, DMs carry significant risk to privacy and intellectual property, as the models have been shown to generate verbatim copies of their potentially sensitive or copyrighted training data at inference time [8 36). This ability has often been linked to their memorization of training data [48 12 55.. Memorization in DMs recently received lot of attention [66
49 g and several mitigations have been proposed [46] 31] (6). Those mitigations usually focus on either identifying potentially highly memorized samples and excluding them from training, monitoring inference and preventing their generation, altering the inputs to prevent the verbatim output training data [3] 36 46). While mitigations that rely on preventing
47
the generation of memorized samples are effective when the DM is developed and deployed in secure
"qqual contribution, corresponding authors: (hintersdorf, sasee Code: htttps://github.. wwtp://weeeeeeeccooccc.eeeaaeaea htt.a//c......aaooo.o....ooooood
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Mcmorized. Treining Sample
Initial Selection +
Candidate Neurons
Refinement
Memorization Neurons
Neuron #124
Naise Simbloity
Neuron #221
Neuron #221
"Eistg in the Light mith
Neuron 4362
Newren-idel
flackie Neurons
Rivsted Candidsiee
Idently Candidate Neurons With Sottart-istrbutiin. Activations
Remove Candidate Neurons Not Responsiblc for Memorization
Generation with Blocked Neurons
Genaration on with Blocked
Genaration with Blocked Memorization Neurons
Candidate Neurons
Figure I: Overview of NeMo. For memorized prompts, we observe that the same (original training) image is constantly generated independently of the initial random seed. This yields severe privacy and copyright concerns. In the initial stage, NEMo & first identifies candidate neurons potentially responsible for the memorization based on cittof-distribution activations. In a refinement step. NEMo & detects the memorization neurons from the candidate set by leveraging the noise similarities during the first denoising step. Deactivating memorization neurons prevents unintended memorization behavior and induces diversity in the generated images.
environment, they hold the inherent risk of adversaries circumventing them. Additionally, they are not effective when the DMs are publicly released, such that users can freely interact with them.
As first step solving this problem, propose FINDING NEURON MEMORIZATION (NRMo), new method for localizing where individual data samples are memorized inside the DMs. NEMo's localization tracks the memorization of training data samples down to the level of individual neurons in the DMs' cross-attention layers. To achieve this, NEMO relies on analyzing the different activation patterns of individual neurons on memorized and non-memorized data samples and identifying
memorization neurons by outlier activation detection (as visualized in Fig. 2b We empirically assess the success of NEMo on the publicly available DM Stable Diffusion [32.. Our findings indicate that most memorization happens in the value mappings of the cross-attentin they highlight that most training data samples are memorized by just a few or even a single neuron, which is surprising given the high resolution and complexity of the training data.
layers of DMs. Furthermore,
Based on the insights about where within the DMs individual data samples are memorized, we can prevent their verbatim output by deactivating the identified memorization neuron(s). We demonstrate the effect of NEMO in Fig. [ Without our approach, the image generated for the memorized input prompt is the same, independent of the random seed for generation. By localizing the neuron responsible for the memorization through NEMO and deactivating it, we prevent the verbatim output of the training data and instead cause the generation of various non- memooized related samples. Hence, by relying on NEMO to localize and deactivate memorization neurons, we can limit memorization, which mitigates the privacy and copyright issues while keeping the overall performance intact.
In summary, we make the following contributions:
We propose NEMo, the first method to localize where memorization happens within DMs down to the level of individual neurons.
Our extensive empirical evaluation of localizing memorization within Stable Diffusson reveals that few or even single neurons are responsible for the memorization.
We limit the memorization in DMs by deactivating the highly memorizing neurons and further show that this leads to a higher diversity in the generated outputs.
2 Background and Related Work
2.1 Text-to-Image Synthesis with Diffusion Models
Diffusion models (DMs) 19 are generative models trained by progressively adding random Gaussian noise to training images and having the model learn to predict the added noise. After the training finished, new samples can be generated by sampling an initial noise image XT N0 I) and then iteratively removing portions of the predicted noise (I t, y) at each time step = T, 1,

---

This denoising process is formally defined by
co co(oeit..))
It-1
vas
v Cie
with variance scheduler 3r E (0, 1), a, = 1 Br and ãr = -II Czg, The noise predictor c y). usually a U-Net [33, receives an additional input for conditional image generation.
34 conditioned text embeddings y computed by pre- The typical way to incorporate the conditioning into the denoising process is the cross-attention mechanism [4.. (Cross-)Attentis
Common text-to-image DMs trained text encoders like CLIP
32
30
29
consists of three main components: query matrices Q = 2, Wo. key matrices K' = yW k, and value matrices V yWy. All three matrices are computed by applying learned linear projections Wo, WK, and Wy to the hidden image representation 2 and the text embeddings The attention outputs are computed by
QK? Vd
Attention Q, V) softmax
V,
with scaling factor Importantly, in most text-to-image models, the noise predictor receives guidance only through the cross-attention layers, which renders them particularly relevant for memorization.
2.2 Memorization in Deep Learning
Memorization. Memorization was extensively studied in supervised models and with respect to data
labels [48] [1. Recently, studies have been extended to unlabeled self-supervised learning [25 43
In both setups, it was shown that memorization is required for generalization [12
[3] 4 However,
memorization also yields privacy risks [4] 14 40) since it can expose sensitive training data.
In particular for generative models, including DMs, was shown that memorization enables the extraction of training data points 4 8 36).
Localizing Memorization. Early work on localizing where inside machine learning (ML) models memorization happens focuses on small neural networks. Initial findings suggested that in supervised models, memorization happens in the deeper layers 2 39.. However, more fine-grained analyses contradict these findings and identify that individual units, i... individual neurons or convolutional channels throughout the entire model, are responsible for memorization
(24). To identify these, Maini et al. (2 deactivate units throughout the network until a label flip on the memorized training input image occurs. However, due to the unavailability of labels, this approach does not transfer to DMs.
Memorization in Diffusion Models. Recent empirical studies connect the model architecture, training data complexity, and the training procedure to the expected level of DM memorization (16). while others connect memorization to the generalization of the generation process [47]. Two types of
memorization are usually distinguished: Verbatim memorization that replicates the training image exactly. And template memorization that reproduces the general composition of the training image while having some non-semantic variations at fixed image positions 455 Existing approaches for detecting memorized training samples are based statistical differences in the model behavior when queried with memorized prompts. These approaches explore differences in predicted noise magnitudes [46), the distribution of attention scores (31), the amount of noise modification in one-step synthesis [45), and the edge consistency generated images (45). Our work orthogonal to these detection methods, focusing on the exact localization of memorization in the DM's U-Net rather than detecting memorized samples.
Previously proposed methods for mitigating memorization during inference either rescale the attention logits (3I] or adjust the text embeddings with a gradient-based approach to minimize the magnitude of noise predictions [46]. However, these inference time mitigation strategies are easy to deactivate in practice and provide no permanent mitigation strategies for publicly released models. In contrast, related training- based mitigation strategies [6] 31
require re-training an already trained model like Stable Diffusion, which is time- and resource-intensive. We show that NEMO can reliably identify individual neurons responsible for memorizing specific training samples. Pruning these neurons effectively mitigates memorization, does not harm the general model performance, and provides a more permanent solution to avoid training data replication.
3

---

Non-Memorized
Non-Memorized
Template
Memorized
Density
Verbatim
-0.5
1.5
(a) Pairwise SSIM scores for different prompts.
(b) Neuron activations standardized by the 2-score.
Figure 2: Differences Between Memorized and Non-memorized Prompts. (a) depicts the distribu- tion pairwise SSIM scores between initial noise differences starting from different seeds. Since the noise trajectories are more consistent for memorized samples, the score reflects the degree of memo- rization. (b) shows the distribution of the 2-scores of each neuron in the first cross-attention value layer. Memorization neurons produce considerably higher activations, here depicted as standardized z-scores, for memorized prompts, allowing them to be identified by outlier detection.
NeMo: Localizing and Removing Memorization in Diffusion Models
NEMo, method for detecting memorization neurons, consists of two-step selection process:
(1) Initial Selection:
We first identify a broad set of candidate neurons that might be responsible for memorizing specific training sample. This initial selection is coarse- coarse-grained to speed up the computation among the many neurons in DMs. Consequently, it might select false positives, i.e. neurons not directly responsible for memorization.
(2) Refinement:
the refinement step, we filter neurons out to reduce the size of the initial candidate set. After refinement, we deactivate the remaining memorization neurons remove memorization.
In our study, apply this two-step approach of NEMO detect memorization neurons in the DM's cross-attention layers, the only components that directly process the text embeddings. Image editing research [17 10 shows that cross-atiin atssnttttnntoon layers highly influence the generated content, so we expect them to be the driving force behind memorization. We analyze the impact of blocking individual key and value layers in Appx. Appx.C.8aam and the influence blocking neurons in the convolutional layers in Appx. C.9 Our results show that the value layers in the down- and mid-blocks of the U- Net indeed have the highest memorization effect, whereas value layers the up-blocks barely affect the memorization. Deactivating the outputs of neurons in value layers completely blocks the information flow of the guidance signal and, hence, potential memorization triggers. Deactivating the key layers in cross-attention also impacts memorization but often impedes the image-prompt alignment. Similarly, deactivating neurons in the convolutional layers of the U-Net did not mitigate memorization. Therefore, we limit our search on memorization neurons to the value layers of the U-Net's down- and mid-blon blocks. While the identified neurons effectively mitigate the data replication problem, emphasize that other parts of the U-Net might also play a crucial role in memorizing training data. Specifically, the identified neurons trigger the data replication, which is then executed by other parts of the U-Net, such convolutional and fully connected layers. Deactivating the memorization neurons value layers effectively interrupts the memorization chain and replication process. Before detailing the two steps of NEMo's 's selection process in Sec. 3.2 and Sec. (.3] first introduce how we quantify memorization strength in the next section. We provide detailed algorithmic descriptions for each of NEMo's components in Appx. D
3.1 Quantifying the Memorization Strength
Intuitively, the denoising process of DMs for memorized prompts follows rather consistent trajectory to reconstruct the corresponding training image, yielding image generations with little diversity, Conversely, the denoising trajectory highly depends on the initially sampled noise for non-memorized prompts [46). We measure the similarity between the first denoising steps for different initial seeds as a proxy to compare the denoising trajectories and to quantify the memorization strength. The higher the similarity, the more consistent the denoising trajectories, which indicates stronger memo- rization. Let .T (0, I) be the initial noise following the denoising process described in Sec. 2.1

---

Let co(2T, T.y) further denote the initial noise prediction. We found that the normalized difference between the initial noise and the first noise prediction cg(xr, T. y) II for memorized prompts is more consistent for different seeds than for non-memorized prompts. We visualize this phenomenon for some initial noise differences in Appx. C.6
To detect the grade of memorization, we, therefore, use the similarity between the noise differences 5(6) and 8(2) generated with seeds i and j as proxy. We measure the similarity with the common structural similarity index measure (SSIM) 44. A formal description of the SSIM E [-1, 1] score and an additional experiment outlining how the SSIM score can be used to detect memorization in the first place is provided in Appx. B.4
higher SSIM indicates higher similarity between the noise differences, reflecting a higher degree of memorization. Notably, the SSIM computation only requires single denoising step per seed, which makes the process fast. To set a memorization threshold Tmem- starting from which we define a sample memorized, we first compute the mean SSIM on a holdout set of non-memorized prompts. We compute the pairwise SSIM between ten different initial noise samples for each prompt and take the maximum score. After that, average the scores across all prompts and set the threshold Tmem to the mean plus one standard deviation. We consider the current image generation non non-memorized if the maximum pairwise SSIM scores are below this threshold Temm Fig. 2 shows the distribution of SSIM scores for memorized and non-memorized prompts, demonstrating that memorized prompts lead to substantially higher score.
3.2 Initial Candidate Selection for Memorization Neurons
With measure for quantifying the strength of memorization defined, move to detail the first step of our NEMo' localization. Our initial neuron selection procedure is based on the observation that activation patterns of memorized prompts differ from the ones of non-memorized prompts on the neuron level. Fig. 2b underlines this observation by plotting the standardized activation scores for memorized and non-memorized samples in the first value layer. Leveraging this insight, we identify memorization neurons as the ones that exhibit an out-o--distribution (OOD) activation behavior. We first compute the standard activation behavior of neurons on a separate hold-out set of non-memorized prompts. Then, we compare the activation pattern of the neurons for memorized prompts and identify neurons with OOD behavior. Let the cross-attention value layers of a be E {L..... L). We denote the activation of the i-th neuron in the I-th layer for prompt y as (y The activation values are averaged across the absolute neuron activations for each token vector in the text embedding. Let ie be the pre-computed mean activation and of the corresponding standard deviation for this neuron. To detect neurons potentially responsible for the memorization of a memorized prompt we compute the standardized z-score 20), defined as
dl(v) m of
The 2-score quantifies the number of standard deviations of by which the activation (f(v) is above or below the mean activation μ1.- Here, the activation a'(y) is calculated by taking the mean over the absolute token activations. To identify neuron as exhibiting threshold Pxst and assume that neuron i in layer has OOD behavior if [æ((y) > 8uc. The lower the threshold Pacs. the more neurons are labeled as OOD and added to the memorization neuron candidate set. An algorithmic description of the OOD detection step is provided in Alg. 2 Appx. D.2
OOD activation behavior, we set
Fig. 2a shows that the pairwise SSIM score can be used to measure the generated sample's degree of memorization. Hence, to get an initial selection of memorization neurons, we calculate the standardized z-scores for all neurons and start with a relatively high value of 0sst 5. We deactivate all neurons with OOD activations given the current threshold Ooet. i.c., setting the output of a neuron to 0 if [x](w) > Omn. to reduce the memorization strength. If, after deactivating these neurons, the memorization score is not below the threshold Tmem, we then iteratively decrease the activation threshold 6m by 0.25 and update the candidate set until the target memorization score Tmem is reached.
The activation patterns of some neurons in the network show high variance, even on non-memorized prompts. Such neurons can also be memorization neurons, but due to their high activation variance, they might not be detected by our OOD approach based solely on the z-scores. Therefore, we also add the top-k neurons of each layer with the highest absolute activation on the memorized prompt y
5

---

to our current candidate set to account for such high-variance neurons. We start by setting k = 0 and increase each iteration by one if the memorization score is still above the threshold 7mem- We detail our initial selection process in Alg. in Appx. D33 All neurons identified by our OOD approach and the neurons with the highest activations are then collected in the neuron set Sminal- Since not all neurons in set initial might be memorization neurons, we refine this set in the next step.
3.3 Refinement of the Candidate Set
In this step, we take the set of identified neurons Stefined = Smitial and remove the neurons that are actually not responsible for memorization. To speed up this process, we first group the identified neurons layer-wise, leading to the neuron set S relimed for layer l. We iterate over the individual layers I E (1..... L} and re-activate all identified neurons Seelned from a single layer I while keeping the identified neurons in the remaining layers deactivated.
We then compute the SSIM-based memorization score and check it is still below the threshold Tiem- If the memorization score does not increase above the threshold Tmem. we consider the candidate neurons St Sefinedd of layer as not memorizing and remove them from our set of neurons Seetnndd After iterating over all layers, the set Sienned only contains neurons from layers that substantially influence the memorization score.
Next, we individually check each remaining neuron in the set Spefiedd by re-activating this particular neuron while keeping all other neurons in the set Spelined deactivated. Again, if the memorization score computed on the remaining deactivated neurons does not exceed the memorization threshold Time- we remove this neuron from the set Stefined After iterating over all neurons in Stefined we consider the remaining neurons as memorizing and denote the final set of memorization neurons as Stmat- We detail this refinement approach in Alg. in Appx. D.4
4 Experiments
We now empirically evaluate NEMO's localization in text-to-image DMs.
Models and Datasets: We follow current research on memorization in DMs 46 31 and investigate memorization in Stable Diffusion v v1.4 (32). Our set of memorized prompts consists of 500 LAION prompts [35] provided by Wen et al. 46). We analyzed the prompts using the Supervsue
Descriptor (SSCD) score (28, a model designed to detect and quantify copying in DMs. The lower the score, the less similar the contents the image pairs. Additionally, we split the dataset into verbatim- (VM) and template-memorized memorized (TM) samples to enable a more detailed analysis of results. The hyperparameter selection and experimental conduction are independent of the type of memorization. If not further specified, we used the same hyperparameters for all the experiments in the paper.
Images generated by VM prompts match the training image exactly, i.e., pixel-wise, independent of the chosen seed. TM prompts, on the other hand, reproduce the general composition of the training image while having some non-semantic variations fixed image positions. Details about the analysis and the annotation can be found in Appx. C.I
Other publicly available models, like Stable Diffusion v2 and Deep Floyd [3), are trained more carefully curated and deduplicated datasets. We thoroughly checked for memorized prompts using the tools by Webster [5] and our SSIM- based memorization score but could not identify any properly memorized prompts. This result aligns with related research on memorization in DMs [46 31.
We split our metrics into memorization, diversity, and quality metrics. The memorization metrics measure the degree of memorization still present in the generated images. We generate ten images for each memorized prompt with activated/deactivated memorization neurons and measure the cosine similarities between image pairs using SSCD embeddings to quantify the memorization. We denote this metric by SSCD Since the generated images without deactivated neurons also differ in their degree of memorization from the original training images, we additionally measure the degree of memorization towards the original training images and denote this metric SSCDoeig Higher SSCD scores indicate a higher degree of memorization.
Metrics:
Our diversity metric assesses the variety of images generated for the same memorized prompt with different seeds. Diversity is usually low for memorized samples, and generated images almost always depict the same image. Deactivating memorization neurons increases the diversity in the generations.

---

Table Impact of Deactivating the Memorization Neurons. Keeping all neurons active (Ist row) and randomly deactivating neurons (3rd row) has no impact on memorization. However, deactivating the memorization neurons located by NEMO (8th row) successfully mitigates memorization, increases diversity, and maintains prompt alignment. These results are comparable to the pradient-hased mitigation strategies adjusting the prompt embeddings (2nd row) and the attention scaling (4th row). Adding random tokens also reduces memorization. However, for or 4 tokens, the memorization, as quantified by the SSCD scores, is still higher than with deactivated memorization neurons. Adding 10 random tokens leads to comparable mitigation but also reduces the prompt alignment score.
Setting
Memorization Type Desctivated Nearoas
J,80DDi
SSCDD
Deam
Acuav
All Neuron Activos (Defsult)
Vedbati
Templane
0.04
0.16
13:00.0
1.400
0.30 0.01
0.17 0.06
01.02
0.00
Prompt Embedding Adjisument (Wen 18
Verbatim
0.04 0.00
0.081:005
0.08 0.05
0.50 1: 0.00
Templane
0.02
0.08 0.03
1.19::000
0..02
Deactivating Random Nearons
Verbatim
Template
1.3.3
+ 18
0.80 0.111
0.05 0.04
0.500.100
0.5074 0.00
0.00:100001
0.16:100.0
0.00
0.02
Attention Scaling (Res et al.
Template
Vedatin
0
0.05 0.00
0 + 0.04
0.14
0.10
0.12:1:005
0.15
0.31 0.00
0.00
Ackligg Random Tokke (Somepulli et al.
35
Tomplac
Vedutin
0
0
0,04 + 0.08
0.59 + 0.31
0.16 2:00
0.68 0.31
0.07 0.33
0.31 =: 0.00
0.31 0.002
Adding Random Tokens (Somepalli et
Verbatim
Template
0
0
0.09 1 0.06
0.04 0.00
0.12
0.05
0.15 0.00
0.15:1:0.01
0.30 0.00
0.30 0.02
Acking 10 Rondom Tokers (Somepalli et al.
Verlatio Template
0
0
0.05 a: 0.00
0J83 + 0.02
0,08 0:05
0.07 0.05
0.11 0.05
0.12 0.12:1:0.04
0.28 0.03
0.29 :: 0.00
Deaetivating Memorization Neuroms (NKMo)
Verhatinn
Template
21 + 18
4
0.06 +: 0.08
0.09 + 0.06
0.05 + 0.04
0.10 + 0.07
0.16 + 0.06
0.12 +: 0.05
0.31 = 0.02
0.31
0.02
We quantify the sample diversity by computing the pairwise cosine similarity between the SSCD embeddings of different images generated for the same prompt. We refer to this metric as DSSCD, for which lower values indicate more image diversity.
To assess the overall image quality of a DM with activated/deacciivtee neurons, we compute the Fréchet Inception Distance (FID) (18), the CLIP-FID (21). and the Kernel Inception Distance (KID) 3 on COCO 22 prompts. All quality computations follow Parmar et al. [26] to avoid biased results. Additionally, we compute the similarities between the generated images and the ipput prompts using CLIP scores (29) to ensure the alignment ACLIP between generated images and their prompts. The higher the alignment, the better the generated images represent the concepts described in the prompt.
Importantly, we use different seeds for detecting memorization neurons with NEMO and the metric computations avoid undesired biases during the evaluation due to seed overfitting. We always state each metric's ' median value and absolute deviation across ten seeds, except the quality metrics (FID, CLIP-FID, KID, and ACLP),, for which we used five different seeds.
Memorization Threshold: We set the memorization score threshold to Tmem = 0.428, which corresponds to the mean plus one standard deviation of the pairwise SSIM score between initial noise differences measured on a holdout dataset of 50,000 LAION [35] prompts.
Baselines: As baseline, we repeated the image generations five times but replaced the deactivated memorization neurons with random neurons from the same layer. We also generated images using the inference mitigation strategy proposed by Wen et al. [46.. which performs a gradient- based adjustment of the text embeddings. Importantly, gradient-based mitigation strategies are memory- intensive, particularly for larger batch sizes. NEMo, however, computes no gradients, which enables the approach to also work on machines with limited computing resources. Additionally, we also selected the attention scaling method by Ren et al. (31 and the addition of random tokens, proposed by Somepalli al. [36), as baselines.
4.1 Localizing Memorization Down To Individual Neurons
We begin by demonstrating the effectiveness of our memorization localization method. Tab. T presents the quantitative results for images generated with the identified memorization neurons deactivated. NEMo detected median of 4 and 21 memorization neurons for VM and TM prompts, respectively. For VM prompts, deactivating these memorization neurons significantly decreases memorization, as reflected by low memorization metrics SSCDorig and SSCD while increasing the image diversity in terms of pairwise similarity DSSCD- However, the SSCDong does not change noticeably for TM prompts.
7

---

Memortattion
Blocked
Figure 3: Impact of Deactivating Memorization Neurons. The top row shows images generated with memorized prompis, closely replicating the training images. The bottom row demonstrates that deactivating memorization neurons increases diversity and mitigates memorization. Notably, only few neurons (counts indicated by digits in the boxes) are responsible for these memorizations.
12
Prompt Count
30
Verbatim
Template
Neuron Count
10
20
10
20
40
1
2
Number of Neurons
Layer Index
(a) Memorization
per prompt.
(b) Memorization neurons per layer.
Figure 4: Distribution of Memorization Neurons. (a) shows the number of prompts that are memorized by fixed number of neurons, e-g., the verbatim memorization of 28 prompts is located in single neurons. (b) depicts the average number of memorization neurons per layer and prompt.
This behavior results from the fact that TM prompts typically memorize specific parts of the original training image, such as objects or compositions, rendering the SSCDong metric less informative. In contrast, the SSCDGen score, which compares similarities between images generated with and without the deactivated neurons, provides a more accurate measure. This score highlights that deactivating the identified neurons effectively alters the images and mitigates memorization. Importantly, the image- prompt alignment ACLIP remains constant in all cases, indicating that deactivating memorization neurons does not result in misguided image generations. We visualize examples of deactivating memorization neurons to avoid data replication and increase diversity in Fig.
Comparing the results of deactivating the neurons identified by NEMO with those obtained from randomly deactivated neurons highlights that only a specific subset neurons is actually responsible for memorizing a prompt. While deactivating the identified memorization neurons significantly impacts both memorization and the diversity of the generated images, randomly deactivating neurons has no noticeable effect.
Moreover, the mitigation effect of deactivating memorization neurons is comparable to the state- of-the-art method of adjusting the prompt embeddings [46]. Yet, adjusting the prompt embeddings requires gradient computations for each seed and prompt, which are time- and memory- memory-expensive, especially with large batch sizes. In contrast, once the memorization neurons are identified using gradient-free NEMO, no additional computations are needed during image generations, thus adding no overhead to the generation process.
4.2 Analyzing the Distribution Memorization Neurons
Next, we analyze the distribution of the memorization neurons. Fig. 4a shows the total number of neurons responsible for memorizing specific prompts. Typically, small of neurons is responsible for verbatim memorization. For instance, 28 VM prompts from our dataset are memorized by single neuron. Additionally, five or fewer neurons replicate two-thirds VM images, indicating that verbatim memorization can often be precisely localized within the model. Template memorization can also frequently be pinpointed to a small set of neurons, with about 30% of TV replication triggered by five fewer neurons.

---

1.0
17.5
Random
Top-K
SaCD
1I.0
SBCD Value
0.8
16.5
TM
SSCDeen
16.0
0.6
6.25e-3
0.4
6.00e-3
KID
5.75e-3
0.2
5.50e-3
0.0
25 100
250
500
750
-1
-0.75
-0.5 -0.25
0 0.25 0.5 0.75
Number of Neurons Blocked
Scaling Factor
(a) Assessing image quality using FID and KID.
(b) Scaling the activations of memorization neurons.
Figure 5: Image Quality and Sensitivity to Scaling Factor. (a) assesses the generated images' quality when blocking an increasing number of neurons. As can be seen, the FID and KID values vary only slightly, indicating that blocking neurons identified by NEMo does not negatively affect image generation quality. Gray lines indicate the baseline without any neurons blocked. (b) investigates the effect of scaling the memorization neurons' activations by scaling factor instead of deactivating them (scaling by zero). Whereas positively scaling memorization neuron activations only slightly reduces memorization, negative scaling reduces the memorization not any further.
However, approximately one-third of TM prompts distributed across 50 or neurons. We hypothesize that this broader distribution results from the higher variation in generated images for TM prompts, where memorization spread across multiple neurons leads to increased image diversity. In contrast, VM prompts, often memorized by small group of neurons, consistently produce the same image without variation. More detailed plots of the identified neurons can be found in Appx. C.2
Interestingly, we identify two neurons the first cross-attention value layer responsible for the verbatim memorization of multiple prompts. Neuron #25 in this layer is associated with depicting people, while neuron #221 is responsible for memorizing multiple podcast covers. Together, these neurons account for memorizing 17% of our dataset's VM prompts. Similarly, neurons #507 and #517 in the third value layer are responsible for multiple TM prompts describing iPhone cases. The impact of deactivating these neurons the image generation of memorized prompts is visualized in Appx. C.3, We also plot the distribution of the average layer-wise number memorization neurons per prompt in Fig. 4b Neurons responsible for VM prompts mappings of the first cross-attention layers within the U-Net's down-blocks (each block contains two aots-tat layers). A similar pattern appears for TM prompts, although value layers located deeper in the U-Net seem to play more crucial role for TM prompts than for VM prompts.
primarily located in the value
4.3 Memorization Neurons Hardly Influence Non-Memorized Prompts
Until now, focus has been on the impact of deactivating memorization neurons memorized prompts. In this part, we investigate how these neurons influence non-memorized prompts and the overall image quality of the DM. To assess their impact, deactivate varying numbers of memorization neurons, ordered by their frequency of occurrence as identified our experiments, and compute the FID and KID scores on the COCO dataset. We also repeat the generations by deactivating the same number of randomly selected neurons that are not among the identified memorization neurons. As shown in Fig. Sal there is no significant degradation in the image quality when blocking either the random neurons or the memorization neurons, even with up to 750 blocked neurons. This finding underscores the potential for pruning memorization neurons in DMs without compromising the overall image quality. The plot for the CLIP-FID metric and more detailed plots of the other two metrics, as well as an additional experiment measuring the disentanglement of the neurons in the value layers, can be found in Appx.
4.4 Ablation Study and Sensitivity Analysis
We further analyze the impact of each component of NEMO and its sensitivity to hyperparameter selection. We discuss the most crucial insights here, with the complete study included in Appx. C.7 First, we evaluate the impact of different memorization thresholds 0men- Lowering this threshold slightly increases the number of identified neurons but has negligible effect performance metrics.

---

Selecting this threshold based on statistics computed on a holdout set provides a simple yet effective way for hyperparameter selection.
Additionally, we compare the results of using both stages of NEMo versus a setting where we only perform the initial candidate selection, skipping the refinement process. Although memorization is successfully mitigated by deactivating the initially selected neurons, the number of identified memorization neurons increases substantially, resulting in median of 26.5 neurons (+22.5 neurons) for VM prompts and 674.5 neurons (+653.5 neurons) for TM prompts. This highlights the importance of the refinement stage in reducing the number of neurons necessary to mitigate memorization efficiently. To further test whether the assumptions about the statistics of memorization neurons hold, we applied NEMo set of 500 non- memommzeoi prompts not used calibrate our thresholds. As we further show, NEMo does not identify any neurons for most of the non non-memorized prompts, underscoring the validity of our assumptions.
We also compared the effect of completely deactivating the identified memorization neurons to down-scaling their activations by fixed factor. The SSCD scores in Fig. 5ig computed for different scaling factors, demonstrate that memorization is not fully mitigated when using a positive scaling factor. Conversely, negative scaling factors do not provide any additional mitigation compared to our default setting of deactivating the neurons (i.e., using a factor of zero).
Conclusion and Outlook
DMs have rapidly become a cornerstone of computer vision. Yet, problems like memorization of training samples can lead to undesired replication of potentially sensitive or copyrighted training images. Previous research has primarily focused on identifying memorized prompts and proposing mitigation strategies by adjusting the DM's input. However, there has been lack of understanding regarding the precise location of memorization within the model.
Our research provides novel insights into the memorization mechanisms in text-to-image DMs. Unlike previous studies that focused on identifying memorized prompts, our approach, NEMo, is the first to localize memorization within the model and pinpoint individual neurons responsible for it. Traditional pruning methods [T]] are orthogonal to our approach by pruning only structures to reduce the total parameter count of the model. Our memorization localization algorithm enables model providers to prune these memorization neurons, effectively mitigating memorization permanently without additional model training, which can be costly in terms of data and resources. Our mitigation can be executed without compromising the model's overall performance or the quality of the generated images, allowing model providers to deploy the resulting models without additional safeguards to prevent memorization.
There are several directions to expand and build upon our method for detecting neurons responsible for memorization. One intriguing avenue to investigate whether an adjusted version of NEMO can detect concept neurons [23]. These neurons are not responsible for memorizing certain prompt but for generating a particular concept. Such an approach could enable model providers and users to perform knowledge editing [15] and remove undesired concepts like violence and nudity. Another exciting application for NEMO is in large language models, also known for memorizing training samples [5]. Identifying neurons responsible for memorizing text from the training data could lead to new mitigation strategies.
Additionally, our insights could be interesting for developing new pruning algorithms for DMs to reduce the number of parameters while eliminating unintended memorization. As demonstrated in our experiments, pruning memorization neurons does not significantly impact the model's 's overall performance, which is crucial for effective pruning strategies.
Acknowledgments and Disclosure of Funding
This work has been financially supported by the German Research Center for Artificial Intelligence (DFKI) project "SAINT". The project also received funding from the Initiative and Networking Fund of the Helmholtz Association in the framework of the Helmholtz AI project call under the name "PAFMIM" with funding number ZTT--PP----277. Responsibility for the content of this publication lies with the author.
10

---

References
[1] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning.
pages 233-242. PMLR, 2017.
[2] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34: 34:10876-10889,
2021.
[3] Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD gans. International Conference on Learning Representations (ICLR), 2018.
[4] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization neural networks. 28th USENIX Security Symposium (USENIX Security 19), pages 267-284. 2019.
[5] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), 2021.
pages
[6] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
[7] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35: 13633-1663,,, 2022.
[8] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposiun (USENIX Security 23), pages 5253-5270, 2023.
[9] Satrajit Chatterjee. Learning and memorization. In International conference machine learning, pages 755-763. PMLR, 2018
[10] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Winter Conference on Applications of Computer Vision (WACV), pages 5331-5341, 2024.
[11] Gongfan Fang. Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[12] Vitaly Feldman. Does learning require memorization? short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
[13] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881 1-891, 2020.
[14] Matt Fredrikson. Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
[15] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing in diffusion models. In Winter Conference on Applications of Computer Vision (WACV), pages 5099-5108. 2024.
[16] Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On Memoriza- tion in Diffusion Models. arXiv preprint, arXiv:2310.02664, 2023.

---

[17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt-to-promp image editing with cross-attention control. In Eleventh International Conference Learning Representations (ICIR), 2023.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In Conference on Neural Information Processing Systems (NeurlPS), page 6629-6640, 2017.
[19] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In Conference on Neural Information Processing Systems (NeurIPS), pages 6840-6851, 2020.
[20] Erwin Kreyszig, Herbert Kreyszig, and E. J. Norminton. Advanced Engineering Mathematics. Wiley, tenth edition, 2011.
[21] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. In Eleventh International Conference on Learning Representations (ICLR), 2023.
[22] Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[23] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept Neurons in Diffusion Models for Customized Generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning (ICML), 2023.
[24] Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi, Zachary Chase Lipton, Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? In International Conference on Machine Learning, pages 23536 23557, PMLR, 2023.
[25] Casey Meehan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do ssl models have déjà vu? a case unintended memorization in self-supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
[26] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11400-11410, 2022.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, Pegeoreerfo Deep Learning Library. In Conference on Neural Information Processing Systems (NeurIPS), pages 8024-8035, 2019.
[28] Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self-supervised descriptor for image copy detection. In Conference on Computer Vision and Pattern Recognition (CVPR), pages 14512 14522, 2022.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference Machine Learning (ICML), pages 3748-8763 2021.
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint, arXiv:2204.0611 2022.
[31] Jie Ren, Yaxin Li, Shenglai Zeng, Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and Mitigating Memorization Text-to-image Diffusion Models through Cross Attention. arXiv preprint, arXiv:2403. 1052, 2024.
12

---

[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution imgge synthesis with latent diffusion models. In Conference on Computer Vision and Pultern Recognition (CVPR), pages 10684-10695, 2022.
[33] Olaf Ronneberger, Philiip Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Inter- vention (MICCAI), pages 234 241, 2015.
[34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[35] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaezmarezyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[36] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 01477783-47803 2023.
[37] Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. In Conference on Neural Information Processing Systems (NeurIPS), pages 12438- 12448. 2020.
[38] StabilityAI. DeepFloyd IF: a novel state-of-the-art open-source text-to-image model with a high degree of photorealism and language understanding. luttps://www.eeepff ai/ deepfloyd-ir 2023. Retrieved on 2023-114 11-08.
[39] Cory Stephenson, Abhinav Ganesh, Yue Hui, Hanlin Tang, Yue Chung, al. On the geom- etry of generalization and memorization in deep neural networks. In International Conference on Learning Representations, 2020.
[40] Lukas Struppek, Dominik Hintersdorf, Antonio De Almeida Correia, Antonia Adler, and Kristian Kersting. Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks. In International Conference on Machine Learning (ICML), pages 20522-20545, 2022.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), pages 5998-6008, 2017.
[42] Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, and Joost van de Weijer. Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing. In Conference on Neural Information Processing Systems (NeurIPS), 2023.
[43] Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Boenisch. Memorization in Self-Supervised Learning Improves Downstream Generalization. In International Conference Learning Representations (ICLR), 2024.
[44] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13:600- 612, 2004.
[45] Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint, arXiv:2305.0869. 2023.
[46] Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, Explaining, and Mitigating Memorization in Diffusion Models. In International Conference on Learning Representations (ICLR), 2024.
13

---

[47] TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K. Ryu. Diffusion Probabilistic Models Generalize when They Fail to Memorize. In ICML Workshop on Structured Probabilistiic Inference & Generative Modeling, 2023.
[48] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2016.
[49] Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin. Intriguing properties of data attribution on diffusion models. In The Twelfth International Conference on Learning Reprexentations, 2024. URL http://pwwreeeeeeeeeeeeoarra//aa
14

---

A Limitations
For certain memorized prompts, our method identifies a set of over 100 neurons. Upon closer examination, we found that in these cases, memorization is distributed across many neurons in various layers, rather than being concentrated in a small group. Even deactivating a substantial number of neurons in the network does not eliminate memorization for these prompts. However, such instances, primarily TM prompts, were rare in our experiments. We provide examples of highly memorized prompts in Appx. C.4
Highly Memorized Prompts:
Mitigating Template Memorization Is Harder Than Verbatim Memorization: Deactivating memorization neurons is effective for mitigating verbatim memorization, and often requires only a small number of neurons to be deactivated. In contrast, mitigating template memorization often requires deactivating more neurons, and even then, complete removal of memorization is not always possible in some rare cases. This difficulty arises because template memorization frequently results in diverse generations, with the memorized content corresponding to only certain aspects of the image. Distinguishing between these memorized parts and the remaining image parts is not always clearly achievable with our SSIM-based memorization strength. However, we emphasize that for the majority of prompts, deactivating the identified memorization neurons successfully removes template memorization.
Runtime: For most memorized prompts, NEMo detects the memorization neurons in a few seconds. We timed the runtime of NEMo and found that average NEMo can identify memorization neurons for verbatim memorized prompts within 14.2 seconds, while for template memorized prompts memorization neurons
identified in 43.7 seconds on average. As discussed in the previous paragraph, mitigating and localizing memorization for template memorized prompts is harder. We suspect this is one reason why NEMo's runtime is slightly longer for template memorized prompts. To get further insight into how long the runtime of each part of our algorithm is, we also timed the runtime for the algorithms D1-D4 separately. The results can be seen Tab.
Table 2: NEMO localize the neurons responsible for memorization efficiently. The average runtime (in seconds) for Alg. and Alg. iis helow one second, while the runtime for Alg. 3s below 10 seconds. While the runtime for Alg. 4is longer than for the other parts of NEMo, the runtime is still only 45 seconds for TM. Alg. 4 has the longest runtime for TM, since the initial candidate set is larger than for verbatim memorized samples.
Memorization Type
Al
A2
A3
A4
Total
VM TM
0.21
0.29
2.31
12.85
15.68
0.18
0.25
6.09
39.37
45.90
B Experimental Details
B.1 Hard- and Software Details
We performed all our experiments NVIDIA DGX machines running NVIDIA DGX Server Version
5.2.0 and Ubuntu 20.04.5 LTS. The machines have 1.5 TB (machine 1) and 2 TB (machine 2) of RAM and contain NVIDIA Tesla V100 SXM3 32GB (machine 1) NVIDIA A 100-SXM4-40GBB (machine 2) GPUs with Intel(R) Xeon(R) Platinum 8174 (machine 1) and AMD EPYC 7742 64-core (machine 2) CPUs. We further relied on CUDA 12.1, Python 3.1 and PyTorch 2.2.2 with Torchvision 0.17.2 27 for our experiments. All investigated models are publicly available on Hugging Face. For access, we used the Hugging Face diffusers library with version 0.27.1.
We provide a Dockerfile with our code to make reproducing our results easier. In addition, all training and configuration files are available reproduce the results stated in this paper.
B.2 Model and Dataset Details
Experiments were mainly conducted on Stable Diffusion v1-4 [32] publicly available at hhttps //uggiingace
v1-4 All details regarding the data, training
15

---

parameters, limitations, and environmental impact are available at that URL. The model is available under the CreativeML OpenRAIL M license
The investigated prompts originate from the LAION2B-en 35 dataset used to train the DM. The set of memorized prompts is taken from Wen al. 46 who collected the prompts by using the tool of Webster [45]. The LAION dataset itself is licensed under the Creative Common CC-BY 4.0, The images of the LAION dataset might be under copyright, so we do not include them in our code base; we only provide URLs to retrieve the images directly from their source.
B.3 Experimental Details and Hyperparameters
All images depicted throughout the paper are generated with fixed seeds, 50 inference steps, and classificr-free free guidance strength of 7 using the default DDIM scheduler. Notably, the seeds used for generating the images and computing the evaluation metrics differ from those used for our detection method NEMO to avoid seed overfitting.
During detection with NEMO, no classifier- free guidance was used, which speeds up the detection since only a single forward pass per seed is required, compared to an additional forward pass on the null-text embedding with classifier-free guidance. We always used ten different seeds for each prompt. The threshold on the SSIM memorization score was set to Tmem = 0.428 during the experiments in the main paper. We vary this threshold and analyze its impact in our sensitivity analysis in Appx. C.7
We all experiments detection with NEMo and the generations for the metric computations with half- precisionn (11oat16) to reduce the memory consumption and speed up the computations.
B.4 Structural Similarity Index Measure (SSIM)
We quantify the memorization strength during our experiments using the structural similarity index measure commonly used in the computer vision domain to assess the similarity between image pairs. Our memorization score is computed as follows: Let Æт N(, I) be the initial noisy image. Let (I T, y) further denote the initial noise prediction without any scaling by the scheduler. We found that the normalized difference between the initial noise and the first noise prediction 6= = eş (IT,T, y) zT for memorized prompts is substantially more consistent for different seeds than for non-memorized prompts. To detect the grade of memorization, we, therefore, use the similarity between the noise differences g(te) and 86) generated with seeds and j as a proxy. We measure the similarity with the common structural similarity index measure (SSIM) 44). The SSIM E [0, 1] between two noise differences 866) and 865) is defined by
(2/u,1 + (20 + C2) SSIM (80)))88888 (863) (4 + 13 + Co) + + +2)
The parameters (bi, Mj and o²? of denote the mean and variance of the pixels in 8(0) an 866) respectively. Likewise, dij denotes the covariance between the images. Following the original paper, C1 and C2 are small constants added for numerical stability.
A higher SSIM indicates higher similarity between the noise differences, reflecting a higher degree of memorization. Notably, the SSIM computation only requires single denoising step per seed, which makes the process fast.
Indeed, the SSIM score itself can be used detect memorization. We visualized the different SSIM score distributions for non-memorized and memorized prompts in Fig. 2a To underline this observa- tion with quantitative metrics, our SSIM-based method. We measured the efficiency of the SSIM score for memorization detection on our dataset of memorized and memorized prompts. Without extensive hyperparameter tuning, this detection method achieves an AUROC of 98%, an accuracy of 94.2% (using a naive threshold of 0.5 on the SSIM similarity), and a TPR @ FFPP of 87.6%. Since the amount of memorization of template memorized prompts varies sigmificantly. we repeated the computation for detecting the verbatim memorized prompts. Here, the SSIM approach even achieves an AUROC of 99. 99.64%, an accuracy of 97. .3% (with a threshold of 0.5), and a TPR@ @ I %FPR of 98 08.21%. These results indicate that the SSIM score can also be used to detect memorization reliably in the first place. However, detection is not the focus of the paper but the localization on the neuron level.
additional experiments explore the detection capabilities of
Available at http//shtt.a..ooooee///aaaaaaaaa http://ffffuusoooooomoriiaaaoooo
16

---

Additional Results
C.1 Distinguishing Between Different Types of Memorization
Webster [45] distinguished between verbatim and template memorized prompts. Verbatim memorized prompts lead the exact reconstruction of training samples, while sumplate-memorizee prompts replicate the composition and structure of the training image. To provide more fine-gr fine-grained analysis of our results, we classify the prompts our dataset into these two categories. We distinguish between both types by computing the SSCD [28] scores between the original training image and ten generations with different seeds. We then classify prompt verbatim memorized if the maximum SSCD score computed otherwise. Fig. plots the distribution of SSCD scores both datasets. We manually inspected and classified the prompts where the original training image is no longer available (16 out of 500).
cosine similarity exceeds threshold of 0.7 and template memorized
Template Memorization
Verbatim Memorization
Density
0.0
0.2
0.4
0.6
0.8
1.0
Maximum SSIM Score
Figure 6: We compare the maximum SSCD score between ten generated images and the original training sample. We categorize the memorized prompts into verbatim memorized if the SSCD score exceeds 0.7 and into template memorized prompts otherwise.
C.2 Detailed Analysis of the Distribution of Memorization Neurons
120
35
60
15
Count
60
10
wilha
30
20
20
40
10
Template
20
500
1000
1250
1500
Number of Initial Neurons
Number of Neurons After Refinement
(a) Number of initial neurons per prompt.
(b) Number neurons per prompt after refinement.
Figure 7: Distribution of Memorization Neurons. We show the number of prompts that memorized by a fixed number neurons. (a) plots the number neurons found in the initial neuron selection. (h) shows the number of neurons after refinement. As drastically reduces the number of found memorization neurons for both the template and the verbatim memorized prompts.
observe, the refinement step
17

---

C.3
Detailed Quality Analysis
17.2
Neurons Blocked
17.0
Random
12.5
16.0
Top-K
16.6
CLP-FDD
12.0
16.4
16.2
Neurons Blacked
16.0
10.5
Random
15.8
Top-K
10.0
100
200
300
400
600
700
100
200
300
400
500
600
700
Number of Neurons Blocked
Number of Neurons Blocked
(a) Assessing image quality using FID.
(b) Assessing image quality using CLIP-FID.
0.0062
0.0060
0.0058
Neurans Blocked Random
0.0054
Top-K
100
600
700
Number Neurons Blocked
(c) Assessing image quality using KID.
Figure Image Quality Does Not Degrade When Deactivating Memorization Neurons, Depicted the generated images' FID, CLIP-FID (FID calculated using CLIP model), and KID scores when blocking an increasing number of neurons. For all three metrics, smaller values are better. As be seen, the FID, KID, and CLIP-FID values vary only slightly, indicating that blocking neurons identified by NEMo does not negatively affect image generation quality. Gray lines indicate base lines without any neurons blocked. We repeated the experiment with five different seeds. Depicted the mean values and the standard deviation.
Table We measured the disentanglement of neurons for deactivating top-k memorization neurons or randomly selected neurons. More specifically, we collected and compared the attention layer outputs for 500 non-memorized prompts with and without neurons deactivated and seeds. We then computed the average cosine similarity between corresponding outputs. The high similarities show that blocking neurons only has a negligible impact on the outputs of the attention layer, suggesting that other neurons substitute the functionality of blocked neurons non-meoriieed prompts.
Natrons
Neuroa Type
Block
Random
Top-K
Randen
Top-K
Random
Top-K
Rendom
Top-K
Random
Top-K
Random
Top-K
Random
Top-K
1.0
1.0
1.0
0.99
L.U
0.98
0.99
0.98
0.99
0.98
0.99
0.98
0.99
0.98
0,99
0.98
0.9
0.98
0.99
0.98
1.0
1.0
0.9
033
0.30
0.99
0.99
1.0
1.0
1.0
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
1.0
0.99
039
0.99
0.99
009
0.99
0.99
0.99
0.99
039
039
0.99
0.99
039
039
0.9
039
0.99
0.9
0.99
0.99
0.9
0.99
039
039
0.99
0.99
We ran additional experiments to analyze the impact and effects of deactivating neurons in the value We deactivated varying number of neurons, either randomly
layers. The result can be seen Tab. B selected or the top memorization neurons, and measured the similarity between the outputs of the cross-attention blocks with deactivated neurons and all neurons activated. Our results, stated in the rebuttal PDF, show that even deactivating large number of neurons only impacts the attention outputs marginally. We, therefore, conclude that other neurons can replace the functionality of specific neurons non-meoriieed prompts and that neurons in the value layers act rather independently. Yet. for neurons memorizing specific prompts, this memorizing functionality is not replaced by other neurons, explaining the mitigation effect of deactivating these neurons.
18

---

C.4 Highly Memorized Prompts
No Blocked Neurons
Blocked Memorization Neurons
Falmouth Navy Blue Area Rug by Andover Mills (256 memorization neurons)
Renegade RSS Laptop Backpack View (232 memorization neurons)
Grieve Cream/Navy Area
Bungalow Rase (136 memorization neurons)
pleat
neurons)
Designart Bohin] Lake Panorama Seashore Canvas ArtPrint Panels (129 memorization neurons)
Red Mums
Throw Pillow (272 memonization neurons)
Dreamscape iPhone Cases
Mermaid iPhone
Small (379 memorization
Fish Lodge Bass Tapestry Throw Blanket (107 memonzation neurons)
Wita Pirelli Scorpion Verde All Season 255/60 R18 112H (101 memonzation neurons)
Australian Silky Terrier Print Car Seat Covers-Free Shipping (242 memorization neurons)
Figure 9; Memorization of highly memorized prompts is distributed across many neurons in various layers, rather than concentrated in a small group of neurons. We show examples such prompts and the impact of deactivating the identified memorization neurons. The number of memorization neurons in each case is stated behind each prompt.
19

---

C.5 Examples for Memorization of Single Neurons
To illustrate that some single neurons are responsible for memorizing multiple training prompts, generated images with and without these specific neurons deactivated. In Fig. aO Fig. [1] we only deactivate single neuron each in the first value layer, whereas in Fig. 12 we deactivate two neurons in the third value layer.
No Blocked Neurons
Blocked Neuron #25
Passion "Lifted (1985)
Aretha Franklin Files $10 Million Suit Over Parti LaBelle Fight Story On Satire Website
Rambo Rocky Spin Off Sylvester Stallone gibt Updates
Prince Meunites With Warner Brothers, Plans New Album
Here's Who lan McShane May Be Playing in <i> of Thronesecio- Season Six
Canceled By Regancy
George R.R. Martin Donates $10,000 to Wolt Sanctuary for 13-Year-Did Fan
Silverman Pilot Diary
Freddy Adu Signs For Yet Another Club You Probably Don t Know
Hanks Film Adaptation Eggers <io-The Circle-ot>
"Lsten to Ricky Gervais Perform "Slough" David Brent"
Figure 10:
We found that neuron #25 in the first cross-attention layer's value mapping is responsible for verbatim memorization of multiple prompts, all associated with depicting people. Deactivating this single neuron mitigates the memorization and introduces diversity into the images (right columns) compared images generated with all neurons active (left columns). Generations were conducted with seeds different from the seeds used for the neuron localization process.
20

---

No Blocked Neurons
Blocked Neuron #221
No Pimits
No Himiss
Vo Phatts
Nh Chmii
COSA
al Hantt
Bininess
Dusiness Worme
Business Worran
Business Worrn
Busines Woimss Aut
Wombo
mimiic Rncertion
The No Limits Business Woman Podcast
Mothers influence on young hippo
Talks on the Precepts and Buddhist Ethics
INSIGITS
INSTORRE
INSIGITS
INSIGIPSS
UIRTAF
Langia WIN
HEUL Insights
HEAL' WADEET
TEAIE
HEILK
MASIITO
MARET
WANSSEF
The Health Mastery Café with Dr. Dave
Panslen
Parsten
Passien Pedccatt
Panslen Pediant
llassn proosti
PasstonPoPwea Pasiast Prodist
Passss POCUS
Pertcctt
Pediaat
Sodsss
Passosn You Pusal
Passion. Podcast. Profit.
Living in the Light with Ann Graham Lotz
Figure We found that neuron #25 in the first cross-attention layer's value mapping is responsible for verbatim memorization of multiple prompts, all associated with depicting people. Deactivating this single neuron mitigates the memorization and introduces diversity into the images (right columns) compared to images generated with all neurons active (left columns). Generations were conducted with seeds different from the seeds used for the neuron localization process.
21

---

No Blocked Neurons
Blocked Neurons #507 #517
Beautiful Space iPhone Case
Kimi Doesn't Know - Phone Case
Pink Icelandic Poppy
iPhone Phone Cases
X
Peace Love Massage iPhone 6 Case
Sloth in Space iPhone Case
Lavender Flames iPhone Xs Max Case
porter robinson madeon shelter Clear iPhane Case
ZZnn Buddha iPhone 6s Plus TOUGH Case Unique Black and White Buddhist Art "BBliss of Being" Zen Meditation iPhone 65 Plus case iPhone 6 6s Plus Tough Case 1'
We found that neurons #507 and #517 in the third cross-attention layer's value mapping is responsible for template memorization of multiple prompts describing iPhone cases. Deactivating these two neurons mitigates the template memorization and introduces diversity into the images (right columns) compared images generated with all neurons active (left columns). Image generations were conducted with fixed seed different from the seed used for the neuron localization process.
Figure 12:
22

---

C.6 Comparison of Initial Noise Predictions
Memorized Prompts
Memorized Prompts
With Blocked Neurons
Non-Memorizer Prompts
Hamedi
Figure 13: Visualizations for generated images and the noise differences between the predicted noise after the first denoising step and the initial Gaussian noise. Noise differences for memorized prompts (left column) have low diversity and are already structurally similar to the final image. The noise differences for non-memorizee memomizod prompts (right column) show no clear structure and differ substantially for different noise initializations. Deactivating the memorization neurons detected with NEMO (middle column) removes the structure in the initial noise differences and adds more diversity, leading to diverse image generations.
23

---

C.7 Ablation Study and Sensitivity Analysis
We conduct an ablation study to investigate the impact of the individual components of NEMo. Additionally, we analyze the sensitivity of the memorization threshold Tmem and explore alternatives to deactivating neurons by setting their activations zero. The results for the various settings are presented in Tab. 4
The first two rows provide evaluation results for the model with all neurons active and with randomly deactivated neurons. Both scenarios exhibit strong memorization. The third row shows the results of blocking the neurons identified as memorizing by NEMo, using the threshold Tmem 0.428 as specified in the main paper. This threshold corresponds to the mean SSIM memorization plus one standard deviation, calculated on a holdout set of 50,000 non-memorized LAION prompts. In row four, we repeat this setting using classifier- free guidance (CFG) with a guidance strength of 7.0, as opposed to our default setting without CFG. Detection with CFG further reduces the number of detected memorization neurons. However, the SSCD scores indicate slightly increased memorization after deactivating the identified neurons. Additionally, running NEMo with CFG doubles the number of forward passes in the U-Net since a separate noise prediction is generated for each initial seed.
Rows five to seven display the results of varying the memorization threshold Tmem- Specifically, we adjust the threshold to one standard deviation below the mean SSIM score, to the mean, and two standard deviations above the mean. A lower threshold identifies more neurons. However, for lower thresholds, the metrics are comparable to those obtained with our default threshold value (Fiem 0.428). Increasing the threshold reduces the number of identified neurons but slightly increases memorization, measured by the SSCD scores. Thus, trade-off exists between reducing the number of identified memorization neurons and their memorization mitigation effect. addition, we provide heat maps to directly compare the impact of different thresholds Tiem used during the initial selection and the refinement step Fig. Fig. 15 further compares the SSCD scores for varying the threshold values.
Rows eight and nine examine the impact of removing the refinement step incorporating no neurons with top-k activations during the initial selection. As anticipated, without refinement, the number of identified neurons increases substantially. Despite this, the various metrics remain comparable to those obtained after the refinement step. even with more neurons deactivated. This underscores the robustness of image generations against pruning ofi-s--distributiin (OOD) neurons. Without the top-k selection, NEMo identifies a larger set of neurons. However, deactivating these neurons does not mitigate memorization as effectively as with a top-k search. Notably, for template verbatim prompts, the SSCD is substantially higher without top-k, indicating increased memorization.
In the remaining rows, we explore the impact of scaling the activations of memorization neurons instead of deactivating them. With negative scaling factors, the results are comparable to those of completely deactivating the neurons. For positive scaling factors, however, the generated images demonstrate higher degrees of memorization, with scaling factor of 0.75 having almost no influence on memorization.
We also apply NEMO to set of 500 LAION non-memorized prompts, different from the 50,000 prompts used to set the memorization threshold. For 442 of these prompts, NEMo identified no memorization neurons, which is to be expected since these prompts show no memorization behavior. For the remaining prompts, median of 62 + 27 neurons was found.
24

---

Table 4: Quantitative Results of Our Ablation Study and Sensitivity Analysis.
Setting
Memorization Type
Deactivated Neurons
D.88CDoi
8SCDD
Dssco
Acup
All Neurons Activate (Default)
Verhatim
Template
0.83 0.16
0.01 0.04
1.0 0.0
1.0
0.99 4 0.01
0.17 + 0.06
0.31 + 0.02
0.32 4 0.02
Deactivating Random Neurous
Template
Verbatim
21
0.80 0.11
0.05 + 0.04
0.997 + 0.0
0.999
0.16 + 0.06
0.99 0.01
0.31 + 0.02
+ 0.02
Default Values (Tows 0.428)
Verhatim Tomplate
21
0.06 = 0.03
0.06 0.04
0.07
0.12 + 0.05
0.31 : 0.02
Classifier
Verbatim
3+:22
0.09 0.06
0.14 0.07
0.06
0.02
Template
6:1:5
0.05 -
0.12 0.07
0.11 : 0.04
0.32 :: 0.02
Trees
0.288
Template
Verbatim
10.3 + 9.5
32.0
0.08 = 0.06
0.06
0.10 0.07
0.11 0.06
0.15 + 0.05
0.14 + 0.05
0.32 0.02
0.03
0.358
Template
Verhatim
30 25
6.1.5
0.10 0.06
0.14 0.07
0.16 1: 0.05
0.31 1: 0.02
j 2n = 0.498
Verhatim
3++-2
0.10- 0.06
0.15 0.07
0.15 + 0.06
0.32 + 0.02
Template
7.6.6
0.06 0.00
0.12.10.00
0,12 1 0.04
0.31 0.03
No Refinement
Verbatim
26.5 22.5
0.07=00005
0.11 0.06
0.15 0.06
+ 0.02
Template
674.5 624.5
0.04++ 0.03
0.09 + 0.05
0.13 + 0.04
0.31 + 0.02
Verbatim
0.112
No top-k Selection
Template
30 +
0.05 - 0.03
0.41 + 0.32
0.13 + 0.03
0.31 : 0.02
Scaling Factor
Verbatim
4
0.79 0.12
0.996
0.96 0.04
Template
21 2::88
0.05 = 0.001
0.966 0.03
0.15 1 0.05
0.31 : 0.02
Scaling Factor 0.5
Verbatim
11:33
0.62 = 0:29
0.97 = 0.02
0.67 1 0.33
0.32 1 0.01
Template
Scaling Factor 0.25
Verbatim Tomplate
4
0.20 0.17
0.32 0.26
0.21 # 0.12
0.32 + 0.02
21
0.05 +
0.16
0.12 0.04
0.02
Scaling Factor 10.25
Verbatim Template
11:33
0.009 = 0.06
0.12 0.06
0.15 + 0.05
0.32 1 0.02
21 +
0.05 0.03
0.09 0.06
0.13 = 0.04
0.31 = 0.02
Scaling Factor 0.5
Template
21 1:
0.05 =: (1.03
0.08 =: 0.06
0.13 1 0.04
0.31 1: 0.02
Scaling Factor 0.75
Verhatim Template
4+-3
0.08 +
0.06
0.17 0.06
0.31 0.02
21 11:1:0 1:18
0.05 0.03
0.08 0.05
0.14 1: 0.05
0.31 0.02
Scaling Factor -1
Verbatim
4+33
0.08 = 0.05
0.11 0.07
0.16 = 0.06
0.31 0.02
Template
18
0.04
0.05
0.05
0.02
25

---

Initial Selection Threshold
0. 288
228
228
228
-20
Initia Selection Threshole old
0. 0.288
942.5
942.5
942.5
9000
850
-50
0.358
0.358
836.5
836.5
800
100
-750
4.428
0.428
26.5
:0
-700
0.288
0.358
0.428
0.288
0.358
0.428
(a) Number of initial neurons found for VMM
(b) Number of initial neurons found for TM.
0.288
B88
Initial Selection Threshold
Initial Selection Thresheld
10.5
330
0.358
28
0.358
7
26
4.428
5
6
0.428
21
24
0.288
0.358
0.288
0.358
0.428
Refinement Threshold
definteent Threshald
(c) Number of refined neurons found for VM.
(d) Number of refined neurons found for TM.
Figure 14: Number of neurons found with different initial and refinement thresholds Tmem- The left plots show the results for verbatim memorization prompts, while the right plots show the results for template memorization prompts. The refinement step significantly reduces the number of identified neurons across all threshold combinations. Notably, using 0.428 for both the initial selection and refinement thresholds results in the smallest set of identified neurons.
26

---

Initial Selection Threshold
0.10
Selection Thresheld
0.288
0.10
0.08
0.08
0.06
0.358
0.06
0.055
0.04
0.428
0.11
0.058
0.288
0.358
0.428
0.288
0.358
0.428
SSCD Gea of VM samples.
(b) SSCD Gen of TM samples.
0.10
0,288
0.10
Initial Selection Threshold
0.084
0.09
0.09
Selection Threshoid
0.057
0.057
0.055
0.08
0.08
0.358
0.06
3.358
0.06
0.096
0.096
0.055
0.04
0.04
424
0.096
0.02
0.02
0.288
0.358
0.428
0.00
0.288 Refinement Threshold
0.358
0.428
0.00
Refinement Threshold
(c) SSCDoas of VM samples.
(d) SSCDang of TM samples.
Figure 15: SSCD memorization scores with different initial and refinement thresholds Timem The left plots show the results for verbatim memorization prompts, while the right plots show the results for template memorization prompts. The value of the thresholds does not seem to have high impact the memorization scores. Since higher thresholds identify much less memorization neurons, choosing threshold of Tinm 0.428 is valid choice.
27

---

C.8 Ablation of Individual Key and Value Layers
During our experiments in the main paper, we limit our search with NEMo for memorization neurons to cross-attention value layers in the down- and mid-blocks of the U-Net. To motivate this decision, we perform an analysis of the influence of neurons in the individual key and value layers of different cross-attention blocks. Let us first recall the computation performed in attention layers:
QK Vd
Attention Q. K) softmax
V.
(5)
The computed key and query matrices K and Q are used to calculate the attention scores, i.e., the weighting of the components in the value matrix V. In the cross-attention layers, the query matrix Q is computed by linearly mapping the current feature maps from the previous U-Net layer. Therefore, information from the textual guidance is only indirectly contained, i.e., of earlier layers the U-Nets input feature map after the first denoising step. Therefore, we can exclude neurons in the query mapping layers since we aim identify neurons directly responsible for memorization. The neurons in the key mapping layers directly process the text embeddings to compute the attention scores. However, strong interdependencies exist between the activations of different neurons through the nature of the softmax function. The impact each neuron's ' activation has on the computed attention score also depends on the activations of all other neurons from the same layer. Removing a single neuron, i.e., setting its activation to zero, does not necessarily imply substantial changes in the attention scores and the corresponding weighting of features from the value mapping layer.
The value mapping layers, however, also directly process the text embeddings, but there is no direct interdependence between the activations of different neurons. Consequently, setting the activations of individual neurons in value layers to zero directly blocks the information flow from the text embeddings. We hypothesize that the neurons in the value layers are mainly responsible for memorizing the text embeddings of specific prompts.
We evaluate this assumption by taking set of 100 memorized prompts, generating ten samples for each prompt, and comparing the impact of removing neurons from different layers. More specifically, remove all activations of individual key and value mapping layers, i.e., setting the output vectors of these layers to zero while keeping all other parts of the model untouched. We then compare the generated images with removed activations to the original training images. Fig. [6] plots the resulting SSCD similarity scores for deactivating individual value (top row) and key (bottom row) layers. We distinguish between verbatim (left column) and template (right column) prompts. The plots show the maximum and median SSCD scores and the deviations for the median scores. We decided not to plot deviations for the maximum score to improve readability. However, deviations are comparable the median scores. Baselines computed without any deactivated neurons are plotted as dashed lines.
Stable Diffusion contains six cross attention layers in the nine in the up-blocks. The vertical lines indicate the separation between the different blocks. For the value layers, the layers with indexes 1 (down-lock) and 7 (mid-block) have the highest impact, whereas layers later in the network hardly change the SSCD scores. Also, the effect of the remaining layers in the bown-blocks is small on their own. However, we expect there to be entwined effects between deactivating neurons in different layers, which is why we also searched for memorization neurons in these down-block layers.
one in the mid- blo-bbock, and
For the key layers, particularly layers 4 and 6 in the down-blocks have the strongest impact on the generated images. However, removing these layers often produces images that no longer align with the concepts in the prompt or degrades the image quality, both leading to lower SSCD scores. We quantify this behavior by computing the alignments between the generated images and the corresponding input prompts in Fig. 17 While deactivating individual value layers only slightly decreases the alignment scores, deactivating some key layers substantially reduces the alignment. To further illustrate this fact, we plot some of the generated images for deactivating individual value layers in Fig. 18 and for key layers in Fig. 19
28

---

Layer Ablation (Verbatim Memorized)
Layer Ablation (Template Memarized)
10-
Down-Blocks
Up-Blocks
10-
Down-Blocks
Up-Blocks
Similarity (SSCD)
Median SSCD
0.6
0.-
Similnnty
0.2
02-
Median SSCD
Max SSCD
12
14
16
3.0
16
Value Layer
Value Layer
Key Layer Ablation (Verbatim Memorized)
Key Layer Ablation (Template Memorized)
Down-Blocks
Up-Blocks
Down-Blocks
Up-Blocks
0.8
Sim larky (SSCD)
0.8
Max SSCD
Sim
0.-
0.4
Median SSCD
0.0
0.0
Key Layer
Layer
Figure 16: SSCD similarity scores hetween memorized generations and the corresponding training samples. Scores computed for 100 prompts and different seeds per generation. We then take the maximum and median scores of each prompt. During the generation, we deactivated individual value and key layers of the cross-attention blocks in the network. A lower SSCD score indicates a lower similarity between generated and training images. Dashed lines denote the median and the maximum SSCD baselines for images generated without deactivating any neurons. For verbatim memorized prompts, both baselines are close, which is why we only plot the median SSCD baseline.
Key/Value Layer Ablation (Verbatim Memorized)
Key/Value Layer Ablation (Template Memorized)
0.30
Down-Blocks
Up-Blacks
0.3-
Down-Blocks
Up-Blocks
Image-Froon Fromppt Alignment
Image-Prent Proimptrtttttt Emage-PreinptAii Alignment
W
0.30
0.30
W
0.26
0.26
0.24
Value Layers
Value Layers
632
22
24
022
24
16
Key/Value Layer
Key/Value Layer
Figure 17: CLIP alignment scores between memorized generations and the corresponding input prompt. Scores are computed for 100 prompts and ten different seeds per generation. We then take median alignment scores of each prompt. During generation, we deactivated individual value and key layers of the cross-attention blocks in the network. A higher alignment score indicates a better representation of the prompt concepts in the generated images. Dashed lines denote the median alignment scores for images generated without deactivating any neurons. For both types of memorization, deactivating value layers decreases the alignment only slightly, whereas deactivating some key layers substantially reduces the alignment.
29

---

INSICLIRE INSTC
Standaar
Laver
POWIGIS
Prinert
Lure poin
Layer
Lanera
INSTCIPPS
Layora
Lapor
Layer
INSICIRE
Layer
INSICURO INNIICURS
Layer
Loner
Layer 10
INSICIPS
Lawer 11
INSICURS
Lawer 12
Layo
Lapor 84
INSICURGS INSIGIRES
Layer
Layer
Figure 18: Images generated with memorized prompts with deactivated individual value layers. Whereas the standard row shows generations with keeping all neurons active, the following rows depict results for deactivating all neurons specific value layer.
30

---

Standard
INSIGIRES
Layer
Loyer
PORWIERSS
Laners
Layora
INSICIPES
Lapor
Layer
Layer
INSICARSS INSICIRES
Layer
Loner
Layer 10
Lawer 11
INSICIA INSIOIRS
Lawer 12
Layor
Laper 84
INSICURS INSIGIRES
Layer
Layer
Figure 19: Images generated with memorized prompts with deactivated individual key layers. Whereas the standard row shows generations with keeping all neurons active, the following rows depict results for deactivating all neurons specific key layer.
31

---

C.9 Ablation of Individual Convolutional Layers
We also experimented with deactivating neurons in other U-Net layers, including both convolutional and fully connected layers, but we did not find any indicators of memorization in these units. Even when deactivating numerous neurons in the convolutional and fully connected layers of the U-Net, the memorized training images were still faithfully reproduced. However, the quality of the images degraded, particularly when deactivating neurons in early layers, which are responsible for defining the image structure. We showcase in Fig. 20 various examples of memorized images generated with 50% deactivated neurons in the convolutional layers to illustrate that these neurons have no noticeable impact on the memorization behavior. However, deactivating too many neurons in early layers can negatively affect the overall generation process. So, to conclude, deactivating other neurons in the U-Net didn't 't seem to impact memorization, which is why our choice to focus on the value layers seems reasonable.
Convolutional Layer
Boltt
Capo
Seed
Figure 20: For each convolutional layer in the U-Net's down blocks, randomly deactivated 50% of the neurons. The results demonstrate that blocking neurons in the convolutional layers does not mitigate memorization. Instead, deactivating the neurons reduces the quality of the generated images and, in some cases, causes the entire generation process to collapse, especially when neurons the early layers are deactivated.
32

---

D Algorithmic Description of NeMo
D.1 Computing Noise Differences
Alg. deefines our algorithm to compute the differences between the initial noise samples and the noise predicted during the first denoising step. The resulting noise differences are used to compute our SSIM-based memorization score during the initial neuron selection and the refinement step. We compute the noise differences always for 10 different seeds avoid undesired biases due to the random sampling process. We further remove noise differences from the set, which have low similarity with other noise differences. By this step, we remove noise differences for seeds that do not lead to memorization and might mislead the algorithm.
Algorithm Compute Noise Differences
Input:
Prompt embedding y
b Text prompt (embedding) > Set of neurons to deactivate Diffusion model
Neuron set Seeurons
Noise predictor Co
Memorization threshold (SSIM) Tiem
b. Target memorization score
Output: Noise differences A
Set A as empty list Eg < deactivate_nne setiiivaate_eeroonneee,ee SNeurons)
Initialize list of noise differences Set activations of neurons in Securons zero
Compute noise differences for each random seed
for i = 1..... 10 do
Iterate over 10 seeds D Set random seed
set (i)
sample JT (N,
D. Randomly initialize noise image
JT - E T,y)
Compute noise prediction Compute noise difference
5 < TT-1 FT 8 m-))n(8)
8
Normalize differences by min-max scaling Add current noise difference to list
max (8) min
append to
end for
// Remove noise differences not leading to memorization for
E A do
Iterate over noise differences
Â - A A/6
Get set of noise differences without 5 D Compute pairwise memorization scores (SSIM)
4- computt__memorzzaioon/ A)
if max(d)
Tiem then
b Highest memorization score is below threshold D Remove noise difference from set
<
end if end for return A
> Return list noise differences
33

---

D.2 Detecting Neurons with Out-of-Distribution Activations
Alg. describes our method detect neurons with out-of-distribution (OOD) activations. Our method detects OOD neurons based on their activation distance for memorized prompt to a neuron' 's mean activation computed on hold-out dataset of non- con-meeorized prompts. In addition, we also add the neurons with the highest absolute activations within each layer to the set.
Algorithm 2 Get OOD Neurons
Input:
Prompt embedding y/ Activation threshold Pact
D Text prompt (embedding)
b Threshold for the OOD detection Value of top-k detection Activation statistics hold-out dataset
Top k
Activation mean and standard deviation o
Output:
Set neurons with OOD activations Smitial
activvttoos
+ collect_adde collect_activationn(())
D Collect activations on prompt Initialize empty neuron
Sinitial
// Check each neuron in each layer for OOD activation for 1 E {1..... ]} do
Iterate over all layers Iterate over all neurons in layer / Compute z-score for current neuron
for [l...., N do
(v) -x(x)- oi
if 21(x) > Oact then
Activation above OOD threshold D Add OOD neuron to set
Sinitial & Smitial U [neuron!)
end if end for
I/ Add k neurons of layer 1 with the highest absolute activations the candidate Stop + top_k_actee sop___activationss Sactivations: 1, k) D Get neurons with highest absolute activations
Siniiial Sinitil Spop
>> Add top-k neurons set
end for
return Smitial
D. Return set with OOD neurons
34

---

D.3
Selecting Initial Candidates of Memorization Neurons
defines our algorithm compute the initial set of memorization neurons. The resulting initial set of selected memorization neurons is then refined in second step, shown in Alg. 4
Alg.
Algorithm 3 Initial Neuron Selection
Input:
Prompt embedding y Memorization threshold (SSIM) Tiem Minimum activation threshold Omin
Text prompt embedding
D Target memorization score
Threshold for stopping neuron search
Output: Set of neuron candidates Siitial, refinement memorization threshold Tmem_ref
Candidate set of memorization neurons Sinitil
Initial memorizing neuron set Memorization threshold for refinement
Memorization threshold (SSIM) memm_ref
mem f- 1.0
D Initialize memorization score as maximum Initialize threshold of OOD activation detection D Initialize k for top-k activation detection
Onct 5
+-0 0
Tiem_ret ref f- Tiem
Set refinement memorization threshold current threshold > Noise differences with all neurons active
Aunblocked f get_ noise _oise_diff(y,.._..
// Increase set of candidate neurons until target memorization score is reached while mem > Tmem do
While memorization score above threshold Detect neurons with OOD activations Compute noise differences unbloocedd Abookkad) Compute memorization score (SSIM)
Simuall f- get _od foood__errossy__. Oua, k))
Ablocked -- get_noise diff(yy Sintiaa)
mem +
if Out < Omin then
P Minimum activation threshold not reached refinement threshold to current memorization score Stop if activation threshold is too low
Tmem_ret ref f- mem
break
end
// Adjust OOD detection parameters to increase set of candidate neurons
Puc
0xc 0.25
Decrease threshold for OOD detection
3 k
b Increase l for top-k activation detection
end while
return
Smitial, Tmem_red ref
Return neuron candidates and refinement memorization threshold
35

---

D.4
Neuron Selection Refinement
defines our algorithm refine the set of candidate neurons identified from NEMo's initial selection step.
Alg.
Algorithm 4 Neuron Selection Refinement
Input:
Initial memorization neuron candidate set Sinitial
Given neuron candidate set Refinement memorization score threhsold
Memorization threshold (SSIM) Tmem_ref
Output:
memorization neurons Syetined
> Refined set memorization neurons
Seetned f Smitiil
uubblockdd get_ noise _oise_diff((y,, 0)
Noise differences with all neurons active
I/ Check all candidate neurons of individual layers at once for memorization
for 1 {1...., L} do
Iterate over all layers to remove low impact layers Get the neurons in the current layer / Compute set of neurons from remaining layers Compute noise differences Compute memorization score (SSIM)
Slnyer get neurons in in_eayerrSSa (Srefied:, 1)
Snewrons + Seeined Slayer
Ablocked + gei_ bosss_ddff(yy-a Sneurons)
mem + compute memorization Ambblocked Dlocked)
if mem < Tmem_re then
Minimum memorization threshold not reached Remove neurons of layer from neuron set
Syelined < Srefined Shayer
end if end for
Check all remaining candidate neurons individually for 1 (1....,L) L.} do
Iterate over each remaining layer D Get neurons current layer /
Suyer t get_neurons_ing lrcpoonssii__gg_e iinninnee(SSeeiiiie 1)
for n E Suaye do Sncurous f- Syetined {n}
Compute set of neurons without neuron > Compute noise differences Compute mem. score (SSIM)
A blocked < pet__noise___ diff(y. Speurons)
commpute_memorizatione unhlacked, Ablocked)
if mem Tmem ref then
Minimum memorization threshold not reached b- Remove current neuron from set
Stefined &- Stefined {n}
end if
end for
end for
return Stetined
Return refined of memorization neurons
36

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims and promises made in the abstract and introduction are accurately reflected in the paper by clear formal description (Section 3 and extensive experimental evaluation (Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appx. A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
37

---

Justification:
The paper does not include any theoretical proof but offers practical method (Section 3 detect and localize memorization in diffusion models.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided the core of the paper should complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe all settings, models, datasets, and hyperparameters used during our experiments in Section M and in Appx. tions for all steps of our method. Therefore, we expect reproduction be possible regardless of whether the code is provided.
provides detailed algorithmic descrip-
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e-g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
model (e.g., a large language model), then there should either
(d) We recognize that reproducibility may be tricky in some cases. in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
38

---

Answer: [Yes]
Justification: The code is publicly available, including all datasets required for reproduction. The README file within the code base explains each step, including the dataset, models, localization algorithm, and the computation of metrics, in detail.
Guidelines:
The answer NA means that paper does include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpps:/nips.cc/ pulllllicc.ccccuulllc/cc
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https: /nwpssip..... httppippppppppppeepeeepppppppppp cc/public/ guides/Code ssioonolccyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We stated all experimental details in Section and provide additional details in Appx. B We also state all used hyperparameters within our code base.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide results computed on a sufficiently large dataset, including the variability for all experimental results in the paper (Section 4. All experiments were performed with ten seeds to provide statistically stable results. We also provide an ablation and sensitivity analysis to add further insights into the robustness and stability of our method and the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
39

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification: We state all hardware and software details in Appx. B.1 We also provide our complete source code and docker image for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethies
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics httpps://eeurips... http:p/p...ccpplcccc///ssusuaaas
Question:
Answer: [Yes]
Justification: Our research identifies neurons in text-to-image systems that are responsible for memorizing training samples. We strongly believe that our research has no negative societal impact or potentially harmful consequences. In stark contrast, our research tackles the memorization problem in diffusion models and might support the development of models and methods to mitigate such undesired memorization.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., ., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss our method's broader impact and advantages in localizing neurons responsible for memorization in diffusion models (Section 5). Localizing such memorization
neurons is effective way of mitigating memorization. We do not expect any potential malicious
or unintended uses of our method.
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address socictal impact.
40

---

Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not develop or release any kind of new model dataset. Instead, we only investigate existing, publicly available models and datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We explicitly state the origin of data and models in the paper and properly credit the source of these assets (Appx. B..2). The authors wrote the code for conducting the experiments by themselves. Any code snippets taken from external sources are explicitly marked.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwpppsaipphheeeecc thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
41

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper releases no new assets. All models and datasets used our experiments publicly available (Appx.B.2 B.2).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codc/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: We did not conduct any kind of crowdsourcing experiments or research with human subjects at any point.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contribution of the paper involves human subjects, then as much detail possible should be included in the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not conduct any kind of crowdsourcing experiments research with human subjects at any point.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
42

---

For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
43


Reviews:
[{"review_id": "6JC0zIETNs", "reviewer": "Reviewer_Rr34", "summary": "This paper introduces NEMO, a method for localizing memorization in diffusion models (DMs) down to the level of individual neurons. The authors empirically evaluate NEMO on the Stable Diffusion model and demonstrate that deactivating the identified memorization neurons effectively mitigates memorization, increases image diversity, and maintains prompt alignment. The paper makes valuable contributions by providing insights into the memorization mechanisms in DMs and proposing a method for permanently mitigating memorization without compromising overall model performance or the quality of the generated images.", "questions": "Can the authors provide more details on the selection of the memorization threshold $τ_mem$? How was it determined based on the pairwise SSIM scores? Did the authors consider any alternative methods for setting the threshold?\nHave the authors considered comparing NEMO with other existing methods for mitigating memorization in DMs, such as rescaling attention logits or adjusting prompt embeddings? How does NEMO perform in terms of effectiveness and efficiency compared to these methods?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper presents an innovative and practical methodology, NEMO, for localizing memorization in DMs. This methodology allows model providers to identify and deactivate specific neurons responsible for memorizing training samples, effectively mitigating memorization.\nThe empirical findings presented in the paper provide valuable insights into the distribution of memorization neurons and the impact of deactivating these neurons on image generation. The experiments demonstrate that deactivating memorization neurons increases image diversity.\nThe paper is well-structured and provides a clear explanation of the proposed methodology, including the initial selection and refinement steps of NEMO. The authors also provide algorithmic descriptions for each component of NEMO in the appendix and supplement it with extensive experimental results.", "weaknesses": "The paper lacks a thorough evaluation of the proposed method's performance compared to existing baselines (if exist) or state-of-the-art methods for mitigating memorization in DMs. It would be beneficial to include comparisons with other methods to assess the novelty and effectiveness of NEMO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison to other mitigation methods?**\n    \n> We have already quantitatively compared our method with the state-of-the-art mitigation approach by Wen et al. (ICLR 2024) in Section 4.1 and Table 1. Additionally, in Appendix C.9, we compared our method to the random token mitigation strategy proposed by Somepalli et al. (NeurIPS 2023), which achieved the best results among the mitigation methods they investigated. For the rebuttal, we also repeated the experiments using the attention scaling proposed by Ren et al. (Preprint 2024). We provide the additional results in Table 1 in the rebuttal PDF. Overall, our method not only performs comparably to Wen et al.'s approach in terms of mitigation and image-text alignment but also offers the significant advantage of precise memorization localization. This capability enables us to permanently eliminate memorization from the model after a single localization, thereby avoiding the need for additional repeated mitigation strategies during inference. Our approach is more cost-effective and computationally efficient for known memorized prompts than existing mitigation methods.\n\n> Conceptually, our method does not alter the input embeddings or attention score distribution; instead, it deactivates individual neurons in the U-Net. This approach eliminates the need for gradient computation, which can be memory-intensive. Notably, previous mitigation methods require adjustments and computations for each new generation. In contrast, once we have localized the memorizing neurons using our method, we can prune these neurons, eliminating the need for additional computations during future inference.\n\n**More details on the selection of the memorization threshold $\\tau_{mem}$?**\n\n> We computed the threshold on a hold-out dataset of non-memorized prompts using 50,000 LAION prompts. By choosing a dataset of non-memorized prompts, we aimed to prevent overfitting our evaluation dataset of memorized prompts and thereby avoid biasing the results. We calculated the pairwise SSIM, our memorization score, between ten different initial noise samples for each of the 50,000 non-memorized LAION prompts and took the maximum score. We then averaged these memorization scores across all samples and added one standard deviation to set $\\tau_{mem}$. This procedure is detailed in Section 3.1 of the paper. Since the SSIM score is only computed after the first denoising step, this threshold computation is computationally efficient.\n    \n**How was $\\tau_{mem}$ determined based on the pairwise SSIM scores?**\n\n> The lower we set the threshold, the more neurons will be deactivated since our algorithm aims to reduce the memorization score of the current sample (see Alg. 3 in Appx. D.3). We repeated our experiments with different choices of the threshold in Appx. C.7 to analyze the impact of different thresholds. A lower threshold identifies more neurons. However, for lower thresholds, the metrics are comparable to those obtained with our default threshold value in the main paper. Increasing the threshold reduces the number of identified neurons but slightly increases memorization, as measured by the SSCD scores. Thus, a trade-off exists between reducing the number of identified memorization neurons and their memorization mitigation effect. We also depict the impact of different threshold choices in Fig. 14 and Fig. 15 in Appx. C.7. \n    \n**Alternative methods for setting the threshold $\\tau_{mem}$**\n    \n> We did not test other methods for setting the threshold. We believe that incorporating an additional hold-out set of memorized prompts might help users find even better thresholds. However, we focused on a practical scenario where the user only has access to a set of non-memorized prompts and demonstrated that this selection process is sufficient to localize and mitigate memorization in the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noDSJjf6Cg", "reviewer": "Reviewer_zaL8", "summary": "The paper localize memorized samples to neurons in the cross attention layers in diffusion models. By deactivating neurons responsible for memorization, the proposed method enables models to generate diverse images different from training images.", "questions": "1. How are tokens handled while calculating neuron activation in Equation 3? Do you consider an average or max over tokens?\n2. How faithful is SSIM for detecting memorized prompts? Can it be used to detect if a prompt is memorized or not?", "limitations": "The authors have clearly addressed the limitations of the proposed method in the paper. I believe some more limitations are addressed in the Weaknesses section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors propose the first method to localize memorization within the model, which is a novel idea that has not been studied in the context of text-to-image generation models before. \n2. The experiments and findings presented in the paper are interesting.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. I understand that detecting memorized propmts is out of scope of this paper. Previous methods in mitigating memorization construct an entire pipeline in which memorized prompts are first detected with a heuristic/metric, and a mitigation strategy is triggered during inference/training if the metric surpasses a certain threshold. My expectation from \"localization of memorization\" was that there is a specific subset of neurons that, when blocked, avoid replication of training data. Such a localization would not require the authors to develop a pipeline to detect memorization as those neurons can be pruned. \n\n(a) From the paper (Table 1 and Figure 4) it seems that the proposed method operates on a per-prompt basis and all experiments have been conducted by separating the memorized prompts into template and verbatim.\n\n(b) I believe that although these findings are interesting, one weakness is that the method is impractical as in practice, it would still require one to know the type of memorization before deciding which neurons to block. For some highly memorized prompts, the highly-iterative neuron discovery process (Alg D.1 to D.4) needs to be repeated. This not only requires interference with the inference pipeline but is also computationally inefficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**My expectation from \"localization of memorization\" was that there is a specific subset of neurons that, when blocked, avoid replication of training data. Such a localization would not require the authors to develop a pipeline to detect memorization, as those neurons can be pruned.**\n\n> It would indeed be very helpful to identify a fixed set of neurons responsible for all types of memorization in a model beforehand, without the need to first identify memorized prompts individually. However, as our analyses demonstrate, such a fixed set of memorization neurons does not exist in diffusion models. Instead, we identified various neurons responsible for the memorization of different prompts. While some sets of neurons memorize multiple prompts, no fixed set of neurons is responsible for memorizing all prompts. This is not a weakness of our method, but rather a reflection of the behavior of already trained models.\n    \n**It seems that the proposed method operates on a per-prompt basis, and all experiments have been conducted by separating the memorized prompts into template and verbatim.**\n    \n> There appears to be a misunderstanding here. We only split the results into template and verbatim memorization to provide better insights into the effects of different types of memorization. Our algorithm and all hyperparameters are fully agnostic to the type of memorization. We applied our method to all memorized prompts from our test set in the exact same way, without setting any memorization- or prompt-specific hyperparameters. We used the same values and thresholds for all memorized prompts. The distinction between memorization types is only made when presenting our results to support our analyses and offer additional insights. We will gladly clarify this point in our final version.\n    \n**The method is impractical, as it requires one to know the type of memorization before deciding which neurons to block.**\n    \n> As discussed in the previous answer, there seems to be a misunderstanding here. Our method is fully agnostic to the type of memorization, so in practice, one does not need to know the type of memorization to apply it. Furthermore, as detailed in our ablation study and sensitivity analysis in Appendix C.7, the user does not even need to know if a prompt is memorized at all. In the sensitivity analysis, we applied our method to non-memorized prompts without any adjustments and found that for the majority of these prompts, our method detected no memorization neurons. This is expected since these prompts show no memorization behavior. For the remaining non-memorized prompts, a median of 62 neurons was identified. Deactivating these neurons should not impact the image quality, as our analysis in Section 4.4 and Fig. 5a) demonstrates.\n    \n**For some highly memorized prompts, the neuron discovery process needs to be repeated. This not only requires interference with the inference pipeline but is also computationally inefficient.**\n\n> We emphasize that our detection method only needs to be run once for each input prompt. While the runtime varies based on the strength of the memorization, for verbatim memorized prompts, which represent the strongest type of memorization, the runtime was only a few seconds (as discussed in our limitations section in Appendix A). Once we have identified the memorizing neurons, we can permanently deactivate them. This allows future inference steps to generate images without the need to repeat the localization process for the memorized prompts or interfere with the inference pipeline at any point. In contrast, related inference-time mitigations need to be applied for each new generation, increasing the total runtime and memory consumption of the generation process. Therefore, we kindly disagree that our method is computationally inefficient.\n    \n    \n**How are tokens handled while calculating neuron activation in Equation 3? Do you consider an average or max over tokens?**\n    \n> We did not interact with or adjust any tokens of the input prompts. The model's text encoder first processes the prompt—the CLIP text encoder in the case of Stable Diffusion—and transforms it into embedding vectors of a fixed size ($77\\times 768$), independent of the prompt length (by padding the input tokens). In Equation 3, we only consider the computed embedding vectors as input to the i-th neuron in the l-th layer, denoted by $a^l_i(y)$. To quantify neuron activation, we take the mean of the absolute activations across all 77 activations in the embedding vectors. We will clarify this computation in our final version.\n    \n**How faithful is SSIM for detecting memorized prompts? Can it be used to detect if a prompt is memorized or not?**\n    \n> Indeed, the SSIM score itself can be used to detect memorization. We visualized the different SSIM score distribution for non-memorized and memorized prompts in Fig. 2a) in the main paper. To underline this observation with quantitative metrics, we ran additional experiments to explore the detection capabilities of our SSIM-based method. We measured the efficiency of the SSIM score for memorization detection on our dataset of memorized and non-memorized prompts. Without extensive hyperparameter tuning, this detection method achieves an AUROC of 98\\%, an accuracy of 94.2\\% (using a naive threshold of 0.5 on the SSIM similarity), and a TPR@1\\%FPR of 87.6\\%. Since the amount of memorization of template memorized prompts varies significantly, we repeated the computation for detecting the verbatim memorized prompts. Here, the SSIM approach even achieves an AUROC of $99.64\\%$, an accuracy of $97.39\\%$ (with a threshold of 0.5), and a TPR@1\\%FPR of 98.21\\%. These results indicate that the SSIM score can also be used to detect memorization reliably in the first place. However, detection is not the focus of the paper but the localization on the neuron level.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7m0nJQQJwa", "reviewer": "Reviewer_K7EP", "summary": "This paper introduces NEMO, a method that identifies and deactivates specific neurons responsible for memorizing training data in DMs. This approach prevents the reproduction of sensitive images, enhances output diversity, and mitigates data leakage, enabling more responsible use of DMs.", "questions": "The author should address weakness 1 and 3.", "limitations": "The limitations that the author mentioned in Appd A is about how to further improve the method. It is not clear solved but propose the key factors is possible to solve in the future work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It explains the connection between neurons and memorization, which provides more understanding in this problem.\n\n2. This paper provides two selection stages to select the different neurons.\n\n3. The paper writing is clear.", "weaknesses": "1. The author should specify their novelty compared with [1]. The novelty seems limited considering the conclusion in [1].\n\n2. No practical mitigation method is provided. Deactivating is not possible without knowing the neurons.\n\n3. The author claim that most memorization happens in the value mappings of the cross-attention layers of DMs. However, actually, the author only prove that some neurons in attention layers are necessary in memorization. They didn't show that those neurons are everything about the memorization. It is possible that deactivating other neurons that are not in attention layers can also remove memorization. Localizing should consider how to exclude other neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The author should specify their novelty compared with [1]. The novelty seems limited considering the conclusion in [1].**\n    \n> We believe there is a reference missing here in the review. Could you please specify which paper we should compare our method to? Reference [1] in our paper does not seem to correspond to the mentioned work. Additionally, we emphasize that we already compare our method to the current state-of-the-art mitigation method by Wen et al. (ICLR 2024) in Section 4.1 and Table 1, as well as to the most effective mitigation strategy proposed by Somepalli et al. (NeurIPS 2023) in Appendix C.9 and Table 3. We also added additional results for the mitigation proposed by Ren et al. (Preprint 2024) in Table 2 in the attached rebuttal PDF. Please let us know if this already addresses your concern or to which paper citation [1] refers.\n    \n**No practical mitigation method is provided. Deactivating is not possible without knowing the neurons.**\n    \n> There seems to be a misconception here. Our method is explicitly designed to identify and localize the neurons responsible for memorizing training data. For each memorized prompt, we identify a set of neurons that can then be deactivated to mitigate memorization. Indeed, deactivating neurons is not feasible without first identifying them, which is why our method focuses on this identification process. Therefore, we respectfully disagree with the notion that our method is impractical. We are happy to address any further questions or clarifications that may arise.\n\n**The authors prove that some neurons in the attention block's value layers are necessary for memorization. What about other layers?** \n    \n> We kindly refer to our analysis in Appx. C.8, where we analyzed the memorization behavior of other neurons in the network. In our paper, we focused on neurons that directly process the text embeddings and are responsible for guiding the image generation process. Focusing on these layers is reasonable since memorization directly depends on the concrete input prompt, as we have shown in our analysis.\n\n> In Stable Diffusion, the key and value layers process the text embeddings of the prompt. Therefore, we conducted experiments showing that the value layers are responsible for memorization in diffusion models. The neurons in the key mapping layers directly process the text embeddings to compute the attention scores. However, strong interdependencies exist between the activations of different neurons through the nature of the softmax function. The impact each neuron’s activation has on the computed attention score also depends on the activations of all other neurons from the same layer. Removing a single neuron, i.e., setting its activation to zero, does not necessarily imply substantial changes in the attention scores and the corresponding weighting of features from the value mapping layer. The value mapping layers, however, also directly process the text embeddings, but there is no direct interdependence between the activations of different neurons.\n    \n> Consequently, setting the activations of individual neurons in the value layers to zero directly blocks the information flow from the text embeddings. Therefore, it is reasonable to focus on the neurons in the value layers to mitigate memorization. We also experimented with deactivating neurons in other U-Net layers, including both convolutional and fully connected layers, but we did not find any indicators of memorization in these units. Even when deactivating numerous neurons in the convolutional and fully connected layers of the U-Net, the memorized training images were still faithfully reproduced. However, the quality of the images degraded, particularly when deactivating neurons in early layers, which are responsible for defining the image structure. We showcase in the rebuttal PDF various examples of memorized images generated with 50\\% deactivated neurons in the convolutional layers to illustrate that these neurons have no noticeable impact on the memorization behavior. However, deactivating too many neurons in early layers can negatively affect the overall generation process. So to conclude, deactivating other neurons in the U-Net didn't seem to have an impact on memorization which is why our choice to focus on the value layers seems to be a reasonable choice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AMGskVCBUp", "reviewer": "Reviewer_RyXV", "summary": "This paper introduces NeMo, a straightforward and easy-to-understand method. NeMo identifies neurons responsible for memorization and deactivates them, offering potential applications in the protection of sensitive or copyrighted data.", "questions": "1, I have some reservations about the connection between copyright/sensitive images and memorization. It seems that copyright issues might only be partially triggered by memorization. Could you provide further clarification on this point? This pertains to the application of your method.\n\n2, I find the activation-based localization method reasonable. However, I'm curious if pruning methods such as [1] or other neuron localization methods such as [2] could also be effective here. I would appreciate some analysis on this, though I do not require any additional experiments.\n\n3, In section 4.3, could you explain why deactivating neurons further reduces FID and KID in some cases?\n\n4, I'm also interested in the potential side effects of this method. You mention that deactivation has a minimal impact on non-memorized generation, which suggests that these neurons do not interact significantly with other neurons and function independently. I believe a more detailed analysis from this perspective would be beneficial.\n\n\n[1] Structural Pruning for Diffusion Models\n[2] Cones: Concept Neurons in Diffusion Models for Customized Generation.", "limitations": "The limitations have been discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This topic is highly interesting as it enhances interpretability and offers a lightweight solution for addressing issues related to sensitive or copyrighted training images.", "weaknesses": "1, There is insufficient comparison with other methods, such as the work mentioned in [1]. The authors state, \"While mitigations that rely on preventing the generation of memorized samples are effective when the DM is developed and deployed in a secure environment, they hold the inherent risk of adversaries circumventing them. Additionally, they are not effective when the DMs are publicly released, such that users can freely interact with them.\" However, this argument is not persuasive enough. Providing more compelling reasons for the unique position of this method in mitigating memorization would enhance my evaluation score.\n\n2, The side effects of this method are not thoroughly investigated.\n\n\n[1] Detecting, Explaining, and Mitigating Memorization in Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Providing more compelling reasons for the unique position of this method in mitigating memorization.**\n> The novelty and positioning of our research is multi-faceted: NeMo is the first method to localize memorization in diffusion models down to individual neurons. Previous research has generally focused on detecting memorized prompts in general. Still, no method has yet offered a way to identify specific parts of the model responsible for this memorization behavior.\n\n> By identifying memorizing neurons, one can prune these neurons, permanently removing them from the model. This enables model providers to eliminate undesired memorization before releasing the model to the public, preventing users from simply deactivating the mitigation technique. Existing mitigation methods rely on embedding or attention adjustments at inference time, which users can easily deactivate, thus failing to offer a permanent solution. \n    \n> Unlike traditional model pruning literature, we aim not to minimize the total number of model parameters without performance loss but to identify those parameters responsible for memorizing particular training samples.\n\n**Comparison to research on pruning and concept neurons for diffusion models.**\n> Structural pruning for diffusion models [1] aims to reduce the number of model parameters by removing sub-structures that can be eliminated without hurting the model's performance. The goal of structural pruning is orthogonal to ours. While structural pruning focuses on reducing the parameter count, it does not address sample memorization. Its optimization criterion is based on the standard reconstruction loss used during the training of diffusion models. Although one might adapt this approach to mitigate memorization, such adjustments would require additional data and resource-intensive training. In contrast, our method identifies memorization neurons using a gradient-free approach and removes only these neurons to mitigate memorization. It requires no additional training or data, acting solely on input prompts.\n    \n> Concept neurons [2] is a gradient-based method designed to identify neurons associated with specific image concepts for customized image generation. Their method requires several images of the target concept with sufficient variations. Concept neurons and our approach differ significantly in their objectives and methodologies. Concept neurons focus on identifying neurons associated with a concept across multiple images, making this method unsuitable for localizing memorization. Furthermore, their approach necessitates fine-tuning the diffusion model on concept images before identifying neurons based on gradient information. In contrast, our method identifies memorization neurons responsible for individual prompts.\n    \n> We thank the reviewer for pointing out the potential connection between our work and these two papers, which offer interesting avenues for future research into memorization. We will include a discussion on the relation of our approach to the pruning literature in our paper.\n        \n**Clarifying the connection between copyright/sensitive images and memorization?**\n    \n> We are happy to elaborate further on this point. Memorization, particularly verbatim memorization, which reconstructs training samples perfectly, can pose significant copyright issues. For instance, the generated images in Figure 3, which (without mitigation) reproduce training samples exactly, may include content that is not license-free. Users might not realize that their generated images closely resemble copyrighted material, leading to legal issues due to copyright violations.\n    \n> Moreover, some memorized images may correspond to training data that users have unintentionally made publicly accessible. While the content described by the corresponding prompt may not be inherently sensitive, the exact replication of the training sample is. In such cases, removing the identified memorizing neurons provides a practical solution to these issues without requiring time-consuming and computationally expensive re-training or fine-tuning.\n    \n    \n**Could you explain why deactivating neurons further reduces FID and KID in some cases?**\n    \n> We hypothesize there are two reasons for the reduction in FID/KID. First, deactivating the memorization neurons increases the variation in generated images with (partly) memorized contents. This effect might also (slightly) improve the overall image quality of non-memorized prompts. Second, there is some noise when computing the quality metrics, as indicated by the standard deviations, which might also explain the improvements. Interestingly, the Structural pruning for diffusion models [1] paper also reports improvements in FID score when pruning a CelebA-HQ model (Table 1 in their paper). Further analyzing the connection between pruning and image quality promises an exciting avenue for future research. The important take-away message for our work is that removing memorization neurons does not hurt the model's general performance.\n    \n**More detailed analysis on side effects and neuron functionality.**\n> We ran additional experiments to analyze the impact and effects of deactivating neurons in the value layers. We deactivated a varying number of neurons, either randomly selected or the top memorizing neurons, and measured the similarity between the outputs of the cross-attention blocks with deactivated neurons and all neurons activated. Our results, stated in the rebuttal PDF, show that even deactivating a large number of neurons only impacts the attention outputs marginally. We, therefore, conclude that other neurons can replace the functionality of specific neurons on non-memorized prompts and that neurons in the value layers act rather independently. Yet, for neurons memorizing specific prompts, this memorizing functionality is not replaced by other neurons, explaining the mitigation effect of deactivating these neurons.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time spent reviewing our paper and their valuable feedback. We very much appreciate all the reviewers tend to accept the paper and think that the paper proposes an innovative and novel approach (Rr34, zaL8), presents interesting findings (zaL8, RyXV), provides a better understanding of the problem of memorization (K7EP), and is clearly written (Rr34, zaL8, K7EP). We address the individual questions and remarks below and are happy to provide further details if additional questions arise.

The attached PDF contains additional quantitative results for the attention scaling mitigation method proposed by Ren et al. (2024, Preprint) (Rr34), as well as images illustrating the impact of deactivating neurons in convolutional layers (K7EP) and the disentanglement of neuron activations in the value layers (RyXV).

We also emphasize that we have already provided a comprehensive comparison of our method to existing mitigation strategies in Table 1 of our main paper, along with additional results in Table 3 in Appendix C.3. An ablation study on the roles of individual key and value layers in memorization is presented in Appendix C.8. Furthermore, we explored the impact of different hyperparameters in our ablation and sensitivity analysis in Appendix C.7.

Meta-review: This paper proposed a simple method to localize memorization to individual neurons and showed that deactivating those neurons can have multiple benefits such as reduced memorization and increased diversity. The reviewers generally found this work interesting and novel. This would could have potential applications for future text-to-image generation model and system designs.
Decision: Accept (poster)