Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Models

Authors: dominik  hintersdorf
lukas  struppek
kristian  kersting
adam  dziedzic
franziska  boenisch
ann  graham lotz
memorization  neurons
neuron # 124 neuron
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 15:25:07

Abstract: Diffusion models (DMs) produce very detailed and high-quality images. Their power results from extensive training on large amounts of data - usually scraped from the internet without proper attribution or consent from content creators.  Unfortunately, this practice raises privacy and intellectual property concerns, as DMs can memorize and later reproduce their potentially sensitive or copyrighted training images at inference time. Prior efforts prevent this issue by either changing the input to the diffusion process, thereby preventing the DM from generating memorized samples during inference, or removing the memorized data from training altogether. While those are viable solutions when the DM is developed and deployed in a secure and constantly monitored environment, they hold the risk of adversaries circumventing the safeguards and are not effective when the DM itself is publicly released. To solve the problem, we introduce NeMo, the first method to localize memorization of individual data samples down to the level of neurons in DMs' cross-attention layers. Through our experiments, we make the intriguing finding that in many cases, single neurons are responsible for memorizing particular training samples. By deactivating these memorization neurons, we can avoid the replication of training data at inference time, increase the diversity in the generated outputs, and mitigate the leakage of private and copyrighted data. In this way, our NeMo contributes to a more responsible deployment of DMs.

Main paper: 
 introduction: 
 In recent years, diffusion models (DMs) have made remarkable advances in image generation. In particular, text-to-image DMs, such as Stable Diffusion [32] , DALL-E [30] , or Deep Floyd [38] enable the generation of complex images given a textual input prompt. Yet, DMs carry a significant risk to privacy and intellectual property, as the models have been shown to generate verbatim copies of their potentially sensitive or copyrighted training data at inference time [8, 36] . This ability has often been linked to their memorization of training data [48, 12, 1, 5] . Memorization in DMs recently received a lot of attention [16, 47, 49, 8] , and several mitigations have been proposed [46, 31, 36] . Those mitigations usually focus on either identifying potentially highly memorized samples and excluding them from training, monitoring inference and preventing their generation, or altering the inputs to prevent the verbatim output of training data [31, 36, 46] . While mitigations that rely on preventing the generation of memorized samples are effective when the DM is developed and deployed in a secure Figure 1 : Overview of NeMo. For memorized prompts, we observe that the same (original training) image is constantly generated independently of the initial random seed. This yields severe privacy and copyright concerns. In the initial stage, NEMO first identifies candidate neurons potentially responsible for the memorization based on out-of-distribution activations. In a refinement step, NEMO detects the memorization neurons from the candidate set by leveraging the noise similarities during the first denoising step. Deactivating memorization neurons prevents unintended memorization behavior and induces diversity in the generated images. environment, they hold the inherent risk of adversaries circumventing them. Additionally, they are not effective when the DMs are publicly released, such that users can freely interact with them.As a first step to solving this problem, we propose FINDING NEURON MEMORIZATION (NEMO), a new method for localizing where individual data samples are memorized inside the DMs. NEMO's localization tracks the memorization of training data samples down to the level of individual neurons in the DMs' cross-attention layers. To achieve this, NEMO relies on analyzing the different activation patterns of individual neurons on memorized and non-memorized data samples and identifying memorization neurons by outlier activation detection (as visualized in Fig. 2b ). We empirically assess the success of NEMO on the publicly available DM Stable Diffusion [32] . Our findings indicate that most memorization happens in the value mappings of the cross-attention layers of DMs. Furthermore, they highlight that most training data samples are memorized by just a few or even a single neuron, which is surprising given the high resolution and complexity of the training data.Based on the insights about where within the DMs individual data samples are memorized, we can prevent their verbatim output by deactivating the identified memorization neuron(s). We demonstrate the effect of NEMO in Fig. 1 . Without our approach, the image generated for the memorized input prompt is the same, independent of the random seed for generation. By localizing the neuron responsible for the memorization through NEMO and deactivating it, we prevent the verbatim output of the training data and instead cause the generation of various non-memorized related samples. Hence, by relying on NEMO to localize and deactivate memorization neurons, we can limit memorization, which mitigates the privacy and copyright issues while keeping the overall performance intact.In summary, we make the following contributions:• We propose NEMO, the first method to localize where memorization happens within DMs down to the level of individual neurons.• Our extensive empirical evaluation of localizing memorization within Stable Diffusion reveals that few or even single neurons are responsible for the memorization.• We limit the memorization in DMs by deactivating the highly memorizing neurons and further show that this leads to a higher diversity in the generated outputs.2 Background and Related Work 
text-to-image synthesis with diffusion models: 
 Diffusion models (DMs) [37, 19] are generative models trained by progressively adding random Gaussian noise to training images and having the model learn to predict the added noise. After the training is finished, new samples can be generated by sampling an initial noise image x T ∼ N (0, I) and then iteratively removing portions of the predicted noise ϵ θ (x t , t, y) at each time step t = T, . . . , 1.This denoising process is formally defined byEQUATIONwith variance scheduler β t ∈ (0, 1), α t = 1 -β t and ᾱt = t i=1 α t . The noise predictor ϵ θ (x t , t, y), usually a U-Net [33] , receives an additional input y for conditional image generation.Common text-to-image DMs [32, 30, 34] are conditioned on text embeddings y computed by pretrained text encoders like CLIP [29] . The typical way to incorporate the conditioning y into the denoising process is the cross-attention mechanism [41] . (Cross-)Attention consists of three main components: query matrices Q = z t W Q , key matrices K = yW K , and value matrices V = yW V . All three matrices are computed by applying learned linear projections W Q , W K , and W V to the hidden image representation z t and the text embeddings y. The attention outputs are computed byEQUATIONwith scaling factor d. Importantly, in most text-to-image models, the noise predictor receives guidance only through the cross-attention layers, which renders them particularly relevant for memorization.Memorization. Memorization was extensively studied in supervised models and with respect to data labels [48, 1, 9] . Recently, studies have been extended to unlabeled self-supervised learning [25, 43] .In both setups, it was shown that memorization is required for generalization [12, 13, 43] . However, memorization also yields privacy risks [4, 7, 14, 40] since it can expose sensitive training data.In particular for generative models, including DMs, it was shown that memorization enables the extraction of training data points [4-6, 8, 36] .Localizing Memorization. Early work on localizing where inside machine learning (ML) models memorization happens focuses on small neural networks. Initial findings suggested that in supervised models, memorization happens in the deeper layers [2, 39] . However, more fine-grained analyses contradict these findings and identify that individual units, i.e., individual neurons or convolutional channels throughout the entire model, are responsible for memorization [24] . To identify these, Maini et al. [24] deactivate units throughout the network until a label flip on the memorized training input image occurs. However, due to the unavailability of labels, this approach does not transfer to DMs.Memorization in Diffusion Models. Recent empirical studies connect the model architecture, training data complexity, and the training procedure to the expected level of DM memorization [16] , while others connect memorization to the generalization of the generation process [47] . Two types of memorization are usually distinguished: Verbatim memorization that replicates the training image exactly. And template memorization that reproduces the general composition of the training image while having some non-semantic variations at fixed image positions [45] . Existing approaches for detecting memorized training samples are based on statistical differences in the model behavior when queried with memorized prompts. These approaches explore differences in predicted noise magnitudes [46] , the distribution of attention scores [31] , the amount of noise modification in one-step synthesis [45] , and the edge consistency in generated images [45] . Our work is orthogonal to these detection methods, focusing on the exact localization of memorization in the DM's U-Net rather than detecting memorized samples.Previously proposed methods for mitigating memorization during inference either rescale the attention logits [31] or adjust the text embeddings with a gradient-based approach to minimize the magnitude of noise predictions [46] . However, these inference time mitigation strategies are easy to deactivate in practice and provide no permanent mitigation strategies for publicly released models. In contrast, related training-based mitigation strategies [46, 31] shows the distribution of the z-scores of each neuron in the first cross-attention value layer. Memorization neurons produce considerably higher activations, here depicted as standardized z-scores, for memorized prompts, allowing them to be identified by outlier detection.3 NeMo: Localizing and Removing Memorization in Diffusion Models NEMO, our method for detecting memorization neurons, consists of a two-step selection process:(1) Initial Selection: We first identify a broad set of candidate neurons that might be responsible for memorizing a specific training sample. This initial selection is coarse-grained to speed up the computation among the many neurons in DMs. Consequently, it might select false positives, i.e., neurons not directly responsible for memorization.(2) Refinement: In the refinement step, we filter neurons out to reduce the size of the initial candidate set. After refinement, we deactivate the remaining memorization neurons to remove memorization.In our study, we apply this two-step approach of NEMO to detect memorization neurons in the DM's cross-attention layers, the only components that directly process the text embeddings. Image editing research [17, 42, 10] shows that cross-attention layers highly influence the generated content, so we expect them to be the driving force behind memorization. We analyze the impact of blocking individual key and value layers in Appx. C.8 and the influence of blocking neurons in the convolutional layers in Appx. C.9. Our results show that the value layers in the down-and mid-blocks of the U-Net indeed have the highest memorization effect, whereas value layers in the up-blocks barely affect the memorization. Deactivating the outputs of neurons in value layers completely blocks the information flow of the guidance signal and, hence, potential memorization triggers. Deactivating the key layers in cross-attention also impacts memorization but often impedes the image-prompt alignment. Similarly, deactivating neurons in the convolutional layers of the U-Net did not mitigate memorization. Therefore, we limit our search on memorization neurons to the value layers of the U-Net's down-and mid-blocks. While the identified neurons effectively mitigate the data replication problem, we emphasize that other parts of the U-Net might also play a crucial role in memorizing training data. Specifically, the identified neurons trigger the data replication, which is then executed by other parts of the U-Net, such as convolutional and fully connected layers. Deactivating the memorization neurons in value layers effectively interrupts the memorization chain and replication process. Before detailing the two steps of NEMO's selection process in Sec. 3.2 and Sec. 3.3, we first introduce how we quantify memorization strength in the next section. We provide detailed algorithmic descriptions for each of NEMO's components in Appx. D. 
quantifying the memorization strength: 
 Intuitively, the denoising process of DMs for memorized prompts follows a rather consistent trajectory to reconstruct the corresponding training image, yielding image generations with little diversity. Conversely, the denoising trajectory highly depends on the initially sampled noise for non-memorized prompts [46] . We measure the similarity between the first denoising steps for different initial seeds as a proxy to compare the denoising trajectories and to quantify the memorization strength. The higher the similarity, the more consistent the denoising trajectories, which indicates stronger memorization. Let x T ∼ N (0, I) be the initial noise following the denoising process described in Sec. 2.1.Let ϵ θ (x T , T, y) further denote the initial noise prediction. We found that the normalized difference between the initial noise and the first noise prediction δ = ϵ θ (x T , T, y) -x T for memorized prompts is more consistent for different seeds than for non-memorized prompts. We visualize this phenomenon for some initial noise differences in Appx. C.6.To detect the grade of memorization, we, therefore, use the similarity between the noise differences δ (i) and δ (j) generated with seeds i and j as a proxy. We measure the similarity with the common structural similarity index measure (SSIM) [44] . A formal description of the SSIM ∈ [-1, 1] score and an additional experiment outlining how the SSIM score can be used to detect memorization in the first place is provided in Appx. B.4.A higher SSIM indicates higher similarity between the noise differences, reflecting a higher degree of memorization. Notably, the SSIM computation only requires a single denoising step per seed, which makes the process fast. To set a memorization threshold τ mem , starting from which we define a sample as memorized, we first compute the mean SSIM on a holdout set of non-memorized prompts.We compute the pairwise SSIM between ten different initial noise samples for each prompt and take the maximum score. After that, we average the scores across all prompts and set the threshold τ mem to the mean plus one standard deviation. We consider the current image generation non-memorized if the maximum pairwise SSIM scores are below this threshold τ mem . Fig. 2a shows the distribution of SSIM scores for memorized and non-memorized prompts, demonstrating that memorized prompts lead to a substantially higher score.With our measure for quantifying the strength of memorization defined, we move on to detail the first step of our NEMO' localization. Our initial neuron selection procedure is based on the observation that activation patterns of memorized prompts differ from the ones of non-memorized prompts on the neuron level. Fig. 2b underlines this observation by plotting the standardized activation scores for memorized and non-memorized samples in the first value layer. Leveraging this insight, we identify memorization neurons as the ones that exhibit an out-of-distribution (OOD) activation behavior. We first compute the standard activation behavior of neurons on a separate hold-out set of non-memorized prompts. Then, we compare the activation pattern of the neurons for memorized prompts and identify neurons with OOD behavior. Let the cross-attention value layers of a DM be l ∈ {1, . . . , L}. We denote the activation of the i-th neuron in the l-th layer for prompt y as a l i (y). The activation values are averaged across the absolute neuron activations for each token vector in the text embedding. Let µ l i be the pre-computed mean activation and σ l i the corresponding standard deviation for this neuron. To detect neurons potentially responsible for the memorization of a memorized prompt y, we compute the standardized z-score [20] , defined asEQUATIONThe z-score quantifies the number of standard deviations σ l i by which the activation a l i (y) is above or below the mean activation µ l i . Here, the activation a l i (y) is calculated by taking the mean over the absolute token activations. To identify a neuron as exhibiting an OOD activation behavior, we set a threshold θ act and assume that neuron i in layer l has OOD behavior if |z l i (y)| > θ act . The lower the threshold θ act , the more neurons are labeled as OOD and added to the memorization neuron candidate set. An algorithmic description of the OOD detection step is provided in Alg. 2 in Appx. D.2. Fig. 2a shows that the pairwise SSIM score can be used to measure the generated sample's degree of memorization. Hence, to get an initial selection of memorization neurons, we calculate the standardized z-scores for all neurons and start with a relatively high value of θ act = 5. We deactivate all neurons with OOD activations given the current threshold θ act , i.e., setting the output of a neuron to 0 if |z l i (y)| > θ act , to reduce the memorization strength. If, after deactivating these neurons, the memorization score is not below the threshold τ mem , we then iteratively decrease the activation threshold θ act by 0.25 and update the candidate set until the target memorization score τ mem is reached.The activation patterns of some neurons in the network show high variance, even on non-memorized prompts. Such neurons can also be memorization neurons, but due to their high activation variance, they might not be detected by our OOD approach based solely on the z-scores. Therefore, we also add the top-k neurons of each layer with the highest absolute activation on the memorized prompt y to our current candidate set to account for such high-variance neurons. We start by setting k = 0 and increase k at each iteration by one if the memorization score is still above the threshold τ mem . We detail our initial selection process in Alg. 3 in Appx. D.3. All neurons identified by our OOD approach and the neurons with the k highest activations are then collected in the neuron set S initial . Since not all neurons in set S initial might be memorization neurons, we refine this set in the next step.In this step, we take the set of identified neurons S refined = S initial and remove the neurons that are actually not responsible for memorization. To speed up this process, we first group the identified neurons layer-wise, leading to the neuron set S l refined for layer l. We iterate over the individual layers l ∈ {1, . . . , L} and re-activate all identified neurons S l refined from a single layer l while keeping the identified neurons in the remaining layers deactivated.We then compute the SSIM-based memorization score and check if it is still below the threshold τ mem . If the memorization score does not increase above the threshold τ mem , we consider the candidate neurons S l refined of layer l as not memorizing and remove them from our set of neurons S refined . After iterating over all layers, the set S refined only contains neurons from layers that substantially influence the memorization score.Next, we individually check each remaining neuron in the set S refined by re-activating this particular neuron while keeping all other neurons in the set S refined deactivated. Again, if the memorization score computed on the remaining deactivated neurons does not exceed the memorization threshold τ mem , we remove this neuron from the set S refined . After iterating over all neurons in S refined , we consider the remaining neurons as memorizing and denote the final set of memorization neurons as S final . We detail this refinement approach in Alg. 4 in Appx. D.4. 
experiments: 
 We now empirically evaluate NEMO's localization in text-to-image DMs.We follow current research on memorization in DMs [46, 31] and investigate memorization in Stable Diffusion v1.4 [32] . Our set of memorized prompts consists of 500 LAION prompts [35] provided by Wen et al. [46] . We analyzed the prompts using the Self-Supervised Descriptor (SSCD) score [28] , a model designed to detect and quantify copying in DMs. The lower the score, the less similar the contents in the image pairs. Additionally, we split the dataset into verbatim-(VM) and template-memorized (TM) samples to enable a more detailed analysis of results. The hyperparameter selection and experimental conduction are independent of the type of memorization. If not further specified, we used the same hyperparameters for all the experiments in the paper.Images generated by VM prompts match the training image exactly, i.e., pixel-wise, independent of the chosen seed. TM prompts, on the other hand, reproduce the general composition of the training image while having some non-semantic variations at fixed image positions. Details about the analysis and the annotation can be found in Appx. C.1.Other publicly available models, like Stable Diffusion v2 and Deep Floyd [38] , are trained on more carefully curated and deduplicated datasets. We thoroughly checked for memorized prompts using the tools by Webster [45] and our SSIM-based memorization score but could not identify any properly memorized prompts. This result aligns with related research on memorization in DMs [46, 31] .Metrics: We split our metrics into memorization, diversity, and quality metrics. The memorization metrics measure the degree of memorization still present in the generated images. We generate ten images for each memorized prompt with activated/deactivated memorization neurons and measure the cosine similarities between image pairs using SSCD embeddings to quantify the memorization. We denote this metric by SSCD Gen . Since the generated images without deactivated neurons also differ in their degree of memorization from the original training images, we additionally measure the degree of memorization towards the original training images and denote this metric as SSCD Orig . Higher SSCD scores indicate a higher degree of memorization.Our diversity metric assesses the variety of images generated for the same memorized prompt with different seeds. Diversity is usually low for memorized samples, and generated images almost always depict the same image. Deactivating memorization neurons increases the diversity in the generations.Table 1 : Impact of Deactivating the Memorization Neurons. Keeping all neurons active (1st row) and randomly deactivating neurons (3rd row) has no impact on memorization. However, deactivating the memorization neurons located by NEMO (8th row) successfully mitigates memorization, increases diversity, and maintains prompt alignment. These results are comparable to the gradient-based mitigation strategies adjusting the prompt embeddings (2nd row) and the attention scaling (4th row). Adding random tokens also reduces memorization. However, for 1 or 4 tokens, the memorization, as quantified by the SSCD scores, is still higher than with deactivated memorization neurons. Adding 10 random tokens leads to comparable mitigation but also reduces the prompt alignment score. We quantify the sample diversity by computing the pairwise cosine similarity between the SSCD embeddings of different images generated for the same prompt. We refer to this metric as D SSCD , for which lower values indicate more image diversity.To assess the overall image quality of a DM with activated/deactivated neurons, we compute the Fréchet Inception Distance (FID) [18] , the CLIP-FID [21] , and the Kernel Inception Distance (KID) [3] on COCO [22] prompts. All quality computations follow Parmar et al. [26] to avoid biased results. Additionally, we compute the similarities between the generated images and the input prompts using CLIP scores [29] to ensure the alignment A CLIP between generated images and their prompts. The higher the alignment, the better the generated images represent the concepts described in the prompt.Importantly, we use different seeds for detecting memorization neurons with NEMO and the metric computations to avoid undesired biases during the evaluation due to seed overfitting. We always state each metric's median value and absolute deviation across ten seeds, except the quality metrics (FID, CLIP-FID, KID, and A CLIP ), for which we used five different seeds.We set the memorization score threshold to τ mem = 0.428, which corresponds to the mean plus one standard deviation of the pairwise SSIM score between initial noise differences measured on a holdout dataset of 50,000 LAION [35] prompts.Baselines: As a baseline, we repeated the image generations five times but replaced the deactivated memorization neurons with random neurons from the same layer. We also generated images using the inference mitigation strategy proposed by Wen et al. [46] , which performs a gradient-based adjustment of the text embeddings. Importantly, gradient-based mitigation strategies are memoryintensive, particularly for larger batch sizes. NEMO, however, computes no gradients, which enables the approach to also work on machines with limited computing resources. Additionally, we also selected the attention scaling method by Ren et al. [31] and the addition of random tokens, proposed by Somepalli et al. [36] , as baselines.We begin by demonstrating the effectiveness of our memorization localization method. Tab. 1 presents the quantitative results for images generated with the identified memorization neurons deactivated. NEMO detected a median of 4 and 21 memorization neurons for VM and TM prompts, respectively. For VM prompts, deactivating these memorization neurons significantly decreases memorization, as reflected by low memorization metrics SSCD Orig and SSCD Gen , while increasing the image diversity in terms of pairwise similarity D SSCD . However, the SSCD Orig does not change noticeably for TM prompts. This behavior results from the fact that TM prompts typically memorize specific parts of the original training image, such as objects or compositions, rendering the SSCD Orig metric less informative. In contrast, the SSCD Gen score, which compares similarities between images generated with and without the deactivated neurons, provides a more accurate measure. This score highlights that deactivating the identified neurons effectively alters the images and mitigates memorization. Importantly, the imageprompt alignment A CLIP remains constant in all cases, indicating that deactivating memorization neurons does not result in misguided image generations. We visualize examples of deactivating memorization neurons to avoid data replication and increase diversity in Fig. 3 .Comparing the results of deactivating the neurons identified by NEMO with those obtained from randomly deactivated neurons highlights that only a specific subset of neurons is actually responsible for memorizing a prompt. While deactivating the identified memorization neurons significantly impacts both memorization and the diversity of the generated images, randomly deactivating neurons has no noticeable effect.Moreover, the mitigation effect of deactivating memorization neurons is comparable to the stateof-the-art method of adjusting the prompt embeddings [46] . Yet, adjusting the prompt embeddings requires gradient computations for each seed and prompt, which are time-and memory-expensive, especially with large batch sizes. In contrast, once the memorization neurons are identified using our gradient-free NEMO, no additional computations are needed during image generations, thus adding no overhead to the generation process.Next, we analyze the distribution of the memorization neurons. Fig. 4a shows the total number of neurons responsible for memorizing specific prompts. Typically, a small set of neurons is responsible for verbatim memorization. For instance, 28 VM prompts from our dataset are memorized by a single neuron. Additionally, five or fewer neurons replicate two-thirds of VM images, indicating that verbatim memorization can often be precisely localized within the model. Template memorization can also frequently be pinpointed to a small set of neurons, with about 30% of TV replication triggered by five or fewer neurons. However, approximately one-third of TM prompts are distributed across 50 or more neurons. We hypothesize that this broader distribution results from the higher variation in generated images for TM prompts, where memorization spread across multiple neurons leads to increased image diversity. In contrast, VM prompts, often memorized by a small group of neurons, consistently produce the same image without variation. More detailed plots of the identified neurons can be found in Appx. C.2.Interestingly, we identify two neurons in the first cross-attention value layer responsible for the verbatim memorization of multiple prompts. Neuron #25 in this layer is associated with depicting people, while neuron #221 is responsible for memorizing multiple podcast covers. Together, these neurons account for memorizing 17% of our dataset's VM prompts. Similarly, neurons #507 and #517 in the third value layer are responsible for multiple TM prompts describing iPhone cases. The impact of deactivating these neurons on the image generation of memorized prompts is visualized in Appx. C.5. We also plot the distribution of the average layer-wise number of memorization neurons per prompt in Fig. 4b . Neurons responsible for VM prompts are primarily located in the value mappings of the first cross-attention layers within the U-Net's down-blocks (each block contains two cross-attention layers). A similar pattern appears for TM prompts, although value layers located deeper in the U-Net seem to play a more crucial role for TM prompts than for VM prompts.Until now, our focus has been on the impact of deactivating memorization neurons on memorized prompts. In this part, we investigate how these neurons influence non-memorized prompts and the overall image quality of the DM. To assess their impact, we deactivate varying numbers of memorization neurons, ordered by their frequency of occurrence as identified in our experiments, and compute the FID and KID scores on the COCO dataset. We also repeat the generations by deactivating the same number of randomly selected neurons that are not among the identified memorization neurons. As shown in Fig. 5a , there is no significant degradation in the image quality when blocking either the random neurons or the memorization neurons, even with up to 750 blocked neurons. This finding underscores the potential for pruning memorization neurons in DMs without compromising the overall image quality. The plot for the CLIP-FID metric and more detailed plots of the other two metrics, as well as an additional experiment measuring the disentanglement of the neurons in the value layers, can be found in Appx. C.3.We further analyze the impact of each component of NEMO and its sensitivity to hyperparameter selection. We discuss the most crucial insights here, with the complete study included in Appx. C.7. First, we evaluate the impact of different memorization thresholds θ mem . Lowering this threshold slightly increases the number of identified neurons but has a negligible effect on performance metrics.Selecting this threshold based on statistics computed on a holdout set provides a simple yet effective way for hyperparameter selection.Additionally, we compare the results of using both stages of NEMO versus a setting where we only perform the initial candidate selection, skipping the refinement process. Although memorization is successfully mitigated by deactivating the initially selected neurons, the number of identified memorization neurons increases substantially, resulting in a median of 26.5 neurons (+22.5 neurons) for VM prompts and 674.5 neurons (+653.5 neurons) for TM prompts. This highlights the importance of the refinement stage in reducing the number of neurons necessary to mitigate memorization efficiently. To further test whether the assumptions about the statistics of memorization neurons hold, we applied NEMO to a set of 500 non-memorized prompts not used to calibrate our thresholds. As we further show, NEMO does not identify any neurons for most of the non-memorized prompts, underscoring the validity of our assumptions.We also compared the effect of completely deactivating the identified memorization neurons to down-scaling their activations by a fixed factor. The SSCD scores in Fig. 5b , computed for different scaling factors, demonstrate that memorization is not fully mitigated when using a positive scaling factor. Conversely, negative scaling factors do not provide any additional mitigation compared to our default setting of deactivating the neurons (i.e., using a factor of zero).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We describe all settings, models, datasets, and hyperparameters used during our experiments in Section 4 and in Appx. B. Appx. D further provides detailed algorithmic descriptions for all steps of our method. Therefore, we expect reproduction to be possible regardless of whether the code is provided. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion and outlook: 
 DMs have rapidly become a cornerstone of computer vision. Yet, problems like memorization of training samples can lead to undesired replication of potentially sensitive or copyrighted training images. Previous research has primarily focused on identifying memorized prompts and proposing mitigation strategies by adjusting the DM's input. However, there has been a lack of understanding regarding the precise location of memorization within the model.Our research provides novel insights into the memorization mechanisms in text-to-image DMs. Unlike previous studies that focused on identifying memorized prompts, our approach, NEMO, is the first to localize memorization within the model and pinpoint individual neurons responsible for it. Traditional pruning methods [11] are orthogonal to our approach by pruning only structures to reduce the total parameter count of the model. Our memorization localization algorithm enables model providers to prune these memorization neurons, effectively mitigating memorization permanently without additional model training, which can be costly in terms of data and resources. Our mitigation can be executed without compromising the model's overall performance or the quality of the generated images, allowing model providers to deploy the resulting models without additional safeguards to prevent memorization.There are several directions to expand and build upon our method for detecting neurons responsible for memorization. One intriguing avenue is to investigate whether an adjusted version of NEMO can detect concept neurons [23] . These neurons are not responsible for memorizing a certain prompt but for generating a particular concept. Such an approach could enable model providers and users to perform knowledge editing [15] and remove undesired concepts like violence and nudity. Another exciting application for NEMO is in large language models, also known for memorizing training samples [5] . Identifying neurons responsible for memorizing text from the training data could lead to new mitigation strategies.Additionally, our insights could be interesting for developing new pruning algorithms for DMs to reduce the number of parameters while eliminating unintended memorization. As demonstrated in our experiments, pruning memorization neurons does not significantly impact the model's overall performance, which is crucial for effective pruning strategies.[ Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our research identifies neurons in text-to-image systems that are responsible for memorizing training samples. We strongly believe that our research has no negative societal impact or potentially harmful consequences. In stark contrast, our research tackles the memorization problem in diffusion models and might support the development of models and methods to mitigate such undesired memorization. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss our method's broader impact and advantages in localizing neurons responsible for memorization in diffusion models (Section 5). Localizing such memorization neurons is an effective way of mitigating memorization. We do not expect any potential malicious or unintended uses of our method.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not develop or release any kind of new model or dataset. Instead, we only investigate existing, publicly available models and datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We explicitly state the origin of data and models in the paper and properly credit the source of these assets (Appx. B.2). The authors wrote the code for conducting the experiments by themselves. Any code snippets taken from external sources are explicitly marked.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13  
 
 Appendix: Highly Memorized Prompts: For certain memorized prompts, our method identifies a set of over 100 neurons. Upon closer examination, we found that in these cases, memorization is distributed across many neurons in various layers, rather than being concentrated in a small group. Even deactivating a substantial number of neurons in the network does not eliminate memorization for these prompts. However, such instances, primarily TM prompts, were rare in our experiments. We provide examples of highly memorized prompts in Appx. C.4.Mitigating Template Memorization Is Harder Than Verbatim Memorization: Deactivating memorization neurons is effective for mitigating verbatim memorization, and often requires only a small number of neurons to be deactivated. In contrast, mitigating template memorization often requires deactivating more neurons, and even then, complete removal of memorization is not always possible in some rare cases. This difficulty arises because template memorization frequently results in diverse generations, with the memorized content corresponding to only certain aspects of the image. Distinguishing between these memorized parts and the remaining image parts is not always clearly achievable with our SSIM-based memorization strength. However, we emphasize that for the majority of prompts, deactivating the identified memorization neurons successfully removes template memorization.Runtime: For most memorized prompts, NEMO detects the memorization neurons in a few seconds.We timed the runtime of NEMO and found that on average NEMO can identify memorization neurons for verbatim memorized prompts within 14.2 seconds, while for template memorized prompts memorization neurons are identified in 43.7 seconds on average. As discussed in the previous paragraph, mitigating and localizing memorization for template memorized prompts is harder. We suspect this is one reason why NEMO's runtime is slightly longer for template memorized prompts.To get further insight into how long the runtime of each part of our algorithm is, we also timed the runtime for the algorithms D1-D4 separately. The results can be seen in Tab. 2. We performed all our experiments on NVIDIA DGX machines running NVIDIA DGX Server Version 5.2.0 and Ubuntu 20.04.5 LTS. The machines have 1.5 TB (machine 1) and 2 TB (machine 2) of RAM and contain NVIDIA Tesla V100 SXM3 32GB (machine 1) NVIDIA A100-SXM4-40GB (machine 2) GPUs with Intel(R) Xeon(R) Platinum 8174 (machine 1) and AMD EPYC 7742 64-core (machine 2) CPUs. We further relied on CUDA 12.1, Python 3.10.13, and PyTorch 2.2.2 with Torchvision 0.17.2 [27] for our experiments. All investigated models are publicly available on Hugging Face. For access, we used the Hugging Face diffusers library with version 0.27.1.We provide a Dockerfile with our code to make reproducing our results easier. In addition, all training and configuration files are available to reproduce the results stated in this paper.Experiments were mainly conducted on Stable Diffusion v1-4 [32] , publicly available at https: //huggingface.co/CompVis/stable-diffusion-v1-4. All details regarding the data, training parameters, limitations, and environmental impact are available at that URL. The model is available under the CreativeML OpenRAIL M license.The investigated prompts originate from the LAION2B-en [35] dataset used to train the DM. The set of memorized prompts is taken from Wen et al. [46] 2 , who collected the prompts by using the tool of Webster [45] . The LAION dataset itself is licensed under the Creative Common CC-BY 4.0. The images of the LAION dataset might be under copyright, so we do not include them in our code base; we only provide URLs to retrieve the images directly from their source.All images depicted throughout the paper are generated with fixed seeds, 50 inference steps, and a classifier-free guidance strength of 7 using the default DDIM scheduler. Notably, the seeds used for generating the images and computing the evaluation metrics differ from those used for our detection method NEMO to avoid seed overfitting.During detection with NEMO, no classifier-free guidance was used, which speeds up the detection since only a single forward pass per seed is required, compared to an additional forward pass on the null-text embedding with classifier-free guidance. We always used ten different seeds for each prompt.The threshold on the SSIM memorization score was set to τ mem = 0.428 during the experiments in the main paper. We vary this threshold and analyze its impact in our sensitivity analysis in Appx. C.7.We run all experiments -detection with NEMO and the generations for the metric computationswith half-precision (float16) to reduce the memory consumption and speed up the computations.We quantify the memorization strength during our experiments using the structural similarity index measure commonly used in the computer vision domain to assess the similarity between image pairs. Our memorization score is computed as follows: Let x T ∼ N (0, I) be the initial noisy image. Let ϵ θ (x T , T, y) further denote the initial noise prediction without any scaling by the scheduler. We found that the normalized difference between the initial noise and the first noise prediction δ = ϵ θ (x T , T, y) -x T for memorized prompts is substantially more consistent for different seeds than for non-memorized prompts. To detect the grade of memorization, we, therefore, use the similarity between the noise differences δ (i) and δ (j) generated with seeds i and j as a proxy. We measure the similarity with the common structural similarity index measure (SSIM) [44] . The SSIM ∈ [0, 1] between two noise differences δ (i) and δ (j) is defined byEQUATIONThe parameters µ i , µ j and σ 2 i , σ 2 j denote the mean and variance of the pixels in δ (i) an δ (j) , respectively. Likewise, σ ij denotes the covariance between the images. Following the original paper, C 1 and C 2 are small constants added for numerical stability.A higher SSIM indicates higher similarity between the noise differences, reflecting a higher degree of memorization. Notably, the SSIM computation only requires a single denoising step per seed, which makes the process fast.Indeed, the SSIM score itself can be used to detect memorization. We visualized the different SSIM score distributions for non-memorized and memorized prompts in Fig. 2a . To underline this observation with quantitative metrics, we ran additional experiments to explore the detection capabilities of our SSIM-based method. We measured the efficiency of the SSIM score for memorization detection on our dataset of memorized and non-memorized prompts. Without extensive hyperparameter tuning, this detection method achieves an AUROC of 98%, an accuracy of 94.2% (using a naive threshold of 0.5 on the SSIM similarity), and a TPR@1%FPR of 87.6%. Since the amount of memorization of template memorized prompts varies significantly, we repeated the computation for detecting the verbatim memorized prompts. Here, the SSIM approach even achieves an AUROC of 99.64%, an accuracy of 97.39% (with a threshold of 0.5), and a TPR@1%FPR of 98.21%. These results indicate that the SSIM score can also be used to detect memorization reliably in the first place. However, detection is not the focus of the paper but the localization on the neuron level.Webster [45] distinguished between verbatim and template memorized prompts. Verbatim memorized prompts lead to the exact reconstruction of training samples, while template-memorized prompts replicate the composition and structure of the training image. To provide a more fine-grained analysis of our results, we classify the prompts in our dataset into these two categories. We distinguish between both types by computing the SSCD [28] scores between the original training image and ten generations with different seeds. We then classify a prompt as verbatim memorized if the maximum SSCD score computed as cosine similarity exceeds a threshold of 0.7 and as template memorized otherwise. Fig. 6 plots the distribution of SSCD scores for both datasets. We manually inspected and classified the prompts where the original training image is no longer available (16 out of 500). Table 3 : We measured the disentanglement of neurons for deactivating top-k memorization neurons or randomly selected neurons. More specifically, we collected and compared the attention layer outputs for 500 non-memorized prompts with and without neurons deactivated and 3 seeds. We then computed the average cosine similarity between corresponding outputs. The high similarities show that blocking neurons only has a negligible impact on the outputs of the attention layer, suggesting that other neurons can substitute the functionality of blocked neurons on non-memorized prompts. We ran additional experiments to analyze the impact and effects of deactivating neurons in the value layers. The result can be seen in Tab. 3. We deactivated a varying number of neurons, either randomly selected or the top memorization neurons, and measured the similarity between the outputs of the cross-attention blocks with deactivated neurons and all neurons activated. Our results, stated in the rebuttal PDF, show that even deactivating a large number of neurons only impacts the attention outputs marginally. We, therefore, conclude that other neurons can replace the functionality of specific neurons on non-memorized prompts and that neurons in the value layers act rather independently. Yet, for neurons memorizing specific prompts, this memorizing functionality is not replaced by other neurons, explaining the mitigation effect of deactivating these neurons.Falmouth Navy Blue Area Rug by Andover Mills (156 memorization neurons)Renegade Figure 9 : Memorization of highly memorized prompts is distributed across many neurons in various layers, rather than concentrated in a small group of neurons. We show examples of such prompts and the impact of deactivating the identified memorization neurons. The number of memorization neurons in each case is stated behind each prompt.To illustrate that some single neurons are responsible for memorizing multiple training prompts, we generated images with and without these specific neurons deactivated. In Fig. 10 and Fig. 11 , we only deactivate a single neuron each in the first value layer, whereas in Fig. 12 , we deactivate two neurons in the third value layer."Watch: Passion Pit's New Video, ""Lifted Up (1985)""" "Watch: Passion Pit's New Video, ""Lifted Up (1985)""" "Listen to Ricky Gervais Perform ""Slough"" as David Brent" "Listen to Ricky Gervais Perform ""Slough"" as David Brent" Figure 13 : Visualizations for generated images and the noise differences between the predicted noise after the first denoising step and the initial Gaussian noise. Noise differences for memorized prompts (left column) have low diversity and are already structurally similar to the final image. The noise differences for non-memorized prompts (right column) show no clear structure and differ substantially for different noise initializations. Deactivating the memorization neurons detected with NEMO (middle column) removes the structure in the initial noise differences and adds more diversity, leading to diverse image generations.We conduct an ablation study to investigate the impact of the individual components of NEMO. Additionally, we analyze the sensitivity of the memorization threshold τ mem and explore alternatives to deactivating neurons by setting their activations to zero. The results for the various settings are presented in Tab. 4.The first two rows provide evaluation results for the model with all neurons active and with randomly deactivated neurons. Both scenarios exhibit strong memorization. The third row shows the results of blocking the neurons identified as memorizing by NEMO, using the threshold τ mem = 0.428 as specified in the main paper. This threshold corresponds to the mean SSIM memorization plus one standard deviation, calculated on a holdout set of 50,000 non-memorized LAION prompts. In row four, we repeat this setting using classifier-free guidance (CFG) with a guidance strength of 7.0, as opposed to our default setting without CFG. Detection with CFG further reduces the number of detected memorization neurons. However, the SSCD scores indicate slightly increased memorization after deactivating the identified neurons. Additionally, running NEMO with CFG doubles the number of forward passes in the U-Net since a separate noise prediction is generated for each initial seed.Rows five to seven display the results of varying the memorization threshold τ mem . Specifically, we adjust the threshold to one standard deviation below the mean SSIM score, to the mean, and two standard deviations above the mean. A lower threshold identifies more neurons. However, for lower thresholds, the metrics are comparable to those obtained with our default threshold value (τ mem = 0.428). Increasing the threshold reduces the number of identified neurons but slightly increases memorization, as measured by the SSCD scores. Thus, a trade-off exists between reducing the number of identified memorization neurons and their memorization mitigation effect. In addition, we provide heat maps to directly compare the impact of different thresholds τ mem used during the initial selection and the refinement step in Fig. 14 . Fig. 15 further compares the SSCD scores for varying the threshold values.Rows eight and nine examine the impact of removing the refinement step or incorporating no neurons with top-k activations during the initial selection. As anticipated, without refinement, the number of identified neurons increases substantially. Despite this, the various metrics remain comparable to those obtained after the refinement step, even with more neurons deactivated. This underscores the robustness of image generations against pruning out-of-distribution (OOD) neurons. Without the top-k selection, NEMO identifies a larger set of neurons. However, deactivating these neurons does not mitigate memorization as effectively as with a top-k search. Notably, for template verbatim prompts, the SSCD Gen is substantially higher without top-k, indicating increased memorization.In the remaining rows, we explore the impact of scaling the activations of memorization neurons instead of deactivating them. With negative scaling factors, the results are comparable to those of completely deactivating the neurons. For positive scaling factors, however, the generated images demonstrate higher degrees of memorization, with a scaling factor of 0.75 having almost no influence on memorization.We also apply NEMO to a set of 500 LAION non-memorized prompts, different from the 50,000 prompts used to set the memorization threshold. For 442 of these prompts, NEMO identified no memorization neurons, which is to be expected since these prompts show no memorization behavior. For the remaining prompts, a median of 62 ± 27 neurons was found. The left plots show the results for verbatim memorization prompts, while the right plots show the results for template memorization prompts. The value of the thresholds does not seem to have a high impact on the memorization scores. Since higher thresholds identify much less memorization neurons, choosing a threshold of τ mem = 0.428 is a valid choice.During our experiments in the main paper, we limit our search with NEMO for memorization neurons to cross-attention value layers in the down-and mid-blocks of the U-Net. To motivate this decision, we perform an analysis of the influence of neurons in the individual key and value layers of different cross-attention blocks. Let us first recall the computation performed in attention layers:EQUATIONThe computed key and query matrices K and Q are used to calculate the attention scores, i.e., the weighting of the components in the value matrix V . In the cross-attention layers, the query matrix Q is computed by linearly mapping the current feature maps from the previous U-Net layer. Therefore, information from the textual guidance is only indirectly contained, i.e., of earlier layers or the U-Nets input feature map after the first denoising step. Therefore, we can exclude neurons in the query mapping layers since we aim to identify neurons directly responsible for memorization. The neurons in the key mapping layers directly process the text embeddings to compute the attention scores. However, strong interdependencies exist between the activations of different neurons through the nature of the softmax function. The impact each neuron's activation has on the computed attention score also depends on the activations of all other neurons from the same layer. Removing a single neuron, i.e., setting its activation to zero, does not necessarily imply substantial changes in the attention scores and the corresponding weighting of features from the value mapping layer.The value mapping layers, however, also directly process the text embeddings, but there is no direct interdependence between the activations of different neurons. Consequently, setting the activations of individual neurons in value layers to zero directly blocks the information flow from the text embeddings. We hypothesize that the neurons in the value layers are mainly responsible for memorizing the text embeddings of specific prompts.We evaluate this assumption by taking a set of 100 memorized prompts, generating ten samples for each prompt, and comparing the impact of removing neurons from different layers. More specifically, we remove all activations of individual key and value mapping layers, i.e., setting the output vectors of these layers to zero while keeping all other parts of the model untouched. We then compare the generated images with removed activations to the original training images. Fig. 16 plots the resulting SSCD similarity scores for deactivating individual value (top row) and key (bottom row) layers. We distinguish between verbatim (left column) and template (right column) prompts. The plots show the maximum and median SSCD scores and the deviations for the median scores. We decided not to plot deviations for the maximum score to improve readability. However, deviations are comparable to the median scores. Baselines computed without any deactivated neurons are plotted as dashed lines.Stable Diffusion contains six cross-attention layers in the down-blocks, one in the mid-block, and nine in the up-blocks. The vertical lines indicate the separation between the different blocks. For the value layers, the layers with indexes 1 (down-lock) and 7 (mid-block) have the highest impact, whereas layers later in the network hardly change the SSCD scores. Also, the effect of the remaining layers in the down-blocks is small on their own. However, we expect there to be entwined effects between deactivating neurons in different layers, which is why we also searched for memorization neurons in these down-block layers.For the key layers, particularly layers 4 and 6 in the down-blocks have the strongest impact on the generated images. However, removing these layers often produces images that no longer align with the concepts in the prompt or degrades the image quality, both leading to lower SSCD scores. We quantify this behavior by computing the alignments between the generated images and the corresponding input prompts in Fig. 17 . While deactivating individual value layers only slightly decreases the alignment scores, deactivating some key layers substantially reduces the alignment. To further illustrate this fact, we plot some of the generated images for deactivating individual value layers in Fig. 18 and for key layers in Fig. 19 . Figure 17 : CLIP alignment scores between memorized generations and the corresponding input prompt. Scores are computed for 100 prompts and ten different seeds per generation. We then take the median alignment scores of each prompt. During the generation, we deactivated individual value and key layers of the cross-attention blocks in the network. A higher alignment score indicates a better representation of the prompt concepts in the generated images. Dashed lines denote the median alignment scores for images generated without deactivating any neurons. For both types of memorization, deactivating value layers decreases the alignment only slightly, whereas deactivating some key layers substantially reduces the alignment.We also experimented with deactivating neurons in other U-Net layers, including both convolutional and fully connected layers, but we did not find any indicators of memorization in these units. Even when deactivating numerous neurons in the convolutional and fully connected layers of the U-Net, the memorized training images were still faithfully reproduced. However, the quality of the images degraded, particularly when deactivating neurons in early layers, which are responsible for defining the image structure. We showcase in Fig. 20 various examples of memorized images generated with 50% deactivated neurons in the convolutional layers to illustrate that these neurons have no noticeable impact on the memorization behavior. However, deactivating too many neurons in early layers can negatively affect the overall generation process. So, to conclude, deactivating other neurons in the U-Net didn't seem to impact memorization, which is why our choice to focus on the value layers seems reasonable. Alg. 1 defines our algorithm to compute the differences between the initial noise samples and the noise predicted during the first denoising step. The resulting noise differences are used to compute our SSIM-based memorization score during the initial neuron selection and the refinement step. We compute the noise differences always for n = 10 different seeds to avoid undesired biases due to the random sampling process. We further remove noise differences from the set, which have low similarity with other noise differences. By this step, we remove noise differences for seeds that do not lead to memorization and might mislead the algorithm. Justification: The paper does not include any theoretical proof but offers a practical method (Section 3) to detect and localize memorization in diffusion models. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.Available at https://github.com/YuxinWenRick/diffusion_memorization. 


Decision: This paper proposed a simple method to localize memorization to individual neurons and showed that deactivating those neurons can have multiple benefits such as reduced memorization and increased diversity. The reviewers generally found this work interesting and novel. This would could have potential applications for future text-to-image generation model and system designs.

Reviews: review: paper_summary: paper_summary: This paper introduces NEMO, a method for localizing memorization in diffusion models (DMs) down to the level of individual neurons. The authors empirically evaluate NEMO on the Stable Diffusion model and demonstrate that deactivating the identified memorization neurons effectively mitigates memorization, increases image diversity, and maintains prompt alignment. The paper makes valuable contributions by providing insights into the memorization mechanisms in DMs and proposing a method for permanently mitigating memorization without compromising overall model performance or the quality of the generated images.
strength_weakness: strength_weakness: The paper lacks a thorough evaluation of the proposed method's performance compared to existing baselines (if exist) or state-of-the-art methods for mitigating memorization in DMs. It would be beneficial to include comparisons with other methods to assess the novelty and effectiveness of NEMO.
questions: questions: Can the authors provide more details on the selection of the memorization threshold $τ_mem$? How was it determined based on the pairwise SSIM scores? Did the authors consider any alternative methods for setting the threshold?
Have the authors considered comparing NEMO with other existing methods for mitigating memorization in DMs, such as rescaling attention logits or adjusting prompt embeddings? How does NEMO perform in terms of effectiveness and efficiency compared to these methods?
limitations: limitations: yes
paper_summary: paper_summary: The paper localize memorized samples to neurons in the cross attention layers in diffusion models. By deactivating neurons responsible for memorization, the proposed method enables models to generate diverse images different from training images.
strength_weakness: strength_weakness: 1. I understand that detecting memorized propmts is out of scope of this paper. Previous methods in mitigating memorization construct an entire pipeline in which memorized prompts are first detected with a heuristic/metric, and a mitigation strategy is triggered during inference/training if the metric surpasses a certain threshold. My expectation from "localization of memorization" was that there is a specific subset of neurons that, when blocked, avoid replication of training data. Such a localization would not require the authors to develop a pipeline to detect memorization as those neurons can be pruned. 

(a) From the paper (Table 1 and Figure 4) it seems that the proposed method operates on a per-prompt basis and all experiments have been conducted by separating the memorized prompts into template and verbatim.

(b) I believe that although these findings are interesting, one weakness is that the method is impractical as in practice, it would still require one to know the type of memorization before deciding which neurons to block. For some highly memorized prompts, the highly-iterative neuron discovery process (Alg D.1 to D.4) needs to be repeated. This not only requires interference with the inference pipeline but is also computationally inefficient.
questions: questions: 1. How are tokens handled while calculating neuron activation in Equation 3? Do you consider an average or max over tokens?
2. How faithful is SSIM for detecting memorized prompts? Can it be used to detect if a prompt is memorized or not?
limitations: limitations: The authors have clearly addressed the limitations of the proposed method in the paper. I believe some more limitations are addressed in the Weaknesses section.
paper_summary: paper_summary: This paper introduces NEMO, a method that identifies and deactivates specific neurons responsible for memorizing training data in DMs. This approach prevents the reproduction of sensitive images, enhances output diversity, and mitigates data leakage, enabling more responsible use of DMs.
strength_weakness: strength_weakness: 1. The author should specify their novelty compared with [1]. The novelty seems limited considering the conclusion in [1].

2. No practical mitigation method is provided. Deactivating is not possible without knowing the neurons.

3. The author claim that most memorization happens in the value mappings of the cross-attention layers of DMs. However, actually, the author only prove that some neurons in attention layers are necessary in memorization. They didn't show that those neurons are everything about the memorization. It is possible that deactivating other neurons that are not in attention layers can also remove memorization. Localizing should consider how to exclude other neurons.
questions: questions: The author should address weakness 1 and 3.
limitations: limitations: The limitations that the author mentioned in Appd A is about how to further improve the method. It is not clear solved but propose the key factors is possible to solve in the future work.
paper_summary: paper_summary: This paper introduces NeMo, a straightforward and easy-to-understand method. NeMo identifies neurons responsible for memorization and deactivates them, offering potential applications in the protection of sensitive or copyrighted data.
strength_weakness: strength_weakness: 1, There is insufficient comparison with other methods, such as the work mentioned in [1]. The authors state, "While mitigations that rely on preventing the generation of memorized samples are effective when the DM is developed and deployed in a secure environment, they hold the inherent risk of adversaries circumventing them. Additionally, they are not effective when the DMs are publicly released, such that users can freely interact with them." However, this argument is not persuasive enough. Providing more compelling reasons for the unique position of this method in mitigating memorization would enhance my evaluation score.

2, The side effects of this method are not thoroughly investigated.


[1] Detecting, Explaining, and Mitigating Memorization in Diffusion Models
questions: questions: 1, I have some reservations about the connection between copyright/sensitive images and memorization. It seems that copyright issues might only be partially triggered by memorization. Could you provide further clarification on this point? This pertains to the application of your method.

2, I find the activation-based localization method reasonable. However, I'm curious if pruning methods such as [1] or other neuron localization methods such as [2] could also be effective here. I would appreciate some analysis on this, though I do not require any additional experiments.

3, In section 4.3, could you explain why deactivating neurons further reduces FID and KID in some cases?

4, I'm also interested in the potential side effects of this method. You mention that deactivation has a minimal impact on non-memorized generation, which suggests that these neurons do not interact significantly with other neurons and function independently. I believe a more detailed analysis from this perspective would be beneficial.


[1] Structural Pruning for Diffusion Models
[2] Cones: Concept Neurons in Diffusion Models for Customized Generation.
limitations: limitations: The limitations have been discussed.

Comments: title: Thank you for the rebuttal
Have concerns been addressed?
Additional Runtime Results
Thank you for the rebuttal
Missed reference and follow-up response
Can you specify the missing reference?
Missing Reference
comment: Thank you for your prompt response and your inclination toward accepting our paper. We are still eager to convince you even more of our method.

**1.)** We respectfully disagree with the concern that our method might not effectively identify the relevant neurons. For our set of memorized prompts, our method successfully identified 200 neurons in the first value layer out of a total of 320 neurons—constituting 62.5\% of the neurons in this layer. The memorization of prompts is actually distributed across multiple different neurons. Importantly, deactivating the neurons associated with the memorization of one prompt does not affect the memorization of other prompts, provided they do not share the same memorizing neurons. Furthermore, if memorization is triggered by neurons in the first value layer, deactivating neurons in subsequent layers does not prevent the generation of memorized samples. This observation extends to deeper layers as well—deactivating neurons in other value layers similarly does not mitigate memorization. Additionally, we have pinpointed neurons within the value layers of the U-Net’s first attention blocks as the primary trigger points of memorization, which represents a relatively small subset of the model's overall parameters.

Regarding the hypothesis that a fixed set of neurons responsible for all memorization might exist, we would like to emphasize that the training objective of diffusion models (U-Nets) differs significantly from that of supervised image classifiers or next-token prediction in language models. Consequently, insights into memorization within these architectures may not directly apply to diffusion models. Without evidence or proof of a fixed set of neurons (which our experiments did not support), we believe this does not constitute a limitation of our method.

We would also like to highlight that, similar to [1], "we determine the minimum number of neurons required to flip an example’s predictions" [1]. From our perspective, our method extends the line of research on memorization in supervised learning. The results we present for text-to-image models provide insights that are comparable to those from supervised learning, which identify a set of neurons for individual training images.

**3.)** Our method does not necessarily rely on Wen et al.'s approach for detecting memorized prompts. As demonstrated in our previous response, our SSIM-based detection method is also capable of identifying memorization. Since it only requires a single denoising step, it can be applied at inference time to detect memorized prompts. We chose to use the evaluation dataset provided by Wen et al. to establish a benchmark and ensure our method is directly comparable to related work.

Overall, we are very pleased that you appreciate our paper and the idea of localizing memorization in diffusion models. We will also expand our discussion in the paper to address the detection of previously undiscovered memorization. We thank you for this fruitful discussion.
Thank you for your detailed rebuttal. We're glad that you are leaning towards accepting the paper. If you have any further questions or comments, we would be happy to provide additional information.
Thank you for your detailed rebuttal and for the engaging and fruitful discussion. Your insights will definitely enhance our paper. We're also pleased to see that you have raised your score in favor of accepting the paper.
Thank you for clarifying my doubts. I have a much better understanding of the paper. I appreciate that the authors calculated the time required to run their algorithm and I encourage them to add this analysis in the paper. I still have some concerns listed below. 

1. The authors said that 
>  While some sets of neurons memorize multiple prompts, no fixed set of neurons is responsible for memorizing all prompts. This is not a weakness of our method, but rather a reflection of the behavior of already trained models.

This may be due to the method you proposed for searching neurons not being able to identify a fixed set of neurons. Previous studies have successfully localized memorization to a specific region of the model in image classifiers [1] and language models [2], suggesting that it could also be achievable in a diffusion model. The proposed method may lack the capability to accomplish this.
[1] Pratyush Maini, Michael C. Mozer, Hanie Sedghi, Zachary C. Lipton, J. Zico Kolter, and Chiyuan
385 Zhang. Can neural network memorization be localized?
[2] Localizing Paragraph Memorization in Language Models, Stoehr et al

3. A significant limitation I see in the paper, which could be quite restricting, is that the proposed method requires observing all memorized prompts extracted for SD using a specific extraction method—in this case, the approach outlined in Wen et al.'s paper. However, this does not represent an exhaustive set of prompts that the model has memorized. Given that their method does not rely on identifying a fixed set of neurons, I am concerned that it may not extend to undiscovered prompts. The authors should consider explicitly stating this limitation in their paper as a disclaimer to avoid giving a false sense of security.

Overall, I will lean toward a positive rating (and keep my rating) because I like the idea of localizing memorization in DMs and this paper presents interesting insights.
I have raised my score.
Dear Reviewers,

We sincerely appreciate your questions and feedback, which have helped us to further improve our paper. We would like to follow up on our responses to ensure that they adequately address your concerns. If there are any additional questions or areas where further clarification is needed, please don't hesitate to reach out.

Thank you very much for your time and effort.

Best regards,  
The Authors
Thank you for pointing that out. We agree that the input prompt serves as the actual initiation of the memorization process, and without a specifically memorized prompt, the model is unlikely to trigger memorization. In this regard, transforming the prompt into an embedding vector is critical and marks the start of this process. If the text encoder would transform the prompt into a different embedding—for instance, by deactivating parts of the model or through fine-tuning—memorization might not occur in the output of the diffusion model. However, as noted in our previous response, we interpret the text embedding as a fixed conditioning factor since the text encoder (CLIP for Stable Diffusion) was trained separately using a contrastive method (on OpenAI’s internal data) and likely never saw the exact prompts memorized by the diffusion model (LAION 2B) during its training. In this scenario, the processing of the text embedding begins with the key and value layers in the U-Net, marking the start of the memorization chain within the U-Net, which is triggered by the input embedding.

We will incorporate the perspective and discussion of the memorization chain, starting from the input prompt, into our final paper. We hope this addresses all your questions and concerns regarding the memorization aspect.
Thanks for your response. For the second question, I would say before K and V, there are also some components that would play some roles in memorization. For example, transformations are conducted from text embedding space to the input space of K and V. That is to say, the memorization is like a chain, while the attetion neurons are one part in the middle. Since it is in the middle, it is not correct to say they trigger memorization. They are only the one that is found to have special behaviors which are connected with memorization. They are actually triggered by their input.
I appreciate the authors' detailed rebuttal and additional insights. Most of my questions have been addressed, and I will be maintaining my original rating.
Dear Reviewer zaL8,

Thank you for your question. We have rerun the runtime experiments and measured the time for each algorithm D1-D4. Please find the values in seconds in the following table:

|    | D1   | D2   | D3   | D4    | Total |
|----|------|------|------|-------|-------|
| VM | 0.21 | 0.29 | 2.31 | 12.85 | 15.68 |
| TM | 0.18 | 0.25 | 6.09 | 39.37 | 45.90 |

In the table above, one can clearly see that the algorithms D1 and D2 take less than a second to run, while D3 takes roughly 2 seconds for verbatim memorized prompts and 6 seconds for template memorized prompts. As can be seen, D4 has the longest runtime. This is because each candidate neuron is checked individually for memorization by iterating over them and setting their output to zero individually. For template memorized samples, this takes a bit longer since, for this kind of memorization, our initial neuron candidate set is larger than for verbatim memorized samples. However, even though this process takes a bit longer for TM than for VM, the overall runtime is still only 45 seconds. Additionally, we want to emphasize here again that our algorithm only has to be run once. In contrast to existing mitigation approaches, once the memorization neurons are identified, the inference time, as well as the inference memory consumption, will be the same as that of the default diffusion model.
Dear Reviewer K7EP,

thank you for your quick response and the clarification of the missing reference.

**1.) The contribution of this paper seems limited given that Ren et al. show that memorization and cross attention may have a connection**: We first emphasize that the referred paper (Ren et al.) is an unpublished preprint made available on Arxiv about 2 months (March 17th) before the NeurIPS deadline, such that it would fulfill the definition of concurrent work. Additionally, a thorough comparison between the two papers yields that they are conceptually different and that our work has substantially novel contributions:
- a) Ren et al. investigate the attention scores, i.e., the distribution of the dot product between key and query vectors, finding that attention concentrates more on the beginning token for non-memorized prompts and is more concentrated on certain trigger tokens for memorized prompts. Whereas this is an interesting finding, it is orthogonal to our work. While Ren et al. investigate the model's input space in terms of CLIP embeddings for memorization, we focus on the U-Net itself and identify the neurons in the value mapping responsible for triggering memorization.
- b) The findings of Ren et al. are more coarse grained (layer focus rather than neuron focus), which enables them to make a related finding to ours. Still, they do not consider the most interesting part of our research, namely that most of the memorization is triggered by 1 or a handful neurons in the value layers.
- c) The inference-time mitigation strategy by Ren et al. is conceptually different from ours. Based on their findings 1 and 2 (both on the input level), where they look at token entropy, their method consists of masking out the summary tokens and increasing the logits of the beginning token. This strategy is very different from ours, which prunes individual neurons from the value mapping in cross-attention layers. Also, their training time mitigation simply removes samples with high attention entropy, which is somewhat related to our detection of out-of-distribution activation in the value layers. Yet, our mitigation strategy acts at inference time, a more realistic and practical scenario given the amount of computing required for training text-to-image models.

We, therefore, kindly disagree that our paper's contribution is limited. We also emphasize that the other three Reviewers all agree on the novelty and importance of our research and findings.

**2.) Memorization in Other Layers**: It seems there might have been some misunderstanding between the question raised and our response. We fully acknowledge that layers following the value mappings in the U-Net also play a significant role in memorizing training samples. The convolutional and fully connected layers, being integral to the process of constructing images by denoising the initial input, naturally contribute to the memorization of the appearance of training samples.

However, the focus of our paper is on identifying the specific signals that trigger the generation and reproduction of memorized data in the first place. Our findings indicate that certain neurons in the value layers of the cross-attention blocks act as such triggers. We do not claim that these neurons are solely responsible for memorization or that other model components are irrelevant to the process. Our research specifically aims to detect these triggering parts within the model, and our analysis identified them in the value layers of cross-attention blocks.

While we also analyzed the convolutional layers, we did not find individual channels exhibiting similar trigger behavior, where removing such channels would mitigate memorization. This does not imply that these layers are unimportant for reconstructing memorized samples. We are happy to clarify this point in our paper.
Dear Reviewer zaL8,

Thank you for your quick response and appreciating our rebuttal. We are happy to provide additional information and clarifications.

1.) We will run some additional experiments for your first question regarding the runtime of individual parts of our algorithm and provide them as soon as they become available.

2.) In our paper, we followed the first approach you sketched. We identify the neurons of each memorized prompt individually and then deactivate/prune the union of these neurons in the model. The metrics in the paper are computed on the U-Net, with all identified neurons removed simultaneously.
Thank you for the effort you put into this rebuttal. I appreciate the authors' detailed clarifications on some things I might have misunderstood but now I have more questions.

> While the runtime varies based on the strength of the memorization, for verbatim memorized prompts, which represent the strongest type of memorization, the runtime was only a few seconds (as discussed in our limitations section in Appendix A). 

1. Can you quantify the time taken to discover these memorization neurons? I would like to see the time taken to run Alg 1,2,3,4 separately.

2. I am a bit confused about the pipeline now as this is not explicitly stated in the paper. Do you (1) first find memorization neurons for every memorized prompt, and then prune the union of these memorization neurons from the model permanently? or (2) For every prompt, find memorized neurons, prune the model, evaluate, and then reset the model?
I would suggest to list your response. Otherwise it would increase the cost to write the response for reviewers, which may decrease the possibility to get a response.
Sorry about the missed reference. Please compare the novelty with Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention. Since they also find the attention score may show connection with memorization. The contribution of this paper seems limited.

I disagree with your reponse to weakness 3. From the end of input to the end of output, the models are structued by layer by layer and module by module. Value may memorize text embedding, but the following layer must also memorize the output of attention layer. Otherwise the final output cannot be memorized. It is like a chain. Memorization will not work without the any one of the layer. The memorization is never something that is totally dependent on one neuron or a few neurons. In other words, these neurons are just some clues or signals of memorization. It is not correct to say they control memorization. It is correct to say value or kay layers has a very special behavior that is related to memorization. But they didn't decides the whole thing. In other layers like the convolutional layers in u-net, they must also memorize their input. That memorization is not probabily about one neuron. It is maybe need many neurons for a memorization. The author cannot say something they didn't find does not exist. If the author acknowledge this and update in their future version, I will increase my score. Otherwise I would possibily decrease my score.
Dear Reviewer K7EP,

Can you specify the missing reference [1] in your review? It seems to have been dropped in your original review. I'm not sure if there will be enough time for the authors to provide comparison but at least some clarifications regarding the novelty could be discussed during this week. Thanks!

AC
Dear ACs and PCs,
dear Reviewer

due to a lack of reference [1] in the review, we were not able to provide the comparison requested by the reviewer during the rebuttal phase. We would have been happy to provide the requested information, but this was not possible during the initial rebuttal phase without further details from the reviewer. 

We, therefore, kindly ask the Reviewer to provide additional information.

Best regards,  
the authors