Exploring Local Memorization in Diffusion Models via Bright Ending Attention

## Abstract

Abstract In this paper, we identify and leverage a novel ‘bright ending’ (BE) anomaly in diffusion models prone to memorizing training images to address a new task: locating localized memorization regions within these models.
BE refers to a distinct cross-attention pattern observed in text-to-image generations using diffusion models. Specifically, memorized image patches exhibit significantly greater attention to the end token during the final inference step compared to non-memorized patches. This attention map effectively highlights regions where the generated image replicates training data.
Furthermore, driven by our observation that local memorization significantly underperforms in existing tasks of measuring, detecting, and mitigating memorization in diffusion models compared to global memorization, we propose a simple yet effective method to integrate BE and the results of the new localization task into these existing frameworks. This integration effectively improves their performances by narrowing the performance gap caused by local memorization.
Our results not only demonstrate the successful execution of the new localization task but also establish new state-of-the-art performance across all existing tasks, underscoring the significance of the BE phenomenon.

## 1 Introduction

Text-to-image diffusion models like Stable Diffusion (Rombach et al., ) have achieved unparalleled proficiency in creating images that not only showcase exceptional fidelity and diversity but also closely correspond with the user’s input textual prompts.
This advancement has garnered attention from a broad spectrum of users, leading to the extensive dissemination and commercial utilization of models trained on comprehensive web-scale datasets, such as LAION (Schuhmann et al., ), alongside their produced images.
However, this widespread usage introduces legal complexities for both the proprietors and users of these models, particularly when the training datasets encompass copyrighted content.
The inherent ability of these models to memorize and replicate training data during inference raises significant concerns, potentially infringing on copyright laws without notifying either the model’s owners or users or the copyright holders of the replicated content.
The challenge is further exacerbated by the training datasets’ vast size, which makes thorough human scrutiny unfeasible.
Illustratively, several high-profile lawsuits (Saveri & Matthew, ) have been initiated against entities like Stability AI, DeviantArt, Midjourney, and Runway AI by distinguished artists. These lawsuits argue that Stable Diffusion acts as a ‘21st-century collage tool’, remixing copyrighted works of countless artists used in its training data.

In response to these legal challenges, Carlini et al. () and Somepalli et al. () proposed similarity metrics to evaluate memorization, and recent efforts (Somepalli et al., ; Wen et al., ; Chen et al., ) have focused on developing strategies to detect and mitigate memorization, achieving notable success. However, these metrics and strategies adopt a global perspective, comparing entire generated images to training images.
We identify a significant gap in these approaches when dealing with cases where only parts of the training image are memorized, which we refer to as local memorization, as opposed to cases where the entire training image is memorized, termed global memorization. Specifically, current methods can be easily tricked in local memorization scenarios by introducing diversity in non-memorized regions, resulting in many false negatives during evaluation and detection. Consequently, this failure to correctly identify local memorization may prevent the activation of mitigation strategies.
Motivated by this research gap, we developed a localization insight as a novel view of the memorization issues, where we argue that improved metrics and strategies should concentrate exclusively on the locally memorized regions, as these areas pose risks. In contrast, unmemorized parts of the image, which do not present risks, should be completely ignored.

Building on this localization insight, we introduce a new task: extracting localized memorized regions. This task is crucial for better understanding local memorization and developing targeted strategies to reduce the associated performance gap. To accomplish this task effectively and efficiently, we introduce the concept of the ‘bright ending’ (BE), a phenomenon observed in the cross-attention maps of text-to-image diffusion models. The bright ending occurs when the end token in the final inference step exhibits abnormally high attention scores on specific image patches in memorized generations compared to non-memorized ones. This distinctive pattern effectively highlights the regions where the model has memorized the training data, requiring only a single inference pass and without relying on access to the training data. To our knowledge, this is the first method to achieve such precise and efficient localization, emphasizing the significance of BE.
Finally, we propose a simple yet effective method to integrate BE and its extracted local memorization masks into existing state-of-the-art memorization evaluation, detection, and mitigation strategies. Our extensive experiments demonstrate that this integration significantly enhances the performance of these existing tasks by narrowing the gap caused by local memorization, further underscoring the contribution of BE.

In summary, our contributions are three-fold:
1) We analyze the performance gap between local and global memorization in existing tasks and introduce a novel localized view of memorization in diffusion models.
2) We pioneer a new task of locating localized memorization regions and propose the novel ‘bright ending’ (BE) phenomenon, which is the first method to successfully achieve this task using only a single inference process without requiring access to training data and conducting a nearest neighbor search over all ground truth images.
3) By integrating the local memorization mask extracted using BE into existing evaluation, detection, and mitigation strategies, we refine these approaches to focus on local perspectives, in contrast to the global focus of all existing strategies. This refinement narrows the gap caused by local memorization and sets a new state-of-the-art in these existing tasks. This simple and effective incorporation further demonstrates the power of BE.

Figure: Figure 1: Memorization in diffusion models can occur both globally (right) and locally (left). The first row displays the memorized training image, the second row shows the corresponding generated image, and the third row highlights the local memorization mask extracted using ‘bright ending’.
Refer to caption: x1.png

## 2 Preliminaries

### 2.1 Diffusion Models

Diffusion models (Rombach et al., ; Ho et al., ; Nichol & Dhariwal, ) represent a state-of-the-art class of generative models, emerging as the predominant choice for various generation tasks.
They entail a two-stage mechanism, beginning with a forward process that incrementally introduces Gaussian noise to an image originating from a real-data distribution $x_{0}\sim q(x)$ across $T$ timesteps, culminating in $x_{T}\sim\mathcal{N}(0,\mathbf{I})$:

$$ $q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{ I})$ (1) $$

$$ $q(x_{1:T}|x_{0})=\prod_{t=1}^{T}q(x_{t}|x_{t-1}),$ (2) $$

where $x_{t}$ represents the version of $x_{0}$ with added noise at timestep $t$, and $\{\beta_{t}\}_{t=1}^{T}$ is the noise schedule controls the amount of noise injected into the data at each step.
By using the reparameterization trick (Kingma & Welling, ), we can derive the Diffusion Kernel, allowing for the sampling of $x_{t}$ at any given timestep $t$ in a closed form:

$$ $q(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_ {t})\mathbf{I})$ (3) $$

where $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.
Subsequently, in the reverse process, generative modeling is accomplished by training a denoiser network $p_{\theta}$ to closely estimate the true distribution of $q(x_{t-1}|x_{t},x_{0})$:

$$ $p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t}),\sigma_{ \theta}^{2}(x_{t})\mathbf{I})$ (4) $$

$$ $p_{\theta}(x_{T:0})=p(x_{T})\prod_{t=T}^{1}p_{\theta}(x_{t-1}|x_{t})$ (5) $$

where $\mu_{\theta}(x_{t})$ and $\sigma_{\theta}^{2}(x_{t})$ are the approximated mean and variance of $q(x_{t-1}|x_{t},x_{0})$, whose true values can be derived using Bayes rule and the Diffusion Kernel as in Eq. 3.
The learning objective simplifies to having the denoiser network $\epsilon_{\theta}$ predict the noise $\epsilon_{t}$ instead of the image $x_{t-1}$ at any arbitrary step $t$:

$$ $\mathcal{L}=\mathbb{E}_{t\in[1,T],\epsilon\sim\mathcal{N}(0,\mathbf{I})}[\left \|\epsilon_{t}-\epsilon_{\theta}(x_{t},t)\right\|_{2}^{2}]$ (6) $$

where the denoiser network can be easily reformulated as a conditional generative model $\epsilon_{\theta}(x_{t},y,t)$ by incorporating additional class or text conditioning $y$.

## 3 A localized view of memorization in diffusion models

Fig. 1 illustrates that diffusion models can exhibit both global and local memorization. Global memorization refers to the model remembering the entire image, whereas local memorization pertains to the model retaining only a portion of the image. Notably, global memorization can be seen as a special case of local memorization when the memorized region encompasses the whole image.

Figure: Figure 2: Left: Variations in non-memorized regions significantly impact similarity scores with existing global metrics. Right: Local memorization is more challenging to detect than global memorization, with more failure cases observed using the current state-of-the-art detection method.
Refer to caption: x2.png

#### Existing global metrics fall short for local memorization evaluation.

We observe that all existing studies on memorization in diffusion models are from a global perspective, where the effectiveness of any proposed detection methods and mitigation strategies are evaluated using global metrics, such as L2 distance or Self-Supervised Copy Detection (SSCD) (Pizzi et al., ). These metrics measure similarity between entire generated images and training images rather than focusing on locally memorized areas.
However, these metrics are not robust to variations outside the locally memorized areas and can be easily tricked. A generated image may memorize only a local area of the training image while making the remaining areas novel and diverse, leading to false negatives. Fig. 2 (left) illustrates failures of the global metric approach, where variations in non-memorized regions significantly reduce the similarity scores measured by SSCD. The current standard considers SSCD scores greater than 0.5 as memorization. This results in some images being falsely identified as non-memorized despite all being locally memorized, with differences only in the non-memorized regions.

Figure: Figure 3: Local memorization tends to have smaller magnitudes than global memorization, making it harder to distinguish from non-memorization. Incorporating BE into the existing detection method effectively increases the magnitude of local cases while keeping the other cases mostly unchanged.
Refer to caption: x3.png

#### Existing global strategies struggle with local memorization detection and mitigation.

Current detection and mitigation strategies are evaluated using global metrics and are thus developed from a global perspective. For instance, Wen et al. () distinguishes memorized cases from unmemorized ones by utilizing the magnitude over the global latent space (Eq. 7), with memorized cases exhibiting larger magnitudes. This detection method has outperformed its predecessors, setting the current state-of-the-art.
However, this approach is less effective for local memorization cases than for global ones, as variations in non-memorized regions impact magnitude computations, increasing variance and diminishing the reliability of the detection signal, which leads to more false negatives.
To verify this theory, we visualized the density plot of magnitudes for all three scenarios (global, local, and no memorization) in Fig. 2 (right) and Fig. 3 and observed that local cases indeed have smaller magnitudes than global ones, making them closer in magnitude to unmemorized cases and harder to detect.
Regarding mitigation, local memorization also underperforms. Wen et al. () employs prompt engineering when a positive detection is made, optimizing the prompt using global magnitude as the loss function, which is, therefore, also sensitive to variations in non-memorized regions. Additionally, the compromised global detection method, which produces more false negatives, often leaves the mitigation strategy untriggered, increasing the risk of unaddressed memorization and potentially leading to legal challenges.

$$ $d=\frac{1}{T}\sum_{t=1}^{T}\left\|\left(\varepsilon_{\theta}(x_{t},e_{p})- \varepsilon_{\theta}(x_{t},e_{\phi})\right)\right\|_{2}\vspace{-0.2cm}$ (7) $$

#### The localization insight: local memorization is a more generalized and practical notion of memorization.

Recognizing the gap in research on measuring, detecting, and mitigating local memorization, we propose that further investigations should adopt a local perspective. Local memorization is a more meaningful and generalized concept for the following reasons: (1) Unmemorized regions pose no litigation risk and can be disregarded, while even a small locally memorized area can present significant legal concerns. (2) Local memorization is a more encompassing definition, with global memorization being a specific instance. Focusing on local memorization does not diminish the value of global investigations; instead, it complements and extends the scope of existing work.

## 4 Bright ending

#### Introducing a new task: extracting localized memorized regions.

The observed performance gap caused by local memorization has prompted us to explore a new task: extracting localized memorized regions as a mask. This task is essential for investigating memorization from a local perspective, and the resulting mask can be integrated into existing evaluation, detection, and mitigation strategies. This integration refines these strategies into localized approaches, effectively reducing the performance gap associated with local memorization.
One naive method to create such a local memorization mask is to compare generated images with the closest training image. However, this reliance on training data presents limitations: (1) It compromises privacy for a task aimed at preventing the memorization of training data and assumes that the user of the pre-trained model has access to the training data; (2) It requires significantly more computation due to the need for nearest neighbor searches on the large LAION training dataset. Consequently, this naive approach diminishes its practical significance when the mask is used in detection and mitigation algorithms. Therefore, we aim to extract such masks without using training data, relying solely on the pre-trained model’s memory to identify distinguishing memorization patterns.

Recent work has highlighted the important role of prompts in causing memorization. For example, Somepalli et al. () identified that duplicated prompts during the training of text-to-image Stable Diffusion models make the prompts become the “keys” to the models’ memory. Similarly, Wen et al. () observed that prompts prone to memorization tend to result in larger text-conditional noise predictions during inference than non-memorization cases. Additionally, Chen et al. () successfully utilized classifier-free guidance to steer generations away from memorized prompts, improving mitigation.
Inspired by these findings, we focused on using prompts to extract our desired local mask. This naturally led us to explore the cross-attention mechanism in the U-Net architecture within diffusion models, which links prompts to specific attention-focusing areas in the generated images.
One of our key intuitions is that the pre-trained diffusion model’s memories resulting from overfitting should differ from those learned through generalization. Memorized generations are inflexible and follow a fixed denoising trajectory that is observably different from non-memorized scenarios. This is evidenced by Wen et al. (), who observed that in memorized cases, the magnitude of text-conditional noise prediction during the denoising trajectory is abnormally higher than in non-memorization cases.

Figure: Figure 4: Visualization of cross-attention maps during the final denoising step in pre-trained Stable Diffusion models. Typically, the end token shows a dark cross-attention map, shifting the denoiser’s attention from semantic meanings to fine details. However, in memorized models, ‘bright ending’ anomaly occurs, where the end token displays abnormally high cross-attention scores, focusing on coarser structures, specifically, the local memorized regions, effectively serving as an efficient automatic extraction of the local memorization mask without needing access to the training data.
Refer to caption: x4.png

Figure: Figure 5: Left: Box plot showing the distribution of end token cross-attention scores during the final inference step for different memorization types. The plot is based on 9,600 generated images, with 16 images generated for 300 memorized and 300 non-memorized prompts each. It clearly distinguishes the three types: non-memorized cases have attention scores close to zero, while memorized cases exhibit abnormally high scores. Global memorization shows higher scores than local memorization due to the larger memorized regions. This further validates the ‘bright ending’ observation. Right: Violin plots displaying the impact on magnitudes for different memorization types.
Refer to caption: x5.png

#### Automatic local memorization mask extraction via bright ending (BE).

Inspired by our intuition, we visualized each token’s cross-attention maps in memorized and non-memorized prompts to identify distinguishing patterns (Fig. 4).
We discovered an interesting phenomenon in pre-trained text-to-image Stable Diffusion models: during the final denoising step, the end token typically shows a dark cross-attention map.
Specifically, these dark maps often contain only extremely low-attention finer edges or random scatter points.
Since the end token summarizes all semantic meanings in the prompt, this indicates that when the noised image is very close to a clean image, the denoiser’s attention on text-conditioning shifts from focusing on its semantic meanings to concentrating only on fine details.

However, when the model exhibits memorization, we observe an anomaly where the end token displays abnormally high cross-attention scores at the end of denoising steps — a phenomenon we refer to as the ‘bright ending’ (BE).
Specifically, it focuses on the coarser structure of image patches rather than finer details or random scatter points.
This indicates that the denoiser remains focused on text-conditioning even during the final inference step, demonstrating overfitting to the text-conditioning. This aligns with the finding in Somepalli et al. () that memorized generations result from overfitting to duplicated caption-conditioning during training, thus following a fixed noise trajectory during inference.
Thus, BE is a robust indicator for differentiating memorized generations from unmemorized ones, with bright regions identifying the memorized areas. This effectively functions as an automatic memorization mask, acting as a global mask during global memorization and a local mask during local memorization, as can be visually observed in Fig. 4 and 1.
To further validate this observation, we used 300 memorized prompts and 300 non-memorized prompts to generate 16 images for each prompt, resulting in a total of 9600 generations. We then employed a box plot (Fig. 5) to visualize the distribution of the attention scores of the end token during the final inference step for these generated images. We observed a clear distinction among the three memorization types: non-memorized cases consistently had attention scores close to zero, while memorized cases exhibited abnormally high scores. Furthermore, global memorization showed higher scores than local memorization due to the larger regions being memorized in global cases compared to local ones.

Beyond BE’s effectiveness in performing the new task of extracting localized memorized regions, it is also remarkably efficient, requiring only a single inference pass without needing access to the training data. To our knowledge, BE is the first method to achieve such precise and efficient localization.

## 5 Localized detection, mitigating and evaluation methods

### 5.1 Incorporating the bright ending (BE)

BE’s success in the new task of extracting localized memorized regions can also benefit existing tasks of evaluating, detecting, and mitigating memorization in diffusion models. By integrating the extracted local mask into current strategies, these approaches can adopt a local perspective, thereby reducing the performance gap associated with local memorization.

#### Detection strategy.

As discussed in Sec. 3, computing magnitudes in the global space, as shown in Eq. 7, can be misleading during local memorization.
To ensure that the magnitude computation corresponds solely to the locally memorized area and remains invariant to other areas, we propose element-wise multiplication of the magnitude by our memorization mask extracted via BE:

$$ $d=\frac{1}{T}\sum_{t=1}^{T}\left\|\left(\varepsilon_{\theta}(x_{t},e_{p})- \varepsilon_{\theta}(x_{t},e_{\phi})\right)\circ\mathbf{m}\right\|_{2}\bigg{/} \left(\frac{1}{N}\sum_{i=1}^{N}m_{i}\right)$ (8) $$

Here, $N$ is the number of elements in the mask $\mathbf{m}$.
Unlike methods requiring hyperparameter tuning, such as setting thresholds to convert attention scores into a binary mask, our approach uses the attention scores directly as weights. This reweights the relative importance of each pixel in the magnitude computations. To ensure comparability, the result is normalized by the mean of the attention weights $\mathbf{m}$.

#### Mitigation strategy.

Also, as discussed in Sec. 3, the baseline mitigation strategy employs prompt engineering when a positive detection outcome is identified, using global magnitude as the loss function for such prompt optimization.
Our proposed BE introduces two key enhancements to this process: (1) Accurate trigger: The mitigation strategy is triggered more accurately due to our improved detection method, which utilizes masked local magnitude by incorporating BE. (2) Improved loss function: By using masked local magnitude as the loss function, we provide more effective gradient information during backpropagation for prompt optimization, further enhancing the efficacy of the mitigation strategy, again thanks to the incorporation of BE.

#### Evaluation strategy.

To verify the effectiveness of the bright ending when used for evaluating metrics, we design a localized similarity metric, denoted as LS, to measure the similarity between generated image $\hat{x}$ and training image $x$. This metric is a masked version of the L2 distance and is combined with the standard SSCD metric, as defined in Eq. 9. We expect this approach to outperform the sole use of SSCD and the combination of SSCD with the original global L2 distance (denoted as S), as in Eq. 10.

$$ $LS(\hat{x},x)=-\mathbbm{1}_{\text{SSCD}>0.5}\cdot\left\|(\hat{x}-x)\circ \mathbf{m}\right\|_{2}$ (9) $$

$$ $S(\hat{x},x)=-\mathbbm{1}_{\text{SSCD}>0.5}\cdot\left\|(\hat{x}-x)\right\|_{2}$ (10) $$

### 5.2 Experiments

#### Setup.

Following previous works, we conducted experiments on Stable Diffusion v1-4. We adhered to the baseline prompt dataset (Wen et al., ), using 500 prompts each from Lexica, LAION, COCO, and random captions as non-memorized prompts. We used the dataset organized by Webster () for memorized prompts. However, since not all 500 prompts in Webster ()’s dataset are prone to memorization, we selected 300 memorized prompts for our experiments. For each memorized and non-memorized prompt, we generated 16 images.

We follow the baseline methodology from Wen et al. () and evaluate performance using the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, the True Positive Rate at 1% False Positive Rate (T@1%F), and the F1 score, and report the detection performance during the 1, 10, and 50 inference steps.
However, the baseline detection performance measures whether a prompt is prone to memorization rather than if a specific generation is memorized. They experimented with how well the average magnitude of 1, 4, and 32 generations per prompt differentiates between memorized and non-memorized cases. This approach is flawed because using different random seeds, a prompt prone to memorization can sometimes generate non-memorized images. Consequently, a prompt that rarely yields memorization might still be classified as memorized. For example, generating four non-memorized images from a prompt classified as memorized due to its occasional memorized output can falsely inflate performance metrics. Conversely, failing to classify these as non-memorized when they are all non-memorized should not be penalized.
To address this, we assess detection performance based on accurately predicting each generation as memorized or not rather than each prompt. We re-implement the baseline method under the same conditions for a fair comparison.
For mitigation, the output utility is evaluated based on the CLIP score, which measures the text-alignment of generated outputs. We evaluate similarity scores using both traditional SSCD and our proposed localized metric to provide a comprehensive performance view, experimenting with multiple privacy levels.
Due to current metrics’ failure to accurately identify memorized cases, we manually label the ground truth for each generated image from the memorized prompts. Webster () organizes the nearest training images for these generations, making it easy to observe local memorization cases with minimal subjectivity in manual labeling. We compare different criteria’s F1 scores, using a threshold of 0.50 for SSCD to determine memorized cases and a threshold of -50 for LS and S metrics as defined in Eq. 10 and Eq. 9.
We separately report performance for local and global memorization in Tab. 1 and Fig. 7 and 7, highlighting how our localized strategies, after incorporating BE, effectively reduce the performance gap specifically caused by local memorization.

Implementationally, we experimented with the cross-attention maps on different layers of the U-Net and found that averaging the first two 64-pixel downsampling layers most effectively extracts the local memorization mask. The inference process takes about 2 seconds per generation using RTX4090.

**Table 1: Detection performance for local and global memorization at different inference steps.**
|  | First Step | First 10 Steps | All Steps |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AUC | F1 | T@1%F | AUC | F1 | T@1%F | AUC | F1 | T@1%F |
| Baseline - Local | 0.918 | 0.864 | 0.629 | 0.989 | 0.982 | 0.953 | 0.990 | 0.983 | 0.560 |
| Ours - Local | 0.943 | 0.893 | 0.731 | 0.995 | 0.987 | 0.985 | 0.996 | 0.988 | 0.926 |
| Baseline - Global | 0.979 | 0.944 | 0.934 | 1.000 | 0.987 | 1.000 | 0.999 | 0.976 | 1.000 |
| Ours - Global | 0.981 | 0.948 | 0.940 | 1.000 | 0.987 | 1.000 | 0.999 | 0.977 | 1.000 |

Figure: Figure 6: Local memorization’s mitigation.
Refer to caption: x6.png

#### Results.

Tab. 1 demonstrates that incorporating the memorization mask into magnitude computation effectively enhances detection performance across all metrics and various inference steps for local memorization while resulting in almost identical results for global memorization.
Additionally, Fig. 3 and 5 (Right) further demonstrate this targeted improvement in local memorization. The magnitudes for non-memorization and global memorization remain almost unchanged, but the magnitudes for local memorization are successfully amplified, making them more distinguishable from non-memorization cases. This effectively reduces the performance gap caused by local memorization by converting previous False Negative cases into True Positives (examples shown in Fig. 8).

For mitigation, we also include the original Stable Diffusion (SD) without any mitigation strategies applied in Fig. 7 and 7 for comparison, illustrating the extent to which we have reduced the local memorization performance gap. Specifically, for local memorization, under the high-utility scenario where we aim to minimally reduce the CLIP score while mitigating memorization, our approach achieves a significantly better trade-off than the baseline. We reduce SD’s similarity score by twice as much as the baseline while simultaneously observing a much smaller reduction in the CLIP score - less than half compared to the baseline. We also tested different privacy levels and consistently achieved a better trade-off for local memorization. Specifically, we observe a lower similarity score for the same CLIP score, as measured by both SSCD and our localized metric. Additionally, we consistently observe a higher CLIP score for the same similarity score. As expected, the results for global memorization remain almost identical to the baseline.

It is worth noting that the effectiveness of these strategies not only demonstrates our improvement over the current state-of-the-art in existing tasks after incorporating BE’s local memorization mask, but also serves as an extrinsic validation of BE’s effectiveness in the new task we propose - extracting local memorization masks. To our knowledge, BE is the first method to precisely extract local memorization masks in a highly efficient manner, requiring only one inference step without the need to query the training data.

Figure: Figure 8: Examples of converting FN into TP. Generated image followed by training image.
Refer to caption: x8.png

## 6 Related work

#### Memorization in Diffusion Models

Numerous studies have previously explored memorization phenomena in language models (Carlini et al., ; Kandpal et al., ; ; Lee et al., ) and GANs (Tinsley et al., ; Arora et al., ; Heusel et al., ).
Over the past year, memorization within diffusion models has increasingly been scrutinized.
Somepalli et al. () and Carlini et al. () concurrently pioneered the examination of this issue, discovering that pretrained text-conditional models, such as Stable Diffusion, frequently replicate their training data. Similar findings were observed for unconditional generations in pretrained DDPMs on CIFAR-10 (Krizhevsky, ), as well as DDPMs trained on smaller datasets like CelebA (Liu et al., ) and Oxford Flowers (Nilsback & Zisserman, ). Additionally, Chen et al. () explored class-conditional DDPMs on CIFAR-10, further highlighting the significance of this issue.

#### Types of Memorization

Webster () categorizes memorized captions into three types: (1) Matching Verbatim (MV), where the generated image using a memorized caption is an exact replica of the corresponding image; (2) Retrieval Verbatim (RV), where the generated image using a memorized caption replicates other training images rather than the corresponding one; and (3) Template Verbatim (TV), where the generated image using a memorized caption replicates only the template shared by a subset of training images, with variations in fixed locations outside the template, corresponding to different captions.
This categorization focuses on the causes of memorization: MV results from overfitting in the one-to-one mapping between image and caption in the training set, while many-to-many mappings lead to RV and TV. However, the concepts of Local Memorization (LM) and Global Memorization (GM) differ, focusing on whether the generated image exactly replicates an entire training image or just a portion of it, regardless of whether the training image matches the prompt’s corresponding image or the causes of memorization. RV and TV can be seen as subsets of LM, as there are additional causes of LM beyond the “many-to-many mapping overfitting” that remain unidentified. For instance, in Fig. 8, the image in the third column is classified as MV by Webster () due to the one-to-one mapping of the training image-caption, rather than many-to-many. However, it only memorizes a local portion of the image in the last column. This suggests that a pre-trained model may occasionally exhibit creativity when recalling overfitted memories, which can also lead to local memorization.

#### Detection and Mitigation Strategies

Carlini et al. () detects memorization by calculating the Euclidean distance between generated images and training images, recommending re-training on de-duplicated data as a mitigation strategy.Daras et al. () suggests mitigating memorization by training models on corrupted data.
Somepalli et al. () proposes to mitigate memorization by injecting randomness into text prompts, such as adding random tokens or noise, to break the link between memorized prompts and their corresponding images.
Chen et al. () employs a nearest neighbor search, similar to Carlini et al. (), to identify potential memorized outputs and then selectively applies guidance methods to instruct these generations away from the memorized training images, demonstrating outstanding mitigation performance.
Recently, Wen et al. () proposed an efficient method for detecting memorization during the inference phase of diffusion models. This approach uses the magnitude of text-conditional predictions for prompt detection, enabling high accuracy from the initial inference step, and triggers a mitigation strategy to optimize the prompt embedding when memorization is detected. Both the proposed detection and mitigation strategies have achieved state-of-the-art results.
A most recent work (Ren et al., ) utilized cross-attention patterns to design detection and mitigation strategies, achieving performance comparable to Wen et al. (). The key insight is that the cross-attention scores for memorized prompts are less concentrated across all tokens in a prompt during multiple inference steps. This lack of concentration is quantified using entropy, which captures the distribution of cross-attention at the prompt-level.
While both leverage cross-attention, it is important to note that our bright ending cross-attention (BE) differs from the cross-attention entropy approach regarding objectives, insights, inputs, and the final computed statistic.
Specifically, different from Ren et al. ()’s objective of leveraging cross-attention for designing detection and mitigation strategies, BE also aims to achieve a new task of precisely and efficiently locating localized memorization regions. Also, BE does not rely on the concentration patterns of all tokens during multiple steps as the insight and inputs but instead uses the raw final step attention score of the end token. Additionally, BE employs the raw image-patch level distribution rather than the prompt-level distribution as the computed statistic, serving as a local mask to highlight areas of local memorization.

## 7 Conclusion and limitations

#### Conclusion.

In this paper, we identified a performance gap in existing tasks of measuring, detecting, and mitigating memorization in diffusion models, particularly in cases where the generated images memorize only parts of the training images.
To address this, we introduced a new localized view of memorization in diffusion models, distinguishing between global and local memorization, with the latter being a more meaningful and generalized concept.
To facilitate investigation from a local perspective, we pioneered a new task: extracting localized memorized regions. We proposed leveraging the newly observed ‘bright ending’ (BE) phenomenon to precisely and efficiently accomplish this task, resulting in a local memorization mask.
Through extensive experiments, we demonstrated that the mask generated by this new task can also be used to enhance performance in existing tasks. Specifically, we proposed a simple yet effective strategy to incorporate the BE local memorization mask into existing frameworks, refining them to adopt a local perspective. This approach reduces the performance gap associated with local memorization and achieves new state-of-the-art results.
The improved results not only highlight our improvements in existing tasks but also serve as extrinsic validation of BE’s effectiveness in the new task of local memorization mask extraction, underscoring the significant contribution of BE.

#### Limitations and future work.

One limitation of using the memorization mask extracted through the bright ending phenomenon in tasks like detection and mitigation is that it can only be observed at the end of the inference process. This requirement results in a longer processing time than existing strategies when applied to existing tasks such as detection and mitigation. However, we believe the additional time is justified by its significant positive contributions to investigating local memorization in diffusion models. Moreover, each inference takes only a few seconds on a consumer-grade GPU, making the process practical for real-world applications.
While existing tasks are well-studied with established evaluation criteria, future work could explore how to evaluate the performance of the newly proposed local memorization region extraction task. Our current approach uses an extrinsic evaluation method that demonstrates improvements by incorporating the mask into existing strategies. Future research could also consider an intrinsic evaluation by directly comparing the extracted mask with ground truth masks.
Another future direction could involve developing more advanced evaluation metrics for local memorization. In our work, we employed a straightforward approach by combining a masked version of the L2 distance with the widely adopted SSCD to compute similarity. Future research could focus on creating methods similar to SSCD, trained in a self-supervised manner but specifically tailored for more accurate localized copy detection.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. AFAIK there's no dedicated EOS token used by stable diffusion. Instead, there are only 77-N padding tokens which are almost identical to each other (except for being added with different positional encodings). Suppose the prompt contains 15 tokens, then there will be 77-15=62 padding tokens which have identical token embeddings. I believe the cross-attention maps of these tokens are highly similar to each other. Do you compute the cross-attention map by averaging these 77-N attention maps? Or do you choose the last padding token to extract the cross-attention map?

EDIT after author response:
Apparently I got it wrong. There's indeed an EOS token inserted before padding tokens.
soundness: 2
strengths: 1. The proposed idea is simple and is a natural extension of Wen et al. (2024).
2. The occurrence of "local memorization" has important practical values but is under-explored.
summary: This paper is a followup work of Wen et al. (2024), which discovered interesting patterns that "abnormally high predicted noise magnitude" indicate "global memorization". This work finds similar patterns between "abnormally high cross-attention values" of the EOS token in the prompt with the image tokens indicate "local memorization". The experiments validate such patterns can indeed be used to find local memorization.
weaknesses: 1. In multiple places, the authors state that the "bright ending" is obvious at the final denoising step. However, in the experimental results presented in Table 1, the 3 designs were using "First Step", "First 10 Steps", and "All Steps", respectively. Only "All steps" include "the final denoising step", but apparently "the final denoising step" plays only a minor role in "All steps". Why this discrepancy?
2. In section 5.1, "we propose element-wise multiplication of the magnitude by our memorization mask extracted via BE". However, the authors didn't give details about how to extract a memorization mask from BE in Section 4. Is this done using a fixed threshold? a dynamic threshold? Or a threshold predicted by a dedicated NN?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. What is the motivation for incorporating SSCD similarity into the metric? Why not just use some function of the mask itself to generate a score?
2. Could you please share more details about the manual labeling process and the dataset itself?
3. Will you share the code implementing your method?
soundness: 3
strengths: 1. The problem presented by the authors is compelling, and they demonstrate failure cases of existing techniques in the literature
2. The authors propose a novel and effective method to address the problem
3. The proposed method seems to solve this problem while also maintaining the success cases of earlier techniques
summary: This paper presents a problem with existing techniques to detect memorization in diffusion models in their lack to detect instances of memorization localized to specific subsections of generated images. To resolve this problem, they propose a new technique that creates a memorization mask for a generation.  The technique is based on a correlation between high attention values and memorization that the authors observe at the end of the diffusion process.
weaknesses: 1. Some of the figures and text are somewhat difficult to understand, especially relating to the method itself. For example, Table 1 and Figs 6/7 are not explained thoroughly enough, even though they seem to be central results for the paper. Section 5.1 could also benefit from further elaboration to ensure that the method and its formulation are clear.
2. The evaluation is not well explained in my opinion. For example, more details about the manual labelling process will make the results more reproducible.
3. Sharing the code and the dataset will help making the method more reproducible

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: How does the proposed BE method perform in scenarios where memorization is more subtle or occurs in highly abstract image regions? Is there a threshold of memorization size or intensity where BE becomes less effective?

Is it possible for the method to be generalized to other types of generative models? Can the BE-based method be generalized to generative models that do not rely on text conditioning, such as unconditional image generation models? Can it be applied to video generation?

Given the increased inference time due to analysis at the final denoising step, do the authors have suggestions for optimizing the process to make it more efficient for real-time applications?
soundness: 3
strengths: The paper introduces the concept of "bright ending" as a novel phenomenon in diffusion models, which has not been explored in prior research. This unique observation enables a new task of localized memorization detection, which broadens the scope of understanding memorization in generative models.

The paper is good written. The paper clearly differentiates between global and local memorization, offering detailed explanations of how BE is used for local region detection. Figures and visualizations of attention maps provide clarity on how the BE mechanism operates in practice, enhancing understanding of the proposed method.

The motivation of the paper is sound. It has implications for legal and ethical considerations, especially in cases where models might unintentionally replicate copyrighted content. The findings are particularly relevant for improving privacy-preserving techniques in model training and inference.
summary: This paper introduces a novel perspective on memorization in text-to-image diffusion models by focusing on local memorization, where specific regions of a training image are memorized rather than the entire image. It identifies a unique "bright ending" (BE) anomaly in cross-attention patterns, where memorized regions show higher attention to the end token during the final denoising step. This insight enables the extraction of localized memorization regions without access to training data. The paper proposes a new task—locating these localized memorization regions—and integrates this approach with existing evaluation, detection, and mitigation strategies. The paper demonstrate that incorporating BE significantly improves performance in existing tasks, particularly in narrowing the gap caused by local memorization, achieving state-of-the-art results.
weaknesses: The efficiency of the proposed approach needs to be further demonstrated. The BE-based approach requires analysis at the end of the inference process, making it slower compared to some existing global methods that might detect memorization earlier in the process. 

While the BE phenomenon is well-documented for text-to-image diffusion models, it is not clear how well this method would generalize to other types of diffusion models, like recently the flow-matching based models.

More evaluation metrics for local memorization could be explored for a comprehensive study.

Meta-review: The submission received the ratings of four reviewers, which recommended 8, 8 and 6, averaging 7.33. Given the plenty of competitive submissions in ICLR, this stands at a score above the acceptance. The reviewers' concerns mainly focus on some writing or presentation issues, which have been well addressed after the rebuttal. Therefore, I tend to recommend acceptance towards the current submission.

Decision: Accept (Spotlight)