Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
Authors: Hyeonggeun Han
Sehwan Kim
Hyungjun Joo
Sangwoo Hong
Jungwoo Lee
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion models
Memorization
Primary area: deep_learning
TL;DR: We propose a novel method to mitigate memorization in text-to-image diffusion models by adjusting the initial noise.

Abstract: Despite their impressive generative capabilities, text-to-image diffusion models often memorize and replicate training data, prompting serious concerns over privacy and copyright. Recent work has attributed this memorization to an attraction basin—a region where applying classifier-free guidance (CFG) steers the denoising trajectory toward memorized outputs—and has proposed deferring CFG application until the denoising trajectory escapes this basin. However, such delays often result in non-memorized images that are poorly aligned with the input prompts, highlighting the need to promote earlier escape so that CFG can be applied sooner in the denoising process. In this work, we show that the initial noise sample plays a crucial role in determining when this escape occurs. We empirically observe that different initial samples lead to varying escape times. Building on this insight, we propose two mitigation strategies that adjust the initial noise—either collectively or individually—to find and utilize initial samples that encourage earlier basin escape. These approaches significantly reduce memorization while preserving image-text alignment.

Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models
Hyeonggeun Han 1.2- Sehwan Kim 1- Hyungjun Joo
Sangwoo H Jungwoo Lee 12,41
ECE & "NextQuantum, Scoul National University
SCSE. Konkuk University
Hoddo AI Labs
(hygnhan, sehwankim, joohj911. 911, junglee)@snu.ace
swhong@6@konkuk.eeg kr.
Abstract
Despite their impressive generative capabilities, text-to-image diffusion models often memorize and replicate training data, prompting serious concerns over privacy and copyright. Recent work has attributed this memorization to an attraction basin-a region where applying classifier-free guidance (CFG) steers the denoising trajectory toward memorized outputs-aa and has proposed deferring CFG application until the denoising trajectory escapes this basin. However, such delays often result non-memorized images that are poorly aligned with the input prompts, highlighting the need to promote earlier escape that CFG can be applied sooner in the denoising process. In this work, we show that the initial noise sample plays crucial role in determining when this escape occurs. We empirically observe that different initial samples lead to varying escape times. Building on this insight, we propose two mitigation strategies that adjust the initial eiisereithr collectively individually- to find and utilize initial samples that encourage earlier basin escape. These approaches significantly reduce memorization while preserving image- text alignment. Code is available at https: httpggtttthh..teggg com wwwoo/hyggan//////e hrrpphhhnniiittnoeee http:/innnttntootooo noise.. dif wfffffsonnnmeee memorizztionn
Introduction
Text-to-image immage diffusion models have garnered significant attention for their remarkable ability to generate high-quality images that are semantically aligned with textual prompts. Despite their success, growing body of work has revealed that these models are prone to memorizing and reproducing content from their training data, sometimes generating images that are nearly identical to those seen during training [39, 5, 40]. This memorization behavior poses serious concerns, including potential leakage of privacy-sensitivy or copyrighted material [4, 23]. In addition to the legal and ethical risks, such behavior undermines the utility of diffusion models by limiting their generative diversity and reducing their capacity to synthesize truly novel content. These concerns highlight the pressing need understand and mitigate memorization in diffusion models to ensure their safe and effective mentoyment.
A range of approaches have been proposed to mitigate memorization in diffusion models [49, 19, 34, 40]. Among them, inference- mitigation strategies have gained particular attention due to their computational efficiency and practicality for real-world deployment. Several studies have linked
'These authors contributed equally to this work Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

memorization to specific prompt-image associations, motivating mitigation methods that perturb prompts during inference [40, 49]. Other researches have focused on internal model behaviors, particularly within cross-attention layers, revealing that memorization can be attributed to sharply focused attention maps [34] or small set of specific neurons [19]. Building on these insights, mitigation techniques such as attention logit rescaling [34] and neuron deactivation [19] have been introduced.
A recent line of work has identified classifier-free guidance (CFG) [20] in the early stages of denoising as a key factor contributing to memorization in diffusion models [22]. To explain this phenomenon, Jain et al. [22] introduced the concept of an aftraction basin- a region in the sample-time cemple-ttmeeeeee space where applying CFG consistently drives the generation process toward memorized outputs, regardless of the random initialization. To mitigate this effect, the authors proposed deferring the use of CFG until the denoising trajectory exits the attraction basin. However, they also highlighted trade-off: applying CFG too late can lead to non-memorized images that are poorly aligned with the conditioning prompt. To address this issue, they emphasized the need to escape the attraction basin earlier in the generation process. Toward that end, the authors introduced opposite guidance, strategy that applies inverted conditional signals during early denoising steps to push the trajectory away from the attraction basin, thereby enabling earlier and safer application of CFG.
In this paper, we propose a novel approach to mitigating memorization by enabling an earlier escape from the attraction basin. In contrast to prior efforts that modify the denoising trajectory through adjustments guidance signals, we instead focus on the starting point of the trajectory. Our empirical findings reveal that different initial Gaussian noise samples result in significantly different escape times from the attraction basin. This observation motivates the hypothesis that certain initializations naturally lie closer to the basin's 's boundary, allowing the generation process to evade steering force more quickly. Building on this insight, we propose two
the memorization-indueee inference-time mitigation strategies that find and utilize favorable initializations to promote early basin escape. Specifically, we propose a batch-wise method, which adjusts initial samples collectively, and per-sample method, which adapts each initialization individually. Our approach offers a novel perspective that diverges from the conventional wisdom [49, 22] that the choice of initialization has minimal impact memorization mitigation.
The key contributions of this paper are summarized as follows:
We offer a novel perspective on the role of the initial noise sample in memorization mitigation, employing the concept of the attraction basin a region in which classifier- free guidance (CFG) steers the denoising trajectory toward memorized outputs. We show that different initializations lead to varying escape times from this basin.
We propose two inference-time mitigation strategies traatgies-Batch--iiss Batch-wise and Per-sample approaches that find and leverage initial samples likely escape the attraction basin earlier. These methods adjust the initial samples to produce non-memorized images while promoting better alignment with the conditioning prompt.
Our empirical evaluations demonstrate that the proposed initial sample adjustment strategies produce more text-aligned, non-memorion outputs than those without adjustment, and outperform existing inference-time mitigation baselines in reducing memorization.
2 Background
2.1 Diffusion models
Diffusion models [21, 42] aim to generate samples from a target data distribution g(z). During training, a data sample xn ~ q(x:) is progressively corrupted by Gaussian noise over T timesteps such that 2T ~ (O. At each timestep I, the forward noising process is defined as:
q( =a(x;: v1 8,x.-1,
(1)
where 3, denotes the variance schedule. The noise predictor is trained to predict the noise added at each timestep. At inference time, the sampling process begins from FT (O, I), and a sample generated by iteratively denoising via the reverse process:
1 Ctt
ae co (see 1)
St Tt-1
Ft
VI

---

where 1- 3e and ā = II.=11 Ag.
Text-to-image diffusion models such Stable Diffusion [35] usually utilize CFG steer the generation process toward a desired conditioning input y. Given a null condition Ynull representing the unconditional case, CFG modifies the noise prediction at each timestep as:
CTG (x,,,, co c0 L, Ymal)) + (eo v) - cØ &, Uman)),
where WCFG controls the strength of the guidance. For notational convenience, we denote the conditional noise prediction ( i, y) eæ( t, Moull) as ég ( y). It is worth noting that the trained noise predictor Eo approximates the score function at each timestep [20, 43]: co t,y) Re -V1 d+ Vz, z, log (xaa y) and co t, Unull) = -V1 Oe Vz, log po(s)..
2.2 Attraction basin
The attraction basin refers to region formed in the denoising trajectory where applying CFG ex-
Training image [780, 1J 1680, 1500.
No guidance
INSIGIRG
erts strong steering force toward memorized out-
puts. To mitigate memorization, Jain et al. [22] proposed deferring the application of CFG until the trajectory escapes this basin. A key aspect of this strategy identifying when this escape occurs. They observed that when sampling proceeds with- out CFG, the magnitude of the conditional noise prediction ( t, y)lla remains high within the basin. This is interpreted as empirical evidence of the memmorzation-induein force present in this region. Just before the escape, the magnitude drops sharply-the this timestep is referred to as the tran- Fig-
16
(x,, fx)
12
10
8.
6
Within attraction basin
2 1000
600
400
260
Transition point Time steps
sition point. This behavior illustrated ure 1. Applying CFG before the transition point (i.e., within the attraction basin) typically results in memorized outputs, whereas applying it after the transition point leads to non-memorized images. However, excessively delaying CFG avoid mem- orization may result in poor alignment with the conditioning prompt. This trade-off underscores the importance of encouraging earlier escape from the basin-i.e., shifting the transition point to an earlier timestep-to generate non-memorized images that remain faithful to the prompt.
Figure 1: The magnitude of éa (z t, y) at each timestep during sampling without CFG for memorized prompt. Each image above corre- sponds to an output generated when CFG is applied over the timestep interval indicated by the associated square brackets.
3 Initial sample adjustment for mitigating memorization
3.1 Motivation
To effectively mitigate memorization, CFG should be applied after the denoising trajectory escapes the attraction basin-i.e. after the transition point. This motivates the need to shift the transition point to an earlier timestep. To this end, we investigate the influence of the initial Gaussian sample Jr, which defines the starting point of the trajectory. As shown Figure 2, we consistently observe across different memorized prompts that varying Ir results in different transition points. This suggests that the location of IT, relative the attraction basin, influences how quickly the trajectory escapes. Based on this observation, we conjecture that initial samples closer to the basin' 's boundary facilitate earlier escapes and thus earlier transition points. To validate this conjecture, we require proxy for estimating sample's proximity to the basin's 's boundary. Prior work [22] reports that the magnitude of the conditional noise prediction, [1, t, y)ll2, remains elevated within the basin and drops sharply upon exit. Leveraging this insight, we hypothesize that initial samples with smaller magnitudes ||Eo (xT, T, 1) are more likely to escape the basin earlier. To test this hypothesis, we introduce novel method that adjusts xr to obtain a modified sample ã7 with reduced magnitude of conditional guidance [āā, T. y)ll2. We then analyze the magnitude of the conditional noise predictions throughout the denoising process.
3

---

|| c,yllaa
20
|| y.lilb m 5
15-
6
10
2-
5
1000
800
600
400
200
9000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
(a) Passion. Podcast. Profit."
"DC All Stars podcast"
(c) "Insights with Laura Powers"
Figure 2: The magnitude of the conditional noise prediction each timestep during sampling without CFG for three memorized prompts. Each line color corresponds to a different initial Gaussian sample. Transition points occur at different timesteps depending the choice of initial sample.
3.1.1 Initial sample adjustment using sharpness
To facilitate clear analysis of the impact of [ 112,, we introduce a one-step adjustment method that modifies the initial sample ær using tunable step size that controls the strength of the magnitude reduction. the magnitude of the conditional noise prediction. According to prior works [20, 11J, the conditional noise prediction can be approximated in terms of the score function:
this end, we adopt the sharpness definition from Foret al. [16] to reduce
Co( 1, -1 αz Vn, log (y)lar))
Letting L(xx) po (yfare), obtain ||āa(Te, 4, 1 e v1 V. L(xx)ll, indicating that reducing the magnitude of conditional guidance corresponds to minimizing the gradient norm of L(x,). To achieve this, adopt the following sharpness measure:
Lsharp
max L(ra + 8) L(x.).
1011.SP
This definition captures the worst-case local increase L within an b--ball of radius centered and penalizing this sharpness term has been shown to correspond to gradient norm minimization [16, 53]. Based on this formulation, we obtain the adjusted initial sample 7r. which yields the reduced magnitude of conditional guidance, as follows:
iT øT way Lisharp (XT),
(6)
where y is step size hyperparameter. To compute V ey Laharp (IT), follow Foret al. [16] and first approximate the inner maximization Equation (5) using a first-order Taylor expansion to obtain
8 (IT)
TV argmax L(z + 8) argmax ((TT) 8 Vay L(IT) argmax 8 Vzy L(xT). (7)
|
||
Since this approximation can be seen a classical dual norm problem, its solution 8(2r) is given by:
Vay L(rr)
8(тT)
(8)
V LaTT)ll
By substituting ((FT) from Equation (8) into the sharpness definition in Equation (5), and differenti- ating with respect to IT, we obtain the update rule Equation (6) as follows:
कт = V (I(xT 8(xT)) L(xT)) 2J FT 7( V ZT [( + Vay [(Tr))
Here, we follow Foret al. [16] and approximate the gradient by omitting second-order terms. Substituting METT) ~ रा- ( T, v), we express the final adjustment rule in terms of the noise predictor:
ëт RJ Fr (( (xT + 8(xT), T, y) ( T, v)),
(10)
where 8(xT) =
Ea(wy (wy. and 7 ---ar Both ÿ and are treated as hyperparameters.
[To( T.

---

"The Happy Scientist"
18
||E,(x, tyllll
15
y- 0.5
Baseline
1lE 1.0x...y. yilla
16-
14
0.5
Baseline
IlE,0x,t yill
20
y 0.5
Baselline
20
y= 1.0
12
10
F 1.0
15
P 1.0
p- 1.5
1.5
p 1.5
10
1000
800
800
400
200
1000
800
600
400
200
1000
800
600
400
200
steps
Time steps
Time steps
"The No Limits Business Woman Podcast"
||E, t.ylll
12
0.5
10
0=
Baseline
t.
14
12
7 0.5
Baselline
1.0
9=1.0
10
9 1.0
y=
1.5
y=1.5
p 1.5
1000
800
600
400
1000
600
400
200
1000
880
400
200
Time steps
Time steps
Time steps
Figure 3: Plots showing the magnitude of the conditional noise prediction at each timestep during sampling without CFG under different adjustment strengths y for two memorized prompts. Each row corresponds to different memorized prompt, and each column corresponds to different initial Gaussian sample TT. The baseline corresponds
3.1.2 Magnitude analysis of conditional guidance for adjusted initial samples
In order to test our hypothesis, we use the adjusted initial sample ä obtained via Equation (10) and observe the f2-norm of the conditional noise prediction at each timestep when CFG not applied. We employ Stable Diffusion [35] with p 50 and vary the adjustment strength 5 E {0, 0.5, 1.0, 1.5) to investigate the influence of the magnitude of conditional noise prediction at the initial timestep T. Figure 3 presents the results across different memorized prompts and initial samples. Each row corresponds to different prompt, while each column shows the effect of applying the proposed adjustment different initial sample. The baseline denotes the unadjusted case (? 0). First, observe that the magnitude of the conditional noise prediction at the initial timestep decreases as increases. This confirms that the proposed adjustment method successfully adjusts ør yield an updated sample žт with reduced magnitude [āor T, 1),)),, as intended. Second-and more notably-we find that the transition point consistently occurs earlier in the denoising process increases. This trend holds across different prompts and initial samples, although the exact degree of the shift varies. These results support our hypothesis that initial samples with smaller magnitudes of conditional guidance ( T,, y) llz are more likely to escape the attraction basin earlier, thereby advancing the transition point.
3.2 Proposed mitigation methods
Motivated by the above observations, we propose two mitigation approaches: Batch-wise mitigation and Per-sample mitigation. Batch-wise mitigation adjusts initial samples ær collectively across batch. This method reduces the magnitude of the conditional noise prediction using small number of additional function evaluations (NFEs), producing adjusted samples रт with minimal computational overhead. contrast, Per-sample mitigation modifies each sample ær individually. It reduces the magnitude of conditional noise prediction through direct backpropagation. continuing the update until the transition point is eliminated for each generation process. As result, this method does not require an additional hyperparameter preeeffiineeeeeeeeeeeeeeeee predefined timestep at which to begin applying CFG.
3.2.1 Batch-wise mitigation
For Batch-wiss mitigation, we adopt the adjustment strategy introduced in the previous section. Given a batch of initial samples æT, we apply the update rule Equation (10) to the entire batch simultaneously. This update is repeated M times, where M is tunable hyperparameter that determines the number of adjustment steps. The resulting adjusted samples ë7 are then used as initial samples for the image generation process. Additionally, following Jain et al. [22], apply
5

---

Algorithm 1: Batch-wise mitigation
Input: adjustment strength 7, sharpness parameter p, number of adjustments M, CFG application start timestep r
Output: generated images æ0
Draw batch of initial noise samples øт (0,
/* Adjust the initial samples dr with small number of additional NFEs */
for i to M do 6x( T. 8(xT) = T 2
är = 2T (Gs(xr + + ((x)), T,y) Co(2r,,,10 T,)))
27 ËT
end
/+ Perform the denoising process using the adjusted initial samples */
for : = T to do
if then
C (I),,,)) c0(xx, t, Ynult)
else
CCG (It, t y)
cœ( t, Ymun) + (. eo (2 Moull)
end
Et-1
1 Vas
It
vT-ai
1-a; CPG (xe,
end
CFG after certain timestep to avoid generating memorized outputs. The complete procedure for Batch-wise mitigation is outlined in Algorithm
3.2.2 Per-sample mitigation
For Per-sample mitigation, adjust each initial sample 2T individually by directly minimizing [[r T, 122 via backpropagation. The adjustment continues until [[² y)ll2 falls below predefined target loss harget- In contrast to Batch-wise mitigation, CFG is applied from the beginning of the sampling process. As shown in the rightmost plot of the bottom row in Figure 3, when the magnitude is sufficiently minimized (e.g.. 7 1.5), the transition point effectively disappears. This suggests that early application of CFG no longer induces memorization, thereby eliminating the need predefine specific timestep at which to begin applying CFG. Consequently. Per-sample mitigation enables the generation of memoozedd images even when CFG is applied from the initial timestep T. We conjecture that this behavior arises because sufficiently strong minimization yields initial sample that lies outside the attraction basin. The pseudocode for Per-sample mitigation is provided in Appendix A.
4 Experiments
4.1 Experimental setup
Diffusion models and datasets. In line with prior works on memorization in diffusion models [40, 49, 34, 19, 8], we evaluate our proposed methods using Stable Diffusion v1.4 [35]. Although Stable Diffusion v2.0 exhibits fewer memorization issues due to being trained on a de-duplicated dataset, still suffers from template memorization
form of memorization in which a set of prompts leads to images that are highly similar to training samples, differing mainly in aspects such as color or style [34]. To account for this, we also employ Stable Diffusion v2.0 in our evaluations, For Stable Diffusion v1.4, we use 500 memorized prompts extracted from the LAION dataset [38], and for Stable Diffusion v2.0, we use 219 memorized prompts. These prompts are provided by Webster [47] and Ren et al. [34]. respectively.

---

Evaluation metrics. We employ four complementary metrics to evaluate memorization, image-text aligmment, image quality, and image diversity. To assess the degree of memorization generated images, we use the Self-Supervised Copy Detection (SSCD) score [32], which measures object- level similarity between generated image and nearest neighbor in the training set. Image-text alignment is assessed with the CLIP score [33], which quantifies the semantic consistency between each generated image and its corresponding text prompt. For image quality and prompt alignment, we adopt ImwgeReward [50], while image diversity is evaluated using LPIPS [52]. Lower SSCD scores indicate stronger mitigation of memorization, whereas higher CLIP, LPIPS, and ImageReward scores reflect better performance. Detailed evaluations of image quality and diversity presented in Appendix D.
Baselines. We compare our proposed methods against six baselines. The first is no-mitigation baseline, where images are generated using Stable Diffusion without applying any memorization mitigation. Additionally, tokens [40], adjusting the prompt embedding [49], scaling cross-attention logits [34], applying opposite guidance during the early stages of sampling [22], and applying guidance only after specified timestep [22]. All baseline methods are applied at inference time and do not require access the training data, making them suitable for fair comparison with our proposed approaches.
consider five inference-time mitigation approaches: adding random
Implementation details. For each diffusion model, we generate 10 images per memorized prompt using identical inference configurations across all baselines and our proposed methods. Specifically, for Stable Diffusion 1..4, we employ DDIM sampler [41] with 50 sampling steps and CFG scale of 7, whereas for Stable Diffusion v2.0, we use Euler discrete scheduler [26] with the same number of steps and CFG scale. Further implementation details are provided in Appendix B.
4.2 Impact of initial sample adjustment on memorization mitigation
To examine the effectiveness of our proposed ini- tial sample adjustment in mitigating memorization while promoting image-text alignment, we compare mitigation results obtained with and without adjust- ment. The adjustment is implemented using the Batch-wise method. For the unadjusted case, the starting timestep for applying CFG, denoted as т. is varied
0.28
No adjustment
With adjustment
0.26
0.24
0.22
0.20
{780, 760, 740, 720], while for the adjusted case it is set to r E (900, 880, 860, 840). This design ensures that we can isolate the impact of the adjustment itself, given that r influences mit- igation performance. The results are illustrated in Figure 4. On the No adjustment curve, the rightmost and leftmost data points correspond to r = 780 and т = 720, respectively; for the With adjustment curve, they correspond to r = 900 and r = 840. Despite the earlier application of CFG (i.e.. higher r). the adjusted cases consistently achieve lower SSCD scores than their unadjusted counterparts at comparable CLIP scores, indicating that the proposed adjustment effectively shifts the transition point earlier in the denoising process and enables memorization mitigation even when CFG is applied early. More importantly, the adjustment yields a more favorable trade-off between SSCD score and CLIP score. These results demonstrate that our initial sample adjustment effectively reduces memorization while encouraging image-text alignment.
0.18
0.235
0.240
0.245
0.250
0.255 0.260
CLIP score
Comparison of SSCD and CLIP scores with and without initial sample adjust- ment.
Figure 4:
4.3 Comparison with baselines
In this subsection, we compare our proposed methods with baseline approaches by analyzing how SSCD and CLIP scores vary across different mitigation strengths, following prior works [34, 49]. We evaluate both the baselines and our Per-sample method under five distinct hyperparameter settings. is worth noting that mitigation strength in our methods is closely tied to the value of |Eo (z0, T, y) l-- To control this value, the Batch-wise method employs three hyperparameters (5: p. and M), whereas the Per-sample method uses only one (lunge). Since both approaches share the core idea that adjusting the initial noise Ir to minimize ||ão (IT; T, 1) improves mitigation performance, we report the performance of Per sample across multiple larges values. Detailed hyperparameter configurations
7

---

No Mitigation
Cross attention Scaling
Ours (Batch-wiss
Random Taken Additioon
Opposite Guidance with Dynamic Transition Paint
Ours (Per sample)
Prompt Embedding Adjustment
Static Transition Point
SD v1.4
SD v2.0
0.30
0.5
0.28
0.26
0.4
SSCD
0.24
0.3
0.22
0.20
0.2
0.200
0.225
0,250 0.275 0.300
0.22
0.24
0.26
CLIP
CLIP score
Figure 5: Comparison of SSCD and CLIP scores among different mitigation methods under Stable Diffusion v1.4 and v2.0. Lower SSCD scores indicate stronger memorization mitigation, while higher CLIP scores indicate better image-text alignment.
N'CNSTII
Training
Somepulli et al. [40]
Wen et al. [49]
Ren et al. [34]
Jain et al. [22]
Ours (Batch-wise)
Ours (Per-sample)
Figure 6: Qualitative comparison of memorization mitigation results. Each column shows generations produced by various baseline methods and our proposed approaches. The prompts used for the generations are provided in Appendix G.
for all methods are provided in Appendix B. The results in Figure 5 show that Batch-wise achieves performance comparable to Cross-attention Scaling under Stable Diffusion v1.4, while achieving the best performance under v2.0. More importantly. Per-sample consistently provides the most favorable ASCD-CLIN trade-off across both diffusion models, highlighting its superior effectiveness in mitigating memorization. Qualitative comparisons presented in Figure 6.
performance gap observed between the Batch-wise and Per-sample methods under Stable Diffusion vl.4. We conjecture that this difference arises because the optimal hyperparameters for

---

the Batch-wise method vary across prompis and initial samples. As shown in Figure 3, fixed set of hyperparameters can lead to different degrees of magnitude reduction in Eg(IT. T, y), resulting in varying transition points depending on the prompt and the initial sample. Moreover, since CFG must be applied after-but not far beyond-thh the transition point, the optimal timestep for CFG application is inherently sample-specific in the Batch-wise setting. In contrast, the Per-sample method directly minimizes the magnitude of éø (IT,T), y) via backpropagation with fixed hyperparameters, effectively eliminating the transition point for each individual generation. This enables robust memorization mitigation while improving image-texi alignment, as demonstrated in Figure 5. Although the Batch- wise approach is less adaptive and slightly underperforms relative to the Per-sample method, it still achieves competitive performance.
Related work
Memorization in diffusion models. Beyond supervised models [4] and language models [3], diffusion models [5, 39, 40] have also been found to exhibit a memorization problem, indicating that models tend to memorize and regenerate training data. This phenomenon raises significant privacy concerns, as it can result unintended data leakage. To address this issue, various mitigation strategies have been explored. While early approaches sought to mitigate memorization either during training or at inference, training-tima mitigation typically requires access to the original dataset and substantial computational resources, limiting its practicality. Consequently, recent efforts have shifted toward inference-tine mitigation strategies [7, 22, 19, 8, 34J. For example, Wen et al. [49] proposes adjusting prompt embeddings to minimize the magnitude of the conditional noise prediction, while Chen et al. [8] introduces attention masking to enhance this adjustment. Somepalli et al. [40] mitigates memorization by perturbing prompts through token addition or replacement, and Jain et al. [22] suggests applying opposite guidance during early denoising steps to counteract memorization induced by CFG. Our work follows this line of research, proposing inference-ting mitigation strategies that adjust initial noise samples to reduce memorization.
Impact of initial noise samples. Recent studies have increasingly focused on the impact of initial noise samples on image generation in diffusion models, exploring how modifying these samples can address various challenges, such improving image-text alignment and enhancing image quality [51, 31, 17, 44, 46, 6, 37, 9]. For example, Wallace et al. [45] proposes an end-to-end framework that backpropagates through the entire diffusion process to adjust the initial noise sample, thereby improving classifier guidance. Guo et al. [18] utilizes attention maps to guide the refinement of initial noise samples, aiming to mitigate semantic errors such subject neglect, subject mixing, and incorrect binding. Similarly, Eyring et al. [15] addresses poor semantic fidelity in text-to-image models by optimizing the initial noise samples based on reward feedback.
Sharpness aware minimization. The sharpness term used to minimize the magnitude of the conditional noise prediction in our method was proposed in Sharpness- Aware Minimization (SAM) [16]. SAM was originally designed to seek flatter minima in the loss landscape, motivated by several studies showing that flatter minima are associated with better model generalization [27, 25, 14]. By simultaneously minimizing both the loss value and the sharpness of the loss surface, SAM aims to find model parameters that reside in regions where the loss values remain consistently low. SAM has achieved -fte-of-thhearrt generalization performance, resulting in numerous follow-up works that aim to improve it by introducing surrogate loss function [54]. proposing geometric measures of neighborhood [29, 28], or enhancing training efficiency [12, 13, 24]. However, a deep understanding of which components of SAM drive its generalization improvement remains limited. As a result, there have been significant efforts to analyze the underlying mechanisms of SAM [1, 48, 10, 53]. For instance, Zhao et al. [53] reveals that SAM effectively minimizes the gradient norm during optimization.
In contrast to previous works, to the best of our knowledge, this paper is the first investigate the impact of the initial noise sample on mitigating memorization in diffusion models. By introducing a novel method that adjusts the initial sample by minimizing the magnitude of the conditional noise prediction, we analyze the relationship between the magnitude of conditional guidance and the transition point. Building on this analysis, we propose two novel mitigation strategies that effectively reduce memorization while promoting image-text alignment.

---

Limitations
While our proposed methods demonstrate strong effectiveness, they also have certain limitations. The Per-sample method introduces a slight increase in computational overhead, and the performance of the Batch-wise method can vary with hyperparameter selection. However, believe that the additional cost of the Per-sample method is well justified by its effectiveness in mitigating memorization and enhancing privacy and intellectual property protection. Moreover, the close relationship between the hyperparameters of the Batch-wise method and the goal of reducing the magnitude of conditional guidance offers a clear and efficient direction for hyperparameter tuning.
7 Conclusion
We present a novel perspective on the role of initial samples in mitigating memorization in diffusion models, viewed through the lens of the attraction basin. We observe that different initializations result in different transition points. Given that earlier transition points lead to non non-memorized images that remain well-aligned with text prompts, further analyze which initial samples facilitate faster escape from the attraction basin and find that adjusting initial noise samples to achieve a smaller magnitude of the conditional noise prediction results in earlier transition points. Building on these insights, we propose two novel inference-time mitigation Beattegies-Bacc---ee sample wpproaches-which adjust initial samples collectively individually and use them for image generation. Experimental results validate our findings and demonstrate the effectiveness of the proposed strategies in mitigating memorization while improving image-text alignment.
and Per-
Acknowledgements
This work is in part supported by the National Research Foundation Korea (NRF, RS-2024- 0045 00451435(20%). ogy Planning & Evaluation (ITTP, RS- 2222-122222222 1068
00004-00041355(200) )) Institute of Information & communications Technol- RS- RRS-0025-022 002305453 RS-2025- 80(10%) RS- 10--2221-1D3 (33((10))) grant funded by the Ministry Science and ICT (MSIT), Institute of New Media and Communica- tions(INMAC), and the BK21 FOUR program of the Education, Artificial Intelligence Graduate School Program (Seoul National University), and Research Program for Future ICT Pioneers, Seoul National University in 2025.
02273157(10%),
25442 24442149911000 (10%),
References
[1] Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International conference on machine learning, pages 639-668. PMLR, 2022.
[2] Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. In International Conference on Machine Learning, pages 3462-3483. PMLR, 2024.
[3] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vone Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633-2650. 2021.
[4] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35:13263-1327, 2022.
[5] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[6] Changgu Chen, Libing Yang, Xiaoyan Yang. Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Find: Fine-tuning initial noise distribution with policy optimization for diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9735-67444 2024.
10

---

[7] Chen Chen, Daochang Liu, and Chang Xu. Towards memorization free diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8425 8834, 2024.
[8] Chen Chen, Daochang Liu, Mubarak Shah, and Chang Xu. Exploring local memorization in diffusion models via bright ending attention. In The Thirteenth International Conference Learning Representations, 2025. URL https htttps:///ppt http//ppeeeviieeeeeee net for fow.ffodoopccccoc id=p4cLtzk4oe.
[9] Sherry X Chen, Yaron Vaxman, Elad Ben Baruch, David Asulin, Aviad Moreshet, Kuo-Chin Lien, Misha Sra, and Pradeep Sen. Tino-edit: Timestep and noise optimization for robust diffusion-based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6337-6346, 2024.
[10] Zixiang Chen, Junkai Zhang, Yiwen Kou, Xiangning Chen, Cho-Jui Hsieh, and Quanquan Gu. Why does sharpness aware minimization generalize better than sgd? Advances in neural information processing systems, 36:72325-72376, 2023.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-878 8794, 2021.
[12] Jiawei Du, Hanshu Yan, Jiashi Feng. Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-awars minimization for improved training of neural networks. In International Conference on Learning Representations, 2022. URL https://openreeiiww
[13] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. Advances in Neural Information Processing Systems, 35:23439-253 23451,, 2022.
[14] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 1703v:1703. 700. 2017.
[15] Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata, Reno: Enhancing one-step rext-to-image models through reward-based noise optimization. Advances in Neural Information Processing Systems, 37: 111112248-1255111 2024.
[16] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awan aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021. URL https:://operreviee. net/ wtttt//foommmmmmmmmmmme LrM.
[17] Paul Grimal, Hervé Borgne, Olivier Ferret, and Julien Tourille. Tiam-a metric for evaluating aligmment in text-to-image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2890-2899, 2024.
[18] Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, and Di Huang. Initno: Boosting text-to-image diffusion models via initial noise optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9380-9389, 2024.
[19] Dominik Hintersdorf, Lukas Struppek, Kristian Kersting, Adam Dziedzic, and Franziska Boenisch. Finding nemo: Localizing neurons responsible for memorization in diffusion models. Advances in Neural Information Processing Systems, 37:88236-888 88278, 2024.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.122 12598, 2022.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[22] Anubhav Jain, Yuya Kobayashi, Takashi Shibuya, Yuhta Takida, Nasir Memon, Julian Togelius, and Yuki Mitsufuji. Classifier-free guidance inside the attraction basin may cause memorization. arXiv preprint arXiv:2411.16738 2024.

---

[23] Harry H Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. Ai and its impact on artists. In Proceedings of the 2023 AAAT/ACM Conference on AI, Ethics, and Society, pages 363-374, 2023.
[24] Weisen Jiang, Hansi Yang. Yu Zhang, and James Kwok. An adaptive policy to employ sharpness aware minimization. The Eleventh International Conference on Learning Representations, 2023. URL https::/t//o/ /ttp:://ppeeeeeeee ropeereview.o net /tr//oorumm rommm fffrummdddwwwW.. 1d- MMMWLL7-MBC.....
[25] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.00 2019.
[26] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
[27] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint
2016.
[28) Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pages 11148-11161. PMLR, 2022.
[29] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905-5914. PMLR, 2021.
[30] Uyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL Bkg6RiCqY7.
[31] Jiafeng Mao. Xueting Wang, and Kiyoharu Aizawa. Guided image synthesis via initial image editing in diffusion model. In Proceedings of the 31st ACM International Conference Multimedia, pages
2023.
[32] Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14532 14542, 2022.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 8748 8748-8763. PMLR, 18 24 Jul 2021. URL https: http::///ooceeeeeg/s/me mlr. prwss/ BBtSSss/v33////// http://ddfooddaaaaaea html.
[34] Jie Ren, Yaxin Li, Shenglai Zeng, Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and mitigating memorization in text-to-image diffusion models through cross attention. In European Conference on Computer Vision, pages 340-356. Springer, 2024.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[36] Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation. Advances in Neural Information Processing Systems, 36:57863-57877 2023.
[37] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4695-4703, 2024.
12

---

[38] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25294 2022.
[39] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein, Diffu- sion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023.
[40] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-4788 2023.
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL httpss///pperreviee..
net/forum net/forumiid-Sttne www.rrnnndddttiiaaii......
[42] Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. Advances in neural information processing systems,
2020.
[43] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https://openrreview. net/forumm1dd=PXTIG122
[44] Wenqiang Sun, Teng Li, Zehong Lin, and Jun Zhang. aware latent initialization for controllable image generation. arXiv preprint arXiv:2401.161551 2024.
[45] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik. End-to-end diffusion latent optimization improves classifier guidance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7280-7290, 2023.
[46] Ruoyu Wang, Huayang Huang, Ye Zhu, Olga Russakovsky, and Yu Wu. The silent prompt: Initial noise as implicit guidance for goal-driven image generation. arXiv preprint arXiv:2412.05101, 2024.
[47] Ryan Webster. A reproducible extraction of training images from diffusion models, 2023. URL https://arxiv.org// /2005. 2385.08694.
[48] Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How sharpness-aware minimization minimizes sharpness? In The Eleventh International Conference Learning Representations, 2023. URL https://openrevieww net wtt.:/orrrrrooooee dosrrmmmdddssspppppppxx
[49] Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. The Twelfth International Conference on Learning Representations, 2024. URL https::/ //pen//////rrvieww www.:nee/ffooo hett:/orrmm.d--B4n33Wttttt
[50] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances Neurul Information Processing Systems, 36: 1990--15935, 2023.
[51] Katherine Xu, Lingzhi Zhang, and Jianbo Shi. Good seed makes good crop: Discovering secret seeds text-to-image diffusion models. 2025 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3034. IEBE, 2025.
[52] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[53] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization deep learning. In International conference on machine learning, pages 26982-26992. PMLR. 2022.
13

---

[54] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, sekhar tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations, 2022. URL https://operreiiee.eete//oore wttp://wwww.....o...o..oooeeeaa.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly outline the contributions of our paper in both the abstract and introduction sections. Specifically, our contributions are itemized in the final paragraph of the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We elucidate the limitations of our method the Limitations section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
This paper does not present any theoretical results; no theorems or formal
assumptions are introduced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are outlined in Section 4 (Experiments) and further elaborated in Appendix A (Implementation details) and Appendix B (Pseudocode for Per- sample mitigation).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
Depending the contribution, reproducibility
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We adhere to the NeurIPS code submission policy.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cttpcc/ppp///pccccccc /uwdddeeeeeeeuummiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are outlined in Section (Experiments) and Appendix A. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We follow the evaluation protocols established by existing works in this area. To highlight our results, report SSCD, CLIP score, and FID metries.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the emor bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). of errors not verified.
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report 1- --iggma error bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information regarding computer resources is provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tturii///eris..... http:ppp....ppp.pittttttttttttaa
Answer: [Yes]
Justification:
We fully comply with the code of cthics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We highlight data leakage issues caused by memorization text-to-image diffusion models in the Introduction and discuss how our proposed methods address these issues in the Conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairnsss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications. let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our research method and area are unrelated to such issues.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We illustrate the sources of assets in Section 4 (Experiments) and Appendix A (Implementation details).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pape rswirswite thcode. com datastss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not conduct experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The proposed method was developed without the involvement of any LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
for Please refer our LLM policy http:///eeerrissccoooeeeeeeeeeee what should or should not be described.
21

---

Appendix
A Pseudocode for Per-sample mitigation
We present the pseudocode for the Per-sample mitigation method in Figure 7.
def adj- latllatten(lateett t, prompt_embeds, lr, target_loss):
2
Latent. requireseeeeeeeee grad = True
3
optim Adaml (1aatent, Lr=lr)
while True:
pred_texx text unet(Latent, rmmees
loss norm text pred_uncond)
if loss target loss:
break
potim.zero-grady baskwrd))wrrd)) optim.step() return latent.detachl
10
def per_ wtrpsallle_mmmm mitigationnprompt.tt embeds,, afg_scale, lr, target_loss)
latent randn(
for i, t in enumerate(tiiesteps))
# Adjust the initial sample using backpropagation
16
if i == 8:
17
latent = adj-latent (latent, t, prompt_embeds, tr, target_loss) Perform the denoising process with the adjusted initial sample pred uncond, pred text unet(latent, t, embeds) noise_pred uned_uncond cfg_scale (pred_text pred_uncond) latent = scheduler. (noise- pred, latent)
return latent
Figure 7: Pseudocode illustrating the Per-sample mitigation method.
B Implementation details
All experiments were conducted using an NVIDIA A100 GPU. The implementation environment included Hugging Face 0.23.4, CUDA 12.0, Python 3.8.0, and PyTorch 2.3.1 with Torchvision 0.18.1. Detailed configurations for the proposed mitigation strategies and baselines are outlined below.
Batch-wise mitigation. For Stable Diffusion v1.4. the adjustment strength set to Ý 0.7, the sharpness parameter to p = 50.0, and the number of adjustments to M = 2. CFG is applied with guidance scale of 7.0 for timesteps S r, and disabled (i.e., set to 0.0) for т, where the CFG application start timestep is set to = 900. For Stable Diffusion v2.0, we set 7 = 0.7, p 30.0. M 4, and = 917.4490.
Per-sample mitigation. For Stable Diffusion v1.4, the initial noise sample is adjusted using the AdamW optimizer [30] with a learning rate 0.01, while all other hyperparameters remain at their default values. The target threshold is varied lurger (0.7, 0.9, 1.1, 1.3, 1.5). For Stable Diffusion v2.0, the initial noise sample is adjusted using the same optimizer with a learning rate 0.1, and the target threshold is varied Larget (5. 10, 15, 20).
Baselines. For Random Token Addition, the number of added tokens is set to {1, 4, 6, 8) for both diffusion models. For Prompt Embedding Adjustment, the target loss set to harget {3, 4, 5,5, 7} for Stable Diffusion v1.4 and harget E {50. 60, 70, 80, 90) for Stable Diffusion v2.0. For Cross- attention Scaling, the re-scaling factor is set to (1.15, 1.2, 1.25, 1.3, 1.35) for both diffusion models. For Opposite Guidance with Dynamic Transition Point, the opposite guidance scale is
22

---

set to 7. For Static Transition Point, the CFG application start timestep is set to T E (999.0, 978.6122, 958.2245, 907.9377..777 937.8367,917.44990 991..4490.
C Analysis of distributional shifts induced by initial noise adjustment
We observed that using initial noise with weaker conditional guidance leads to earlier transition point, thereby improving memorization mitigation performance. Consequently, our objective is to effectively reduce the norm []ēg( T, y) l2. A natural question arises: is it appropriate minimize this norm without an explicit regularization term to prevent the adjusted initial noise from becoming out-of-distribution (OOD)? We argue that the weight decay in the Adam optimizer partially fulfills this role. Furthermore, with the default weight decay value of w = 0.01, we observed no degradation in either mitigation performance image quality. The following subsections provide both theoretical and empirical evidence supporting this claim.
C.1 Effect of weight decay in mitigating distributional shift
According to prior works [2, 36], the L2 norm of the initial noise rr (O,, follows chi distribution:
|
Σ
ot x" 1|-T|11-1 - -11==|1/// /
(1I)
where d denotes the dimensionality of PT and I(-) is the Gamma function. Based on this, to prevent the adjusted initial noise from deviating into an OOD state, one could consider minimizing the objective
(0T, T, 9) log ((122)112)
(12)
where p represents the chi distribution and w is weighting coefficient. By substituting the chi distribution from Equation (11) into p(|[TT|e) in Equation (12), the objective can be rewritten as
| 12
| (ZT,T, 2l222 + + w(d log | 2|2 +
(13)
where e is a constant independent of FT- We argue that the weight decay term in the AdamW optimizer implicitly minimizes the final quadratic term, | thereby constraining adjusted initial 2.
samples from becoming OOD. To empirically validate this, we compared 5000 unadjusted and 5000 adjusted initial noise samples, measuring the Jensen Shannon divergence (JSD) between their distributions under varying weight decay values. The results shown in Table I demonstrate that as w increases, JSD decreases. This suggests that stronger weight decay encourages the adjusted samples to remain closer to the original noise distribution.
Table I:
Jensen-Shannon divergence between adjusted and unadjusted initial noise distributions under varying weight decay values.
0.01
w 0.05
w 0.2
JSD
0.0665
0.0663
0.0638
0.046
C.2 Effect of weight decay on mitigation performance
We varied the weight decay parameter w described in the previous subsection and evaluated the corresponding performance. The results are presented in Table 2. In this experiment, the target threshold was fixed lurget 0.9. Across all tested settings, the performance remained largely consistent regardless of the value of w, suggesting that the default weight decay value = 0.01 does not cause degradation in either mitigation performance or image quality. We attribute this stability to the fact that the default value does not cause the adjusted initial noise deviate significantly from the original distribution (O, I). This interpretation further supported by the JSD results in Table 1, 0.01 indicates minimal distributional shift.
where JSD value of 0.0665
23

---

Table Performance comparison across different weight decay values.
0.01
0.05
0.1
0.2
SSCD
0.2265
0.2265
0.2265
0.2275
CLIP
0.2647
0.2647
0.2647
0.2657
LPIPS
0.7615
0.7615
0.7615
0.7610
ImageReward
-0,3522
-0.3522
-0,3522
-0,3501
D Performance comparison on image quality and diversity
No Mitigation
Cross attention Scaling
Ours (Batch-wiss)
Random Token Addition
Opposite Guidance with Dynamic Transition Point
Ours (Per-sample)
Prompt pt Embedding Adjustment
Static Transition Point
SD v1.4
SD v2.0
0.5
0.28
0.26
0.4
SSCD
0.24
0.22
0.20
0.2
0.5
0.6
0.7
0.8
0.75
0.80 LPIPS
0.85
LPIPS
0.30
0.28
0.26
0.4
SSCD
0.24
0.3
0.22
0.20
0.2
0.18
-1.00 -0.5 -0.50 -025 0.00
-1.2
-1.0
00.8
-0.6
-0.4
ImageReward
ImageReward
Figure 8: Comparison different mitigation methods under Stable Diffusion v1.4 and v2.0. The first row shows SSCD) versus LPIPS, while the second row shows SSCD versus ImageReward. Lower SSCD scores indicate stronger memorization mitigation, whereas higher LPIPS and ImageReward values reflect greater image diversity and quality.
We compare our proposed methods with baseline approaches by analyzing how LPIPS and ImageRe- ward values vary across different mitigation strengths. To this end, we present two sets of results: SSCD versus LPIPS and SSCD versus ImageReward, using the same hyperparameters as in Figure 5, The results are shown in Figure 8. Since effective memorization mitigation enables the generation of non-memorized images, also promotes greater diversity in the outputs, as illustrated in the -SCD-LPIPS results in the first row. Under Stable Diffusion v1.4, the Batch-wise method achieves the highest LPIPS values at comparable SSCD scores, while Per-sample achieves the second- second-highest LPIPS values at lower SSCD scores. In contrast, under Stable Diffusion v2.0, Batch-wise yields the second-highest LPIPS values, whereas Per-sample attains the highest LPIPS values with substan- tial margin. The SSCD--ImaeRReaa results in the second row further show that the Per-sample approach provides the most favorable trade-off between memorization mitigation and image quality across both diffusion models. These findings demonstrate that our proposed methods introduce minimal quality degradation while substantially enhancing image diversity, thereby improving the practical utility of diffusion models.
24

---

E Magnitude of conditional noise prediction across varying adjustment strengths
"Insights with Laura Powers"
Baseline
IiE, t...t.y Lillla
Baseline
1|ealx, t.l. villa
HMW New 25 5n do
Baseline
y- 0.5
1
0.5.5
1.55
y- 1.0
y-1.0
y= 1.5
1=1.5
p- 1.5
1000
800
600
400
200
1000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
"There's s a <i>Mrs. Doubtfire</i> Sequel in the Works"
14
y-
Baseline
14
16
ς- 0.5
Baseline
IlE,tx, t. t/ll
Baseline
||E,/: 5.
122
10
10
12
1.5
1.5
1000
800
600
400
200
1000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
"Mothers influence her young hippo"
[|Ex cyllla
20
Baseline
IlEgfx. REalx..tyll Lyllla
15
Baseline
lleg0r, t.ylll, t.lllll
Baselline
15
0=0.5
0.5.5
15
y= 0.5
y= 1.0
10
== 1.0
g= L.0
10-
p- 1.5
7 1.5
p 1.5
1000
800
600
400
200
9000
800
600
400
200
1000
800
600
400
200
steps
Time steps
steps
Figure 9: Magnitude of (0 t, y) at each timestep during sampling without CFG under varying adjustment strengths 7. Each row corresponds to a different memorized prompt, and each column corresponds to a distinct initial noise sample.
We provide additional plots in Figure 9 illustrating the magnitude of conditional guidance each timestep during denoising without CFG, under varying adjustment strengths ÿ. The results show consistent trend: 7 increases, the magnitude of č (āT, T, y) decreases, leading an earlier occurrence of the transition point across different prompts and initial noise samples.
F Magnitude analysis of conditional guidance under Per-sample mitigation
In the Per-sample method, we adjust the initial noise sample øT by minimizing the magnitude of (oTr,TT y) until the transition point effectively disappears. To validate this effect, compare the magnitude of the conditional noise prediction with and without applying the Per-sample method. In these experiments, the results: each row corresponds to a different memorized prompt, and each column corresponds different initial noise sample Fr. The Per-sample method significantly reduces the magnitude of (ITT T' y). Moreover, unlike the baseline, no sharp drop in magnitude is observed, indicating that the transition point has been effectively eliminated. This enables CFG be applied from the beginning of the denoising process.
the target loss hurget 0.9 and learning rate of 0.01. Figure 10 presents
25

---

"The Happy Scientist"
35
20
25
30
Baseline Per-sample
1lEa(x, t. G.yllaa
15
per-sample
Baselline
11E,0r,. tyll
Per sample
Baseline
20
6
10
[|Eg
15
1000
800
800
400
200
1000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
"DC All Stars podcast"
25
15-
Per-sample
Baseline
1i .. yilla
12
10
16
Per-sample
Baseline
l| c.yl
20
15
Baseline Per-sample
||
10
9000
800
600
400
200
1000
800
640
400
200
Y000
800
600
400
200
Time steps
Time steps
Time steps
"The No Limits Business Woman Podcast"
16
[| t.yl
20
Per-sample
Baseline
ll t.1lll
12
Per-sample
Baseline
llEatxo 1.11
12
Per-sample
Baseline
10
10
10
9000
800
600
400
1000
600
400
1000
200
Time steps
Time steps
Time steps
"There's s <i>Mrs. Doubtfire</i> Sequel in the Works"
12
Baseline Per-sample
[| t. villa
15
Baseline sampa
to
Baseline Per sample
||,,K,,
0.00
10
1000
800
600
400
200
1000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
"Here's What You Need to Know About St. Vincent's Apple Music Radio Show"
Baseline Per-sample
| 2
Baseline
1E t. yl1l2
Baseline Per-sample
Per-sample
2 N un 01
3-
1000
800
600
400
200
1000
800
600
400
200
1000
800
600
400
200
Time steps
Time steps
Time steps
Figure 10: The magnitude of the conditional noise prediction at each timestep during sampling without CFG for five memorized prompts. The baseline corresponds to the case without adjustment, while the Per-sample method is applied with target loss lurget 0.9.
G Prompts for Figure 6
The following memorized prompts were used to generate the samples shown in Figure
Watch the Trailer for NBC's <i>Conttanine<</>
As Punisher Joins <i>Daredevil</i> Season Two, Who Will the New Villain Be?
Aero 50-975035BLU 50 Series 15x7 Inch Wheel, 5 on 5 Inch BP 3-1/2 BS
Signature Purple Ombre Sugar Skull and Rose Bedding
If Barbie Were The Face of The World's Most Famous Paintings
Plymouth Curtain Panel featuring Madelyn White Botanical Floral Large Scale by heatherdutton
26

---

H
Additional qualitative results
Training
Somepalli et al. [40]
Wen et al. [49]
Ren et al. [34]
Jain al. [22]
Ours (Batch-wise)
Ours (Per-sample)
Additional qualitative comparison between proposed methods and baseline approaches. Each column presents images produced by baseline or one of our methods, with the leftmost column showing the corresponding training images for reference.
Figure 11:
We present additional qualitative results comparing the baseline methods with our proposed ap- proaches in Figure 11. The memorized prompts used generate these samples are listed below:
Daniel Radcliffe Dons a Beard and Saggy Jeans in Trailer for BBC GTA Miniseries <i>The Gamechangerse/iz
Director Danny Boyle Is Headed To TV With FX Deal
30A27S4 Shaded Peach Brown Red/Strawberry Blonde Blend, Shaded w/ Dk Brown
Chris Messina In Talks to Star Alongside Ben Affleck in <i>Live By Night</i>
Air Conditioners & Parts
George R.R. Martin Donates $10,000 to Wolf Sanctuary for 13- Year-Old
27

---

I Computational cost analysis
Table 3: Computational cost of different mitigation methods.
gpu memory usage (MB)
inference time (sec)
No Mitigation
3564.94
1.85
Random Token Addition
3573.57
1.85
Prompt Embedding Adjustment
8115.53
1.98
Cross-attention Scaling
3170.38
1.93
Opposite Guidance with Dynamic Transition Point
3573.57
Ours (Batch-wise)
3573.57
2,01
Ours (Per-sample)
9701.62
2.8
We conducted an additional experiment to compare baseline mitigation methods with our proposed approaches terms of computational cost. For fair comparison, we measured GPU memory consumption and inference time per image. The results, presented in Table 3, show that the Batch- wise method introduces negligible computational overhead, while the Per-sample method requires relatively higher memory usage and longer inference time. However, as demonstrated in Figure 5, the Per-sample method consistently achieves superior memorization mitigation and robustness across different models. Given that effective mitigation is crucial for enhancing privacy and safeguarding intellectual property while reducing potential risks, we believe that the additional computational cost of the Per-sample approach is reasonable trade-off.
28


Reviews:
[{"review_id": "AXVlp5d7v5", "reviewer": "Reviewer_JH8S", "summary": "This paper introduces a novel, training-free approach to mitigate memorization in text-to-image diffusion models by focusing on the initial noise sample, $x_T$. The work is grounded in the \"attraction basin\" theory, which suggests memorization occurs when classifier-free guidance (CFG) steers the generation process towards a training example. The authors' core insight is that the choice of $x_T$ significantly impacts when the denoising trajectory escapes this basin. They empirically demonstrate a correlation between a lower initial conditional guidance magnitude and an earlier, more desirable escape time. Based on this, they propose two inference-time strategies to find such favorable initializations: a fast \"Batch-wise\" method that adjusts a batch of samples collectively using a few forward passes, and a more intensive \"Per-sample\" method that uses iterative backpropagation to optimize each $x_T$ individually until a target loss is met. Experiments on Stable Diffusion v1.4 show that these methods significantly reduce memorization while preserving strong image-text alignment.", "questions": "- **On the Per-sample Optimization Loop:** Could the authors clarify the details of the per-sample optimization. Specifically:\n\t- (1) What is the average runtime or number of iterations required for the optimization loop to converge?\n\t- (2) Could you justify the choice of AdamW and explain the effect of its weight decay? Simple gradient descent seems like a more natural choice due to the optimization problem being non-stochastic. The effect of weight decay on the noise latents is unclear. How does it affect the performance of the method? How does it compare to standard Adam or gradient descent in this context?\n\t- (3) The method does not regularize the latent $x_T$ to keep it in a high likelihood region of the initial Gaussian distribution, as previous methods have proposed [1, 2]. Have the authors observed any \"reward hacking\" behavior where the optimization pushes the latent to an out-of-distribution state to minimize the objective? For instance, what happens to the generated images if the optimization loop is allowed to run for a significantly longer duration? Is there a reason why this type of regularization is not needed here?\n- **On Decomposing Evaluation into Quality and Diversity:** The evaluation relies on global metrics like FID and CLIP score, which can make it difficult to disentangle the specific effects of the proposed methods. FID is a composite metric reflecting both image fidelity and diversity, the lack of clear correlation with CLIP score in Table 1 suggests these entangled aspects might be obscuring the true trade-offs.\n    - To provide a clearer picture, have the authors considered evaluating these two axes separately? For **diversity**, one could measure the average pairwise similarity (e.g., using LPIPS or DINO like in Dreambooth [3]) between images generated from the same prompt, which would directly quantify if the methods reduce variance. For **quality and prompt alignment**, a more direct reward model like ImageReward could offer a more nuanced signal than CLIP or FID alone.\n- **On Synergy with Other Methods:** Have the authors considered if this memorization mitigation technique could be combined with other noise optimization approaches that are designed to improve visual quality or prompt adherence?\n- **On the Generality of the Framework:** The proposed method is fundamentally tied to CFG. For a purely unconditional model, one might hypothesize that memorization corresponds to anomalies like an unusually high unconditional score norm ($||ϵ_θ(x_t, t)||_2$). Do the authors believe their core principle, adjusting $x_T$ to minimize a proxy signal, could be adapted to the unconditional case using this or a similar intrinsic signal?\n\n[1] Ben-Hanmu et al. \"D-Flow: Differentiating through Flows for Controlled Generation\". ICML 2024.\n\n[2] Eyring et al. \"ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization\". NeurIPS 2024.\n\n[3] Ruiz et al. \"DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation\". CVPR 2023.", "limitations": "The proposed framework is specifically designed for diffusion models that use Classifier-Free Guidance, which makes it not directly applicable to unconditional diffusion models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n- **A Novel Perspective on the Role of Initial Noise (Originality/Quality):** The paper's primary contribution is its novel perspective on the impact of the initial noise sample on memorization. The empirical observation that different initializations lead to varying escape times from the \"attraction basin\" is an important insight in itself, separate from the mitigation methods built upon it.\n- **Practical and Effective Mitigation Methods (Originality/Signifcance):** Building on this core insight, the paper proposes two effective methods that cater to different needs. The \"Batch-wise\" approach is computationally cheap and highly practical, while the \"Per-sample\" method, though more intensive, achieves stronger results, providing a valuable trade-off for users. The per-sample method is elegantly formulated by defining an objective based on a single model evaluation at the initial timestep formulating an efficient test-time optimization problem.\n\n**Weaknesses:**\n- **Missing Computational Cost Analysis:** A significant weakness is the lack of a comparative computational cost analysis (e.g., in wall-clock time, NFEs, or GPU memory (for per-sample)) across all evaluated methods. This omission makes it hard to assess a performance vs. cost trade-off, e.g. also for the proposed per-sample vs per-batch methods.\n- **Further Justification of Methodological Choices:** The per-sample optimization strategy would benefit from further justification. The choice of the AdamW optimizer is not clearly explained, particularly the potential effect of weight decay on a noise latent. Furthermore, the absence of a regularization term to keep the optimized noise within the initial Gaussian distribution raises questions about the potential for out-of-distribution \"reward hacking.\"\n- **Opportunity to Enhance Evaluation Metrics:** The evaluation could be enhanced by decoupling different aspects of generation quality. Relying primarily on global metrics like FID can make it difficult to pinpoint the specific effects of the methods on distinct axes like image fidelity versus sample diversity. This presents an opportunity to provide a more granular understanding of the methods' impact.", "was_revised": true, "final_justification": "After careful consideration of the authors' rebuttal and our subsequent discussion, I maintain my score of 4: Borderline accept.\n\n**Issues Successfully Resolved:**\n\n- Computational cost analysis was provided with detailed GPU memory and timing comparisons\n- The choice of AdamW optimizer was justified with comparative experiments showing superior performance over SGD and Adam\n- Evaluation was enhanced with LPIPS and ImageReward metrics, providing better separation of quality and diversity effects\n- LPIPS computation methodology was clarified, confirming higher scores indicate greater diversity as intended\n- SGD performance differences were explained as resulting from learning rate tuning rather than fundamental algorithmic issues\n\n**Areas for Improvement:**\n\n- Theoretical presentation: The discussion revealed that weight decay serves as implicit regularization to keep noise in-distribution, which is a fundamental aspect of the method's theoretical soundness that was not included in the original paper\n- Methodological documentation: Key design choices such as the regularization strategy would benefit from more comprehensive explanation in the main text\n- Implementation details: The method's sensitivity to optimizer choice and hyperparameter tuning suggests some aspects could be better documented for reproducibility\n\n**Overall Assessment:**\n\nThe paper presents a novel and valuable perspective on memorization in diffusion models with practical mitigation strategies. The empirical results are convincing and the core insights about initial noise impact are important. The authors demonstrated good responsiveness to feedback and committed to including important theoretical details in the revision. The technical contributions and practical value support acceptance, though some methodological aspects could be better documented in the original submission.", "rebuttal": "Thank you for your insightful comments. We've carefully considered them and provided responses. Please let us know if you need any clarification or have additional questions.\n\n> **Q1-1**: Computational Cost Analysis.\n\nTo ensure a fair comparison, we measured GPU memory usage and inference time per image. As shown in the table, Batch-wise incurs minimal overhead, while Per-sample requires slightly more resources.\n\nNonetheless, as addressed in our responses to Reviewer tNu7’s Weaknesses 2 and 4, Per-sample consistently achieves the best Pareto fronts (SSCD vs. CLIP) across SDv1.4 and SDv2.0 compared to other baselines.\n\nThese findings suggest that Per-sample offers superior memorization mitigation performance and robustness across different models. As discussed in Section 6, effective mitigation of memorization plays a crucial role in enhancing privacy and intellectual property protection while significantly reducing potential risks. Considering this, we believe that the additional cost of the Per-sample method is a reasonable trade-off.\n\n|                                                 | gpu memory usage (MB) | inference time (sec) |\n|-------------------------------------------------|-----------------------|----------------------|\n| No Mitigation                                   | 3564.94               | 1.85                 |\n| Adding 4 Random Tokens                          | 3573.57               | 1.85                 |\n| Prompt Embedding Adjustment (l_target=3)        | 8115.53               | 1.98                 |\n| Cross-attention Scaling                         | 3170.38               | 1.93                 |\n| Opposite Guidance with Dynamic Transition Point | 3573.57               | 1.87                 |\n| Ours (Batch-wise)                               | 3573.57               | 2.01                 |\n| Ours (Per-sample)                               | 9701.62               | 2.8                  |\n\n> **Q1-2**: The choice of AdamW and the effect of weight decay.\n- On the choice of AdamW\n\nAs shown in Section 3.1.2, we observed that initial noise with a smaller conditional guidance leads to an earlier transition point, which in turn improves memorization mitigation performance. Therefore, our goal is to effectively reduce the norm $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$.\n\nWe experimented with minimizing this norm using both SGD and AdamW. The results show that AdamW reduces $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$ more effectively than SGD. This leads to a substantial improvement in mitigation performance, as demonstrated in the table below. As shown in the table, AdamW achieves the best mitigation performance, and we therefore adopt it as our optimizer.\n\n|                    | SSCD   | CLIP score | FID      |\n|--------------------|--------|------------|----------|\n| AdamW              | 0.2265 | 0.2647     | 111.7207 |\n| Adam               | 0.2265 | 0.2647     | 111.7207 |\n| SGD (momentum=0.9) | 0.4209 | 0.294      | 131.2913 |\n| SGD (momentum=0)   | 0.5326 | 0.301      | 151.0111 |\n\n- The effect of weight decay\n\nWe believe that weight decay plays a partial role in preventing adjusted initial samples from entering an out-of-distribution (OOD) state. This belief is grounded in the following reasoning. According to prior works [1, 2], the L2 norm of the initial noise $x_T \\sim \\mathcal{N}(0, I)$ follows a chi distribution: \n\n$||x_T|| = \\sqrt{\\sum_{i=1}^d {x_T^i}^2} \\sim \\chi^d = ||x_T||^{d-1}e^{-||x_T||^2/2}/(2^{d/2-1}\\Gamma({d \\over 2}))$,\n\nwhere $d$ is the dimensionality of $x_T$ and $\\Gamma(\\cdot)$ is the Gamma function. Based on this, to prevent the adjusted initial noise from deviating into an OOD state, one could consider minimizing the objective $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2 -w\\log{p(||x_T||)}$, where $p$ is the chi distribution and $w$ is a weight (hyperparameter). Since $p$ is the chi distribution, this objective can be rewritten as: \n\n$||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2 + c - {w (d-1)\\log{||x_T||} + {w{||x_T||^2 \\over 2}}}$, \n\nwhere $c$ is a constant independent of $x_T$. We argue that AdamW's weight decay implicitly minimizes the final term, ${||x_T||^2 \\over 2}$, and therefore helps prevent adjusted initial samples from becoming OOD. To empirically validate this, we conducted an experiment comparing 5000 unadjusted and 5000 adjusted initial noise samples. We measured the Jensen–Shannon divergence (JSD) between the two resulting distributions, varying the weight decay parameter $w$. The results, shown below, indicate that as $w$ increases, JSD decreases. This suggests that stronger weight decay encourages the adjusted samples to remain closer to the original (in-distribution) noise distribution.\n\n|     | w=0.01 | w=0.05 | w=0.1  | w=0.2 |\n|-----|--------|--------|--------|-------|\n| JSD | 0.0665 | 0.0663 | 0.0638 | 0.046 |\n\nFinally, to examine the effect of weight decay on mitigation performance, we varied the weight decay parameter $w$ as described above and measured the corresponding performance. The table below presents the results of this experiment.\n\n|             | w=0.01   | w=0.05   | w=0.1    | w=0.2    |\n|-------------|----------|----------|----------|----------|\n| SSCD        | 0.2265   | 0.2265   | 0.2265   | 0.2275   |\n| CLIP        | 0.2647   | 0.2647   | 0.2647   | 0.2657   |\n| FID         | 111.7207 | 111.7221 | 111.7221 | 112.0909 |\n| LPIPS       | 0.7615   | 0.7615   | 0.7615   | 0.7610   |\n| ImageReward | -0.3522  | -0.3522  | -0.3522  | -0.3501  |\n\nAcross the experimental settings we used, the performance remained relatively consistent regardless of the value of $w$. We believe this is because the default value used in our experiments, $w=0.01$, does not cause the adjusted initial noise to significantly deviate from the original distribution $\\mathcal{N}(0, I)$. Indeed, as shown in the JSD results table above, the JSD at $w=0.01$ is 0.0665, indicating minimal distributional shift.\n\n> **Q1-3**: A reason why this type of regularization is not applied.\n\nWe did not explicitly incorporate a regularization term to ensure that the initial noise $x_T$ remains within the high-likelihood region of the Gaussian distribution. However, as mentioned in our response to the previous question, the experimental setting used for performance evaluation in this paper (AdamW with $w = 0.01$) does not cause the adjusted initial noise to deviate significantly from the original Gaussian distribution. In fact, it results in a low JSD value.\n\nMoreover, even when we increased the value of $w$—thereby encouraging the adjusted initial noise $x_T$ to stay closer to the high-likelihood region of the Gaussian distribution—no significant performance improvement was observed. These findings suggest that the adjusted initial samples are unlikely to fall into an out-of-distribution (OOD) state that might trigger \"reward hacking\" behavior.\n\nThat said, as you pointed out, explicitly regularizing the adjusted initial noise $x_T$ to remain in-distribution—for example, by adding the regularization term $-\\log{p(||x_T||)}$—could contribute to safer deployment of diffusion models. Since\n\n$-\\log{p(||x_T||)}= c - { (d-1)\\log{||x_T||} + {{||x_T||^2 \\over 2}}}$,\n\nand weight decay already minimizes the ${{||x_T||^2 \\over 2}}$ term, it should be straightforward to incorporate the full regularizer by augmenting the existing metric $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$ with the additional term $- { (d-1)\\log{||x_T||}}$.\n\n> **Q2**: Decomposing evaluation into quality and diversity.\n\nAs suggested, we evaluated performance using LPIPS and ImageReward. For both metrics, higher values are better. Despite similar CLIP and ImageReward scores, methods like Adding 4 Random Tokens, Prompt Embedding Adjustment, and Per-sample show notable differences in SSCD and LPIPS, underscoring the strength of our approach.\n\n|                                                 | SSCD   | CLIP   | LPIPS  | ImageReward |\n|-------------------------------------------------|--------|--------|--------|-------------|\n| No Mitigation                                   | 0.5439 | 0.3012 | 0.4835 | -0.014      |\n| Adding 1 Random Token                           | 0.4794 | 0.2903 | 0.5471 | -0.1013     |\n| Adding 4 Random Tokens                          | 0.3402 | 0.2608 | 0.6671 | -0.3532     |\n| Prompt Embedding Adjustment (l_target=3)        | 0.1325 | 0.2372 | 0.7522 | -0.6466     |\n| Prompt Embedding Adjustment (l_target=5)        | 0.2527 | 0.2644 | 0.7003 | -0.3338     |\n| Cross-attention Scaling                         | 0.2779 | 0.2661 | 0.7194 | -0.2115     |\n| Opposite Guidance with Dynamic Transition Point | 0.2396 | 0.2409 | 0.7502 | -0.6639     |\n| Ours (Batch-wise)                               | 0.2642 | 0.2603 | 0.7384 | -0.4527     |\n| Ours (Per-sample)                               | 0.2265 | 0.2647 | 0.7615 | -0.3522     |\n\n> **Q3**: Synergy with other methods.\n\nAs shown in [3], $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$ is typically higher for memorized prompts. Our method reduces this value by adjusting the initial noise $x_T$ for such cases.\n\nThis design is compatible with other noise optimization techniques aimed at enhancing visual quality or prompt alignment, as long as they do not increase $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$. Integration can be done by simply adding our term to existing loss functions.\n\n> **Q4**: Generality of the Framework.\n\nIf an analogous attraction basin could be defined based on unconditional noise prediction $\\epsilon_\\theta(x_t, t)$ and large $||\\epsilon_\\theta(x_t, t)||_2$ values similarly indicate strong steering, our method could be adapted accordingly.\n\n[1] Samuel et al. “Norm-guided latent space exploration for text-to-image generation”. NeurIPS 2023.\n\n[2] Ben-Hanmu et al. \"D-Flow: Differentiating through Flows for Controlled Generation\". ICML 2024.\n\n[3] Jain et al. \"Classifier-Free Guidance inside the Attraction Basin May Cause Memorization\". CVPR 2025."}, {"review_id": "OJaXdvzxH1", "reviewer": "Reviewer_XBNj", "summary": "The task that this paper tackles is the memorization issue in text-to-image diffusion models, which represent the fact that these models often memorize and replicate training data, leading to privacy and copyright concerns. This paper builds on a recent work by Jain et al., where the baseline method views this problem in the lens of the attraction basin, which refers to a region where applying classifier-free guidance (CFG) can cause memorization during denoising. Also, the baseline paper identifies a phenomenon that the earlier a generation can escape from such region, the better alignment can be preserved during the memorization mitigation. Based on these insights, this paper proposes a new mitigation strategy that adjust the initial noise by search initializations that are more easily to escape from the region. Two variants of noise adjustment methods are propose: batch-wise and per-sample. Experiments are conducted on Stable Diffusion v1.4 architecture.", "questions": "- It is recommended to also present the comparisons of performance with and without adjustment for the per-sample strategy in addition to the current Figure 4.\n- The effectiveness of the proposed methods are not convincing.\n- A more comprehensive comparison over different privacy levels is very important for effectively comparing performances across different mitigation strategies.\n- Can you explain the experimental settings you used for producing Figure 4?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- The writing of this paper is clear and structured. Visuals are well-employed to facilitate the discussions. Thus, I would evaluate its clarity to be good.\n- The motivation of the addressed problem is well-justified, making it convincing for the readers that mitigating memorization in text-to-image diffusion models is a practical challenge that needs to be addressed.\n- The proposed mitigation strategy is developed from a novel perspective (i.e., adjusting the initial noise) from existing methods that tackle this problem by modifying the denoising process during inference.\n\n**Weaknesses**:\n\n- The baseline comparisons are quite insufficient as the paper currently only compares with one or two privacy levels for each compared method. For example, the cross-attention scaling method is only shown in one privacy level, and the random token and prompt embedding adjustment methods are only shown in two privacy levels. However, it is essential to have much more results compared under different privacy levels to sufficiently compare how much alignment (CLIP score) or fidelity (FID) is sacrificed, so that the effectiveness can be demonstrated.\n- The effectiveness of the proposed method appears suboptimal compared to the baselines. For example, the batch-wise strategy shows inferior SSCD and CLIP scores relative to the baseline from Wen et al. Additionally, all three metrics (SSCD, CLIP score, and FID) indicate that the proposed method underperforms compared to the baseline from Jain et al. The per-sample strategy also frequently yields worse results than the baseline on one or two of these metrics. Combined with the earlier concern about the very limited scope of the current evaluation, which means the proposed method already struggles in this narrow performance snapshot, let alone in a broader comparison across different privacy levels. Therefore, I am concerned about the paper’s practical significance.\n- The comparisons of performance with and without adjustment should be conducted for both batch-wise and per-sample strategies. However, only the batch-wise strategy’s results are shown in Figure 4. The authors also acknowledged that the performance of batch-wise strategy is quite sensitive to the choice of hyperparameter values, while the effectiveness is mainly demonstrated in the per-sample strategy. This makes it more important to present the result for per-sample strategy.\n- The settings of Figure 4 need to be elaborated in more details: are the settings align with the ones used in creating Table 1 that both follow the setup in section 4.1? If yes, does the “no adjustment” version correspond to one of the baseline versions?", "was_revised": true, "final_justification": "The rebuttal has effectively addressed my major concerns regarding the experimental performance and insufficient comparisons. Therefore, I have increased my final rating from 3 to 4 and the significance rating from 1 to 3. I recommend that the authors include the revised results in a Figure format for better presentation of a more comprehensive result (which is understandable and perfectly fine to provide tables in the rebuttal due to the constraint that Figures are not supported in this rebuttal).", "rebuttal": "We thank the reviewer for the constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.\n\n> **Q1**: Comparisons of performance with and without adjustment for the Per-sample strategy.\n\nAs described in Section 3.2.2, the *Per-sample* strategy involves adjusting the initial noise $x_T$ and applying CFG from the beginning of the sampling process. Without the initial noise adjustment, hence, the *Per-sample* strategy becomes equivalent to the *No Mitigation.*\n\nTherefore, the performance comparison between *No Mitigation* and *Per-sample* reported in Table 1 allows us to assess the effect of applying the initial noise adjustment. Additionally, Appendix D.3 presents the performance of the *Per-sample* strategy under different adjustment strengths. The comprehensive results are summarized in the table below, demonstrating that initial noise adjustment effectively mitigates memorization.\n\n|                           | SSCD   | CLIP Score |\n|---------------------------|--------|------------|\n| No Mitigation             | 0.5439 | 0.3012     |\n| Per-sample (l_target=1.5) | 0.3593 | 0.2850     |\n| Per-sample (l_target=1.3) | 0.3220 | 0.2810     |\n| Per-sample (l_target=1.1) | 0.2778 | 0.2743     |\n| Per-sample (l_target=0.9) | 0.2265 | 0.2647     |\n| Per-sample (l_target=0.7) | 0.1731 | 0.2505     |\n\n> **Q2**: The effectiveness of the proposed methods.\n\nTo conduct a more comprehensive performance evaluation, we varied the mitigation strength for both the baselines and our proposed methods. The results are summarized in the table below, which also includes evaluations using LPIPS (diversity) and ImageReward (quality and prompt alignment). All other experimental settings are consistent with those in Table 1.\n\nOur main finding is that adjusting the initial noise $x_T$ to minimize $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$ leads to an earlier transition point and improves mitigation performance. Based on this, we proposed two strategies—*Per-sample* and *Batch-wise*—which share the same core idea.\n\n*Batch-wise* involves three hyperparameters ($\\tilde{\\gamma}$, $\\rho$, and $M$), while *Per-sample* uses one (l_target). Since mitigation strength is closely related to the value of $||\\tilde{\\epsilon}_\\theta(x_T, T, y)||_2$, we report the performance of Per-sample across different l_target values.\n\nUsing the data in the table, one can plot SSCD vs. CLIP curves and observe that *Per-sample* achieves the best Pareto fronts. *Batch-wise* also shows strong performance, comparable to *Cross-attention Scaling*. These results demonstrate the effectiveness of our proposed methods.\n\nWhile we are unable to include figures in this rebuttal, we will provide graphical results in the revised version of the paper.\n\n|                                                              | SSCD   | CLIP   | FID      | LPIPS  | ImageReward |\n|--------------------------------------------------------------|--------|--------|----------|--------|-------------|\n| No Mitigation                                                | 0.5439 | 0.3012 | 160.4486 | 0.4835 | -0.014      |\n| Adding 1 Random Token                                        | 0.4794 | 0.2903 | 146.8346 | 0.5471 | -0.1013     |\n| Adding 2 Random Tokens                                       | 0.4171 | 0.2778 | 132.5187 | 0.5714 | -0.1581     |\n| Adding 4 Random Tokens                                       | 0.3402 | 0.2608 | 127.291  | 0.6671 | -0.3532     |\n| Adding 6 Random Tokens                                       | 0.2728 | 0.2405 | 110.9416 | 0.679  | -0.5496     |\n| Adding 8 Random Tokens                                       | 0.2147 | 0.2240 | 104.5567 | 0.7098 | -0.7387     |\n| Prompt Embedding Adjustment (l_target=3)                     | 0.1325 | 0.2372 | 97.5206  | 0.7522 | -0.6466     |\n| Prompt Embedding Adjustment (l_target=4)                     | 0.1994 | 0.2545 | 104.7461 | 0.723  | -0.443      |\n| Prompt Embedding Adjustment (l_target=5)                     | 0.2527 | 0.2644 | 108.8194 | 0.7003 | -0.3338     |\n| Prompt Embedding Adjustment (l_target=6)                     | 0.3117 | 0.2733 | 116.4698 | 0.668  | -0.251      |\n| Prompt Embedding Adjustment (l_target=7)                     | 0.3586 | 0.2811 | 123.8324 | 0.6355 | -0.1835     |\n| Cross-attention Scaling (c=1.15)                             | 0.3094 | 0.2732 | 121.5525 | 0.6983 | -0.1236     |\n| Cross-attention Scaling (c=1.2)                              | 0.2935 | 0.2696 | 119.1731 | 0.7095 | -0.1666     |\n| Cross-attention Scaling (c=1.25)                             | 0.2779 | 0.2661 | 117.6667 | 0.7194 | -0.2115     |\n| Cross-attention Scaling (c=1.3)                              | 0.2662 | 0.2632 | 117.2156 | 0.7275 | -0.2506     |\n| Cross-attention Scaling (c=1.35)                             | 0.2566 | 0.2609 | 117.0359 | 0.7329 | -0.2803     |\n| Opposite Guidance with Dynamic Transition Point (og_scale=1) | 0.2396 | 0.2409 | 94.2193  | 0.7502 | -0.6639     |\n| Opposite Guidance with Dynamic Transition Point (og_scale=3) | 0.2136 | 0.2403 | 93.8351  | 0.772  | -0.6772     |\n| Opposite Guidance with Dynamic Transition Point (og_scale=5) | 0.1908 | 0.2315 | 90.0785  | 0.7897 | -0.766      |\n| Opposite Guidance with Dynamic Transition Point (og_scale=7) | 0.1711 | 0.2202 | 87.6463  | 0.8367 | -0.8884     |\n| Opposite Guidance with Dynamic Transition Point (og_scale=9) | 0.1247 | 0.2003 | 84.7859  | 0.8393 | -1.1194     |\n| Batch-wise                                                   | 0.2642 | 0.2603 | 105.5606 | 0.7384 | -0.4527     |\n| Per-sample (l_target=0.7)                                    | 0.1731 | 0.2505 | 106.06   | 0.8028 | -0.5552     |\n| Per-sample (l_target=0.9)                                    | 0.2265 | 0.2647 | 111.7207 | 0.7615 | -0.3522     |\n| Per-sample (l_target=1.1)                                    | 0.2770 | 0.2743 | 117.8896 | 0.7174 | -0.2556     |\n| Per-sample (l_target=1.3)                                    | 0.322  | 0.281  | 124.5573 | 0.6796 | -0.1847     |\n| Per-sample (l_target=1.5)                                    | 0.3593 | 0.285  | 128.595  | 0.6473 | -0.1414     |\n\n> **Q3**: A more comprehensive comparison over different privacy levels.\n\nThrough the above response, we provide a comprehensive comparison across different privacy levels. The fact that Per-sample achieves the best Pareto fronts further demonstrates the effectiveness of our proposed method.\n\n> **Q4**: The experimental settings used in Figure 4.\n\nThe experimental setup in Figure 4 follows the same configuration as in Table 1; however, the *No adjustment* shown in Figure 4 does not correspond to any of the baselines reported in Table 1. Before explaining this further, we would like to clarify the purpose of Figure 4.\n\nOur goal was to examine whether initial noise adjustment meaningfully affects mitigation performance. For this, we used the *Batch-wise* strategy. The mitigation performance of *Batch-wise* is primarily influenced by two factors: (1) whether initial noise adjustment is applied, and (2) the starting timestep for applying CFG (denoted as $\\tau$).\n\nWe observed that increasing $\\tau$—that is, applying CFG earlier—tends to improve both SSCD and CLIP scores, indicating a trade-off between the two. Importantly, this trend holds consistently regardless of whether initial noise adjustment is used. This is likely because the point at which a sample exits the attraction basin varies depending on the seed (i.e., $x_T$) and the prompt; thus, applying CFG too early (i.e., at higher $\\tau$) increases the chance of generating memorized images.\n\nFor this reason, we varied the $\\tau$ values under both settings—with and without initial noise adjustment—and reported the results in Figure 4, labeled as *With adjustment* and *No adjustment*, respectively. As shown, initial noise adjustment yields a more favorable trade-off between SSCD and CLIP scores compared to the *No adjustment*. This demonstrates that initial noise adjustment indeed enhances memorization mitigation performance.\n\n(Note: In Figure 4, the rightmost data point on the *No adjustment* curve corresponds to $\\tau = 780$, and the leftmost to $\\tau = 720$. Similarly, the rightmost and leftmost data points on the *With adjustment* curve correspond to $\\tau = 900$ and $\\tau = 840$, respectively.)"}, {"review_id": "ZSv3CzoH6B", "reviewer": "Reviewer_ye9A", "summary": "Previous research has found an attraction basin that leads to memorization in T2I Diffusion models. Deferring the use of classifier-free guidance helps prevent memorization. However, if the transition point of escaping the attraction basin is in later timesteps, the alignment of the image with the given prompt is poor. Therefore, this paper presents the finding that with different initial noise initializations, the transition point can be in earlier or later timesteps.\nAs a mitigation, two methods are proposed: searching for the optimal initial noise to shift the transition point to earlier timesteps, effectively mitigating memorization.", "questions": "- Q1: Why is the FID score so high? On what dataset was the FID score calculated? Is the FID score calculated on the memorized samples?\n- Q2: Why is the FID score improving when performing mitigation?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper presents the critical insight that the choice of the initial noise highly influences the degree of memorization\n- The paper is well written\n- The paper is easy to read and easy to understand\n\n**Weaknesses:**\n- In the batch-wise mitigation, multiple additional time steps need to be computed\n- Finding the initial noise is quite computationally intensive because optimization needs to be done\n- It is a bit confusing that in the experiments 50 time steps are used, but in the graphs in Figure 3 1000 time steps are shown\n- Comparison to the inference time mitigation of Hintersdorf et al. is missing", "was_revised": true, "final_justification": "My concerns have been appropriately addressed, which is why I raise my score to \"accept\".", "rebuttal": "We are very grateful for your constructive comments. We have provided answers to each comment. Please let us know if you need any clarification or have additional questions.\n\n> **W1 & W2**: Computational Cost Analysis.\n\nTo ensure a fair comparison, we measured GPU memory usage and inference time per image. As shown in the table, Batch-wise incurs minimal overhead, while Per-sample requires slightly more resources.\n\nNonetheless, as addressed in our responses to Reviewer tNu7’s Weaknesses 2 and 4, Per-sample consistently achieves the best Pareto fronts (SSCD vs. CLIP) across SDv1.4 and SDv2.0 compared to other baselines.\n\nThese findings suggest that Per-sample offers superior memorization mitigation performance and robustness across different models. As discussed in Section 6, effective mitigation of memorization plays a crucial role in enhancing privacy and intellectual property protection while significantly reducing potential risks. Considering this, we believe that the additional cost of the Per-sample method is a reasonable trade-off.\n\n|                                                 | gpu memory usage (MB) | inference time (sec) |\n|-------------------------------------------------|-----------------------|----------------------|\n| No Mitigation                                   | 3564.94               | 1.85                 |\n| Adding 4 Random Tokens                          | 3573.57               | 1.85                 |\n| Prompt Embedding Adjustment (l_target=3)        | 8115.53               | 1.98                 |\n| Cross-attention Scaling                         | 3170.38               | 1.93                 |\n| Opposite Guidance with Dynamic Transition Point | 3573.57               | 1.87                 |\n| Ours (Batch-wise)                               | 3573.57               | 2.01                 |\n| Ours (Per-sample)                               | 9701.62               | 2.8                  |\n\n> **W3**: Used timesteps in the experiments.\n\nFollowing the experimental setup of prior work [1], we report our results in Figure 3. Using 50 inference steps for denoising results in a step size of 20, meaning the timesteps used are [981, 961, 941, …, 1].\n\n> **W4**: On the missing inference time mitigation baseline (Hintersdorf et al.)\n\nOur proposed method is an inference-time mitigation approach that does not require access to training data. Therefore, as described in Section 4.1, we selected baseline methods that also operate at inference time and do not rely on training data, to ensure a fair comparison.\n\nTo the best of our knowledge, NeMo [2] requires access to training data to compute activation statistics on unmemorized samples.\n\n> **Q1**: On the calculation of FID score.\n\nWe computed the FID using 10000 randomly sampled images from the LAION dataset and 5000 generated images produced from memorized prompts. We suspect that the number of generated and reference images, along with the exclusive use of memorized prompts for generation, may have influenced the scale of the FID score.\n\n> **Q2**: The reason why mitigation methods improve FID score.\n\nWithout mitigation, diffusion models tend to generate identical or highly similar images for memorized prompts, which significantly reduces diversity. In contrast, applying mitigation alleviates this issue, increases diversity, and is likely to improve the FID score as a result.\n\nThe table below extends Table 1 by including LPIPS (a diversity metric) and ImageReward (a metric for quality and prompt alignment), where higher values are better for both. As shown, applying mitigation leads to higher LPIPS scores, confirming that diversity indeed improves.\n\n|                                                 | SSCD   | CLIP Score | FID      | LPIPS  | ImageReward |\n|-------------------------------------------------|--------|------------|----------|--------|-------------|\n| No Mitigation                                   | 0.5439 | 0.3012     | 160.4486 | 0.4835 | -0.014      |\n| Adding 1 Random Token                           | 0.4794 | 0.2903     | 146.8346 | 0.5471 | -0.1013     |\n| Adding 4 Random Tokens                          | 0.3402 | 0.2608     | 127.2910 | 0.6671 | -0.3532     |\n| Prompt Embedding Adjustment (l_target=3)        | 0.1325 | 0.2372     | 97.5206  | 0.7522 | -0.6466     |\n| Prompt Embedding Adjustment (l_target=5)        | 0.2527 | 0.2644     | 108.8194 | 0.7003 | -0.3338     |\n| Cross-attention Scaling                         | 0.2779 | 0.2661     | 117.6667 | 0.7194 | -0.2115     |\n| Opposite Guidance with Dynamic Transition Point | 0.2396 | 0.2409     | 94.2193  | 0.7502 | -0.6639     |\n| Ours (Batch-wise)                               | 0.2642 | 0.2603     | 105.5606 | 0.7384 | -0.4527     |\n| Ours (Per-sample)                               | 0.2265 | 0.2647     | 111.7207 | 0.7615 | -0.3522     |\n\n\n[1] Jain et al. \"Classifier-Free Guidance inside the Attraction Basin May Cause Memorization\". CVPR 2025.\n\n[2] Hintersdorf et al. \"Finding nemo: Localizing neurons responsible for memorization in diffusion models\". NeurIPS 2024."}, {"review_id": "w4PfnRcRhk", "reviewer": "Reviewer_tNu7", "summary": "The paper investigates how to stop diffusion models from recreating train-data images. It builds upon the framework of attraction cones and proposes that the initial noise can be shifted to avoid falling into attraction cones. This allows to stay outside the memoization cones from an earlier point on, so that CFG can be applied almost from start (in the case of the single-sample method even from start, though at higher computational cost), which retains prompt-adherence better.", "questions": "Section 3.1.2. currently relies only on a qualitative analysis. Could you add a quantitative study across a whole dataset? \n\nCurrently, your analyses focus on 1000-step diffusions. How does it look like for 5-20 step diffusions (for example on SD3 (which you already use) or FLUX (just if you have time to test it; it uses the same schedulers etc as SD3, so it should be plug-and-play if you load it via huggingface))?\n\nCurrently, you base the stop-point analysis on the timestep index. However, this implicitly depends on which scheduler you use. Could you have a look at how it looks with the Signal-to-noise ratio on the x-axis? (as suggested by “Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation“, 2023)\n\nGiven that you find that the initial noise matters — can you identify any regions in the initial noise space? If we could sample from that region from the get-go, we could prevent falling into attraction cones.\n\nDo you have an intuition of how far your initial noise adjustment pushes the noise away? This could give a rough estimate of how large the diameter of the attraction cones is at t=T, so how much of the initial noise Gaussian is affected by memorization.\n\nIs there any runtime difference between the batchwise mitigation and the per-sample mitigation? It seems like the only difference is whether we use pooled backprop gradients or per-sample ones (please correct me if I’m wrong), so probably single-sample only uses more VRAM to store the individual gradients? \n\nCould you try out combining your method with orthogonal methods like prompt embedding adjustment? It would be interesting to see if they are truly orthogonal (and thus add performance in combination) or doing the same by different means (no performance boost by combining them). It would be best to plot pareto fronts. \n\nHow was the CFG weight of 7 chosen? In my experience, modern diffusion models usually have lower CFGs to, e.g., give the best FID.", "limitations": "The authors discuss the runtime overhead and hyperparamter tuning in an explicit Limitations section. The fact that SD2 (and above) have close to no problems with exact memorization is discussed, but outside the limitations section.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* The problem at hand, memorization is well-motivated and relevant\n* The authors nicely first introduce the principles and some proof-of-works of the method before applying it on large scale\n* The used metrics are reasonable, and the authors measure both the desired memorization but also impacts on prompt adherence (CLIP Score) and image quality (FID)\n\n## Weaknesses\n\n### Weaknesses that influence my score, in order of strength:\n\n* The motivation for the work — that diffusion models memorize exact images — might be a problem of the past. The authors note that they use Stable Diffusion v1.4 because SD2 already does not have the problem anymore (let alone SD3). This is in line with my experimental experience. Also, the CFG strength is quite strong (=7), probably at more modest strengths there is less memorization to begin with. This limits the relevance of the work, but I still find it interesting to investigate the effect, e.g., for diffusion models that were trained on less data (even in the future).\n* The evaluation of the work is very thin. The method is tested on one model (SD v1.4) on 500 prompts (generating 10 images each). Since the authors already note that more modern Diffusion models have less memorization problems, it would be interesting to see results across different scales (SD2, SD3) and architectures (FLUX, RGB-space diffusion) to understand it better and to show the robustness of the proposed method.\n* According to Table 1, the proposed method is not much better than prompt embedding adjustment. This is a slight performance-weakness, but in my view not a complete rejection reason, because prompt adjustment and noise adjustment are orthogonal approaches, and in some approaches prompt embedding adjustment might not be available (unconditional generation) or desired (might be attached with adversarial prompting).\n* Table 1 currently shows results for only one hyperparameter, and it is not explained how that was chosen. The results would be more trustworthy if there were Pareto fronts across different hyperparameters.\n* There are three hyperparamters: two to control how strongly to apply mitigation, one to control how many mitigation steps to make (plus one to decide when to start CFG, but other methods also have this one, and it is removed in the single-sample method)\n\n### Other notes (that do not influence my score but could improve the camera-ready)\n\n* It might be interesting to add FD_Dinov2 as an evaluation metric, if computationally feasible, because it is more detailed than FID (based on Inception v3).\n\n## Justification for score\n\nThe paper is logically structured and the method reasonable in theory and performant in practice. However, there are two bigger issues: First, the experimental evaluation is very thin with only one model and 500 prompts. Second, the tested model is quite old (SD1.4) because more modern diffusion models have less memorization problems to begin with. This gives this paper more of an impression of an initial proof-of-concept suitable for a workshop rather than a full conference paper. I am willing to increase my score if the authors can improve the experimental evaluation in particular.", "was_revised": true, "final_justification": "After the rebuttal discussion and having read the rebuttal discussions with all other reviewers, I think the score of 3 still applies to the paper. The method gives an interesting view into the diffusion dynamics, but the results are thin: All evaluations are on (10x) 500 prompts as opposed to the typical 50k, and models include only SD 1.4, and now with the rebuttal SD 2.0, although more modern diffusion models already exist (even with the same Euler Discrete Scheduler, so no code changes are needed), and usually behave much smoother in terms of memorization prevention. What adds to my scepticism is that in the rebuttal, when providing the results on SD 2.0, the authors claim \"Per-sample achieves the best Pareto front\" but when plotting the results it is obvious that the Pareto front is exactly overlaying with a very simple baseline. These points leave me sceptical with the paper's sometimes bold claims. \n\nI believe the responsibility of a reviewer towards the field is to be willing to defend the paper should future studies find it to not reproduce, and with the current thin evaluation I am not convinced enough to to so. As such, I believe the most secure way forward is to reject at this point, but keep working on the paper a few more weeks to make the findings more robust by analyzing modern Diffusion models, not shying back from admitting when the method does _not_ work to give an honest scientific insight into diffusion dynamics, and then resubmit to, e.g., ICLR.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work aims to reduce memorization in diffusion-based text-to-image generation by proposing a method that adjusts the initial noise (either at a batch-level or sample-level) which encourages the model to escape the attraction basin. The method is novel, has solid theoretical groundings and experimental results are satisfactory. Discourse with reviewers have mostly addressed all concerns. Although the relevance of the work may be diminishing given the latest image-generation models don't have a severe memorization issue, I am willing to recommend acceptance due to the algorithmic contributions of this paper.
Decision: Accept (poster)