On the Edge of Memorization in Diffusion Models
Authors: Sam Buchanan
Druv Pai
Yi Ma
Valentin De Bortoli
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
memorization
generalization
phase transition
denoising
Primary area: deep_learning
TL;DR: We build a nearly-perfect predictive model for memorization in diffusion models using theory and controlled experiments.

Abstract: When do diffusion models reproduce their training data, and when are they able to generate samples beyond it? A practically relevant theoretical understanding of this interplay between memorization and generalization may significantly impact real-world deployments of diffusion models with respect to issues such as copyright infringement and data privacy. In this paper, we propose a scientific and
mathematical “laboratory” for investigating memorization and generalization in diffusion models trained on fully synthetic or natural image-like structured data. Within this setting, we theoretically characterize a crossover point wherein the weighted training loss of a fully generalizing model becomes greater than that of an underparameterized memorizing model at a critical value of model (under)parameterization. We then demonstrate via carefully-designed experiments that the location of this crossover predicts a phase transition in diffusion models trained via gradient descent, as our theory enables us to analytically predict the model size at which memorization becomes predominant. Our work provides an analytically tractable and practically meaningful setting for future theoretical and empirical investigations. Code for our experiments is available at https://github.com/DruvPai/diffusion_mem_gen.

Onn the Edge of Memorization in Diffusion Models
Sam Buchanan" TTIC
Druv Pai' UC Berkeley
Yi Ma UC Berkeley, HKU
Valentin De Bortoli Google DeepMind
Abstract
When do diffusion models reproduce their training data, and when are they able to generate samples beyond it? A practically relevant theoretical understanding of this interplay between memorization and generalization may significantly impact real-world deployments of diffusion models with respect to issues such copyright infringement and data privacy. In this work, to disentangle the different factors that influence memorization and generalization in practical diffusion models, intro- duce a scientific and mathematical "laboratory" for investigating these phenomena in diffusion models trained fully synthetic natural image-like structured data. Within this setting, we hypothesize that the memorization or generalization behavior of underparameteriize trained model is determined by the difference in training loss between an associated memorizing model and a generalizing model. To probe this hypothesis, we theoretically characterize a crossover point wherein the weighted training loss of fully generalizing model becomes greater than that of an underparameterized memorizing model at critical value of model (un- der)parameterizzaton. We then demonstrate via sarefully-desinee experiments that the location of this crossover predicts a phase transition in diffusion models trained via gradient descent, validating our hypothesis. Ultimately, our theory enables us to analytically predict the model size which memorization becomes predominant. Our work provides an analytically tractable and practically meaningful setting for future theoretical and empirical investigations. Code for our experiments is available https: //tgth/gbbbtt wwwp/Druuaaa//iiffiiio....o
1 Introduction
Diffusion models are one of the premier methodologies for deep generative modeling. They exhibit great capabilities across modalities and are state-of-the-art at synthesizing images [Saharia et
2022
Podell et al., (023), videos [Ho et al.) 2022 Blattmann et al. 2023). or proteins [Watson et al.
2023
Despite significant success when used in practice, in the context of large-scale commercial
deployment [Ramesh et al. 2022] diffusion models are often plagued with data privacy and copyright
infringement issues Ghalebikesabi et al. 2023 Carlini et al. 2023 Nasr et al. 2023 Cui et al. 2023
Wang al. 2024a Vyas et al. 2023 Franceschelli and Musolesi, 2022 which may have significant
long-term consequences. These issues stem from the possibility and (sometimes) propensity for
diffusion models memorize their training data. Namely, in some cases, trained diffusion models generate outputs which are verbatim copies of the training samples (Zhang et al., 2023 Kadkhodaic al. 2023). Yet, in many other cases, these models can generate outputs which appear natural but are not present in the training set; this behavior is often described as creativity (Kamb and Ganguli
2024 or generalization (Zhang et al. 2023 Kadkhodaie et al. 2023
Niedoba et al.] 2002).
In general, memorization and generalization are difficult to disambiguate. There has been significant empirical work focusing on building heuristic approaches to detect and study memorization in large-
scale diffusion models [Zhang et al. 2023 Gu et al. 2023 Yoon et al. 2023 Carlini et al. 2023
"Theee authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Somepalli et al., 2023, Wen et al. 2024 Ross et al. 2024 Wang et al. 2024b a. Chen et al., 2024a Most such approaches use ad-hoc defmitions of memorization which compare the features of the generated samples (w.r.. some deep neural network encoder) to features of samples from the training data (Pizzi et al., 2022]. This choice effectively reduces the question of how exactly can we define memorization? to how are the features of the chosen encoder related to the input data? which is also a difficult problem [Papyan et al., 2020 Yu et al., 2023]. While such methods appear to perform reasonably well in some practical cases, we emphasize that such measures of memorization are ultimately heuristic and there is still scientific disagreement about their efficacy (Stein et al. [024]. This is unsuitable for building a scientific understanding of diffusion models that could, among other things, potentially suggest resolutions to the aforementioned legal and societal issues.
Thus, building theoretical and scientific understanding of memorization in diffusion models is critical. The central problem that such theory needs to contend with is that the training loss promotes memorization: given fixed sample set, sufficiently powerful and perfectly-optimizer diffusion model will always reproduce the training data exactly, i.e., every single one of its generations will be exactly training point [Peluchetti] 2023 Biroli et al. 2024, Kamb and Ganguli 2024]. Any theory of memorization must therefore explain why well-trained diffusion models do not always memorize. Several previous works have proposed different explanations for this issue in terms of certain aspects of the training procedure, such as the landscape of the stochastic optimization problem which minimizes the training loss [Wu et al., 2025 Vastola 2025) and the parameterization of the backbone denoiser in the diffusion model Yoon et al. [023]] Zhang et al.] 2023 Wang et al.
2024c| Kamb and Ganguli 2024 Niedoba et al. 2024 George et al. [2025] However, precise and predictive theoretical characterization of memorization remains elusive.
Our contributions. In this work, we theoretically investigate memorization and generalization in diffusion models. We first introduce memorization laboratory, a natural setting for investigating memorization and generalization in diffusion models trained on synthetic data. We justify this setting by proving that within it, we may distinguish a target distribution from its empirical version for many configurations of problem parameters. To explore the behavior of trained models, we hypothesize that denoisers trained via gradient descent-like methods memorize or generalize depending on whether the empirical (training) loss is lower for parameter-matcher memorizing denoisers or generalizing denoisers. This hypothesis, to the best of our knowledge, has not been formally explored by prior work, and can only be probed here due to the controlled laboratory setting. To formalize and eventually attempt to falsify this hypothesis, we introduce a partially memorizing denoiser, an underparamettrrize denoiser whose output distribution's 's samples are always memorized. Within simple setting of our laboratory, we characterize the critical level of model (under)parameteriiatti ("crossover point") at which the training loss of the partially memorizing denoiser first becomes lower than the idcalized generalizing denoiser, by deriving and using tight theoretical approximations to these losses. We show that, if our hypothesis is true, then this crossover characterization naturally provides the location of the phase transition from generalization to memorization in trained denoisers. Ultimately. we use these theoretically-derine derived tools build predictive model for the phase transition from generalization to memorization in terms of a minimal set of problem parameters, and show via experiments that our model achieves extremely low error in practice, validating our hypothesis. We finish our experiments by examining another setting of our laboratory which captures more of the complexities involved in training diffusion models on natural images, showing that despite its additional complexity it is qualitatively very similar to the first case. Beyond the current investigation, the framework we propose provides analytically tractable yet rich setting for further investigation of memorization and generalization in diffusion models.
All proofs are included in the supplementary material. Our notation is presented in Tables I
and
2 A Memorizatioa/GGnerrliaattooo Laboratory
Diffusion models show a complex tradeoff between model capacity, training compute, and dataset size with respect to key behaviors such as memorization and generalization. We present a framework ("laboratory") to disentangle these different factors. The class of models we study is sufficiently expressive to admit a rich family of behaviors while remaining tractable for theoretical analysis.
2

---

Diffusion models. Given a target probability distribution TT and training dataset (x') ) from T.. the goal of generative modeling to define an outpu probability distribution ă which approximates the underlying target .. Diffusion models (Sohl-Dickstein et al., 2015 Song and Ermon 2019 Ho et al., 2020] define such an output distribution स as the result of a stochastic process. More precisely, we consider, for E [0, 11. noising process with marginals as follows:
X₁ Xx + ozZ, Z ~ ~0,,), Xo Tx,
with oog 01 = and (1) = = 0. This noising process can be associated with the dynamic
dx, fxX,dt gd
Xo T
where fa and Is can be obtained in closed form, see Gao et al. d-dimensional Brownian motion. We recall that the backward process associated with 2 is given by
2024 for instance, and (B, (B.)e is
dy, = f-fi- + (si- (og + -1-idB, Yo I),,
where pt is the density of X, at time t. Using Tweedie's identity [Robbins 1956), we have
v (e = (a,ā(t, - xz)/o?;
(4)
where at,,,,) L X; zz). Thus, using the samples ( practical diffusion models define a denoiser that approximates z by solving the training loss minimization problem
min (x
(5)
where
CN (z. (x,)())x..(E)).
(x) = N
1
E
a(t X;) x x11"]
(6)
over parametric class of denoisers ãe with A(() weighting function, distributed as Uni 1]), and X; distributed according to the noising process IT with X,, - x i.e., X axæx + o. We then substitute the learned denoiser (t. 24) into suitable discretization the backward process (3) via 4 which defines the output distribution र as the marginal distribution of Y1.
As has been well noted in the literature, the training objective (5) is odds with the stated goal of diffusion models. This because the non-parametric minimizer of (5) memorizes the training data:
N
arg min CN A) &
Σ x softmax(w(-))) -))),
where ø(t, -) is square-integrable, for any u E R we have softmax(v)) (v) = e" / e", and
w;(xx:) = 20? |laxx - xel1? i= 1, N.
We denote the memorizing denoiser in (7) as Fmem (t, xt). Towards theoretically studying the muemorizationn/geeralizzz trade-off in diffusion models, first specify main assumptions regarding the data and our models which define our momorization//eeccrliiztoooo laboratory.
Data and model assumptions. We set n, to be an weighly- mixture of K Gaussians in Rd with means μ E Rd and covariances Σ 2
K
Σ N(u I)
(8)
K
Gaussian mixture models are flexible enough to represent a large class of datasets while being amenable to theoretical investigation, see (Wang and Vastola 2024 Shah et al. [023] Waag et al.)
2024c| for instance, Section 4.2 for an example, and Appendix A for further discussion.
Let M E N denote the number of equally mixture components in generic Gaussian mixture Te. with 0 (1r², Σ Ei....,,"" MM). Tweedie's 's identity (4) links the statistical model my
and its associated denoiser Te, which we recall below in Lemma 2.1 and prove in Appendix

---

Lemma 2.1 (Gaussian mixture model denoiser): Assume that To (1/M) E/ M Then, we have that
N(k', , E.)
M
Bo(t, xi)
21 - of a
(9)
i=1
where for all i E [M
1 log det + +f [ - a (a + + ((pt:-app))...
1
(t, 2
aip').
The class of M --parameter Gaussian mixture models constitutes a rich framework for investigating memorization and generalization. For at one extreme, where M K, Σ* Σ and μ² = -,, for
E [K] we recover the generalizing denoiser associated to the target distribution 8 which we will denote as æ.; and at the other extreme, with M N, E' recover the memorizing denoiser 3mem as in 77). We will therefore consider the training loss (5) with the model class specified by Lemma 2.1 and study the role played by the model capacity M function of the data complexity K, the dimension d, and the number of training samples N.
and μ x² for i [N] we
Memorization and generalization. We now aim to quantify our notions of memorization and generalization. Quantifying memorization generative models is complicated issue and many metries have been proposed to evaluate in practice. We adopt a popular, relatively strict metric for memorization, proposed by Yoon et al. 2023].
Definition 2.2 (Memorization): Given a datasel (a²) a small absolute constant E (0,1). 1), and the outpur distribution it of a diffusion model, we say that sample à ~ it is memorized if || x: (1) 1 S cl (2 [2, where (:( is the k-tth nearest neighbor in Eg norm to in (a*) )
On the other hand, generalization is easily formalized in statistical learning terms (in contrast to, e.g., assessing creativity of practical image diffusion models), see Appendix [A] In experiments, we estimate the generalization error on a held-out set of samples from n., which we can freely generate.
Given these quantitative definitions, the key question that we investigate in our laboratory is:
For a trained denoiser Bg, can we quantitatively predict, based on problem parameters M, N, d, and K, whether it memorizes generalizes?
By the phrases "the denoiser memorizes/generalizes we mean that its associated output distribution स produces memorized samples samples (approximately) from the true distribution, defined above. As we will immediately see, it is also fruitful to understand these behaviors as स being "close" to the empirical distribution - of the training set (ax ) (=r)
or the target distribution n, respectively.
Scaling the number of samples. Under the Gaussian mixture model v, that generates the training data (xr) ) the parameters and dl (together with the means f and covariances Σś) control the geometric complexity of the data. Relative to these measures of complexity, the scaling of the number of samples N plays fundamental role in the dichotomy between memorization and generalization behavior we seek to establish. We will focus on the case where N = poly(d), which argue below is a correct scaling in which to study memorization and generalization.
For assessing the similarity of probability distributions, it is standard to use the 2- 2-Wasserstein distance W₂ Blau and Michaeli 2017]. We will argue that when N = exp(d log d), there no meaningful distinction between memorization and generalization in sufficiently high dimensions. For simplicity. assume all covariance matrices Σ in the definition of π. in ( are full rank. Then by Weed and Bach
2019 Theorem 1, Proposition 7J. we have for all 2 that EE 22 m/\ ]] S Co. I/2d < Co/ d. for a constant Co 2 0. Therefore, W2(T,,, <.²) + 0 and we cannot distinguish the true distribution T. from its samples NN memorization and generalization are equivalent. On the other hand, using (Weed and Bach 2019 Theorem 1, Proposition 2]. one has for any draw of the empirical measure that W2( > 2-2 for a constant Ci 2 O. In particular, if N poly(d) then the Wasserstein distance is lower bounded by constant for any dimension, implying a meaningful distinction between memorization and generalization.

---

Notation (KE) (n²) 1=1:00 F+(1, F4)
Interpretation
Data generating process parameters
Notation
Interpretation
Learned model parameters
N
Number of samples
MMSE denoiser, data generating process
M
Capacity of the model
To (t aA.)
MMSE denoiser, learned model MMSE denoiser, empirical distribution Partially memorizing denoiser Softmax weight vector
K
Number of modes Problem dimension
meem Øt)
d
2 pmmmm, M (t, 22:)
Table 1: Data and model scalings.
w
Table 2: Data and model parameters.
Training Losses and Memorization
Now that we have set up the framework. turn answering the following fundamental question: when do trained denoisers (i.e., approximate optimizers of (5)) memorize, and when do they general- ize? One first approach to answering this question would be to directly examine the critical points of the diffusion model training optimization problem (5) w.r.t. the denoiser parameterization (9). However, this is not straightforward; the problem is non-convex and there many spurious critical points. Therefore, we posit a hypothesis which, roughly speaking, says that the training losses of surrogate memorizing denoisers with M parameters and generalizing denoisers is all that matters for predicting the behavior of trained models with M parameters.
To formalize this hypothesis, we will define two simple surrogate denoisers whose training losses will help us identify transitions between memorization and generalization as a function of the models' underparamettrizzaion.. The first of these two summary models is the generalizing denoiser æ.. The second is a partially memorizing denoiser pp which memorizes the first M training samples
Fpmem, MM (1, Σ xt (ollmax((o(12xx))) where we((, --5
-- xz 1/////0 (10)
Note that here wi I X Rd - RM, and in the case where M = N, it holds Pmmem. = Zmem- Therefore, we can formulate our hypothesis as follows:
There exists a loss weighting À such that a trained denoiser Ey with M parameters memorizes and only if LN M, A) S Lx (x X).
To attempt to verify this hypothesis, we will estimate the (excess) training losses of the two denoisers Tmmmm, Mr and T. and compare them to develop a criterion for when trained denoiser memorizes. For tractability, we focus on special case of the general mixture of Gaussians model (8) where each mixture component has identical covariance, equal to a constant multiple of the identity matrix (i.c.. Σ = I for each E [K] for the ground truth model (8) and Σ' = o²I for each E [M] for general denoisers Øg), reflected in Table 2 This is common assumption in the theoretical
literature on diffusion models [Shah et al. 2023 Gatmiry et al. 2024 The structure of the learned
denoiser Ee implied by Lemma] 2.1 simpl to:
M Σ
of
Ze (1, x:) = 0²'02 + oz
04 +
4 softmax(w((x)))
(11)
0F +0t
where
Wy (2:) - 1la wy 12/120/00 + of) i= M,
(12)
with the analogous structure for C. x m ..... ue K o?) In contrast to 11.. the form of the memorizing and partially memorizing denoisers 7 and (TOp does not change. Note that when M 2 K, the denoiser By with M parameters can songgenee generalize, and when M 2 N. it can memorize
For our theory, the choice of the subset of samples we use in the partial memorizing denoiser is irrelevant, as long as its size is M and it is chosen independently of the samples.
3 As we show rigorously in Appendix B using properties of the softmax function, the loss 55) any M- parameter denoiser & &in" .....,,.. =² can be achieved by sequence of A + 1 parameter denoisers. Therefore for a fixed value of M 2 K. a denoiser with M parameters can be (arbitrarily close to) the generalizing denoiser, ie.. denoisers with arbitrarily many parameters can generalize, and similarly for memorization with M > N.
5

---

Training Loss, the 6.704
Training losses of memorizing and generalizing denoisers. In what follows, we are going to state our main theoretical results which characterize the behavior of two denoisers interest. More precisely, we compare the train- ing loss (5) computed for the K- parameter generalizing denoiser 2. with the training loss ob- tained by M -ppaameter par- tially memorizing denoiser (10), with M K. To do this we com- pute the high dimensional asymp- toties of these denoisers' excess training losses. Note that these asymptotics provide approxima- tions (annotated with hats, i.e., f...)) which agree well with ex- periments at even moderate dimen- sions (see Figure D and Section 4).
10
10
10-
LN, (Fpmem,
10 10-4
Lx,a
CN.eeFpmem..l NN (F
10-6 -
10
CN,((E,)
10 10-10
10
0.00
0.25
0.50
0.75
1.00
M/N
Figure 1: We observe a remarkable degree of agreement between our loss approximations and the empirical losses. A simulation of the loss of the partially memorizing and generalizing denoisers æ. and & pmeem, M high-SNR value of f, with d 50, K = 12, and N 200, and compare them to the approximations introduced in Theorems 3.1 and 3.2 We use a constant 2 the e(.) expression
appearing in Theorem 3.2 which we can prove is also an upper-bound. bound.
We recall the standard re-expression of the objective in
via the orthogonality principle: for any t,
N
N
(x,) =
E
[ - a mem (t, x;)1
EE
@ (t, Xi) - x²11²) .133
N
N
1
E
Xt) mmem (t. x()l
(Tmem).
where meem is the memorizing denoiser 7 and Xj is distributed as initialized with wf, i.e., X; axx' + Z. where Z ~ I).
Our first result fully characterizes the expected excess training loss of the generalizing denoiser T.. given by (IT) with / = (p....., ........, μ oo), over draw of the random i.i.d sample (x',... x from T., under an assumption that the cluster centers f.f are well-separated in distance. For simplicity, we state our results in the regime where max; || = (d)) and o² ( Denote the signal-to- noise ratio (SNR) of the noising process (I by wh (0, 1) + (0, +00), where w o which is assumed be decreasing, and its inverse by <: (0, +00) (0, 1).
Theorem 3.1: Assume that N (dly(d), mink || n. 2 = (d)), max; || Odd) and o = (1). Let k(d) <(O(log(d)) We have that uniformly on t [0, x(d)
do
E (ā.,)(..) L,, = (,r smm
In particular, the cradeng-order coefficient for the right-hand side is 1.
Although Theorem 3.1 is stated for the excess training loss, our proofs further establish high- probability bounds on the behavior of the excess loss of the same order of magnitude. Moreover, in the scaling regime for o² and ( treated Theorem 3.1 is casily shown that for practically- relevant choices of ar and oe, for example the scheme aa --- L_, a, = 1 that we use in our experiments in Section 4 x(d)
K
- co. As consequence, for sufficiently large d, the uniform control of the risk for t E [0, (x)) that is established in Theorem 3.1 limplies uniform control of the weighted integrated loss (5) via (13).
1
"To motivate this regime, note that in intuitive terms, images sampled from such a n. are centered at a nominal image whose pixel intensities are normalized to Jo, 1]. and the variability due to the Gaussian noise can change each pixel by constant amount.

---

Our second result estimates the excess training loss of the partially memorizing denoiser 2 Ppmem,,
Theorem 3.2: Assume that N poly(d), mingge ll - u I| ( max || (dd)
and _f = (1). Let re(d) = <(O(log(d)//d)) We have that uniformly on t E [0, k(d))
( - L.. [ (0
E
do
(or (o=))=))
In particular, the leading-one order coesficient for the right-hand side is between 1 and 2.
Note that by following the proof, we can also derive a corresponding high-probability bound.
When does each denoiser have lower training loss? Comparing Theorem in high dimensions (d -> oo) estimate the training loss difference
3.1 and Theorem 3.2
[CN Mr A) Cv( A)] R x(t) f (mr)) A
(c(1- M
d v4. + +1/
1
do
where C' [1, 2] is constant. Notice that this expression is monotonically decreasing M N, so there exists a "crossover point" M. such that the (approximate) training loss of the partially memorizing denoiser is higher than that of the generalizing denoiser for M I M,, and lower for M 2 M.. Solving for M., we obtain
E (Ix M,, [x), 20 - M, R N / Er (ACt)((rr0 + +1]] CR((A(0)
f (14)
(
which linear function of N. This criterion provides a test for our hypothesis: if we believe that there exists loss weighting À such that the memorization and generalization properties of trained denoisers are controlled by the location of the crossover point w.r.t. X, then we should see an approximately linear relationship between the location of memorization and the number of samples. We conduct this test in Section in the immediate sequel.
Experiments
In this section, illustrate the versatility and efficacy of our laboratory through experimental analysis. First, we show that inside the setting of our laboratory, trained models exhibit a phase transition from generalization to memorization. Specifically in the isotropic Gaussian mixture setting, our main result builds a predictive model for the onset of memorization, and shows that it resolves to simple linear fit in (4). which validates our hypothesis from Section 3 Finally,
showease the flexibility of our laboratory by studying a low-rank Gaussian mixture setting which is constructed imitate the structure of training a denoiser natural images, yet still belongs the setting of our theoretical laboratory and exhibits similar phase transition behavior as in the simple isotropic case.
We provide further details and more results, including mechanistic analyses, Appendix H
High-level experiment details, and memorization criterion. For training, we use the training loss (5) with the loss weighting X(t) = (a./ar)² (i.e., "noise prediction"). For sampling, we use the DDIM sampler (Song et al. 2020], more precisely the implementation suggested by (De Bortoli et al. 2025), J with the "variance preserving" coefficients
ví 11-12 1 and the time interval (e, e] uniformly into L+ I timesteps (40) where = 10-3, and use these timesteps for both training and sampling. For completeness formally describe the end-to-end procedure in Appendix H.2 We consider the memorization criterion introduced in Definition[2..1 with constant = 1/9, i.e., is memorized if || =12 < (1/9)] a x: (2))) where å the generated sample, and (k) is the ttth closest point to & in the training dataset. Then, we say that denoiser is memorizing on average if its memorization ratio is > 1/2, i.e., at least 50% of the samples à drawn from the associated output measure are memorized; we say has started the phase transition if its memorization ratio is 2 1/10 and ended the phase transition if its memorization ratio is 2 9/ 9010.
t. We discretize
7

---

Memorization Ratio
Truining Loss, V 6.704
Test Loss, V 6.704
1.0
10
100
0.8
0.6
Lxte
Lv (epeaaanemeae
60
C, (F
0.4
40
L, (3mem)
0.2
Cxal(e))
(.(..)
0.0
0
0.2 0.4 0.8 1.0
0.0
0.2
0.6
0.8
0.0
0.2
0.4
0.6
0.8 1.0
M/N
M/N
M/N
Figure 2: We observe clear phase transition between from generalization to memorization in trained models. Left: A plot of the "memorization ratio" the ratio of generated samples that memorized over a set of Neval
50 generations - as number of components of the trained denoiser increases. The start and end of the phase transition, as formally defined at the heginning of Section 4 are bracketed by the gray shaded region. Middle: A plot of the training loss of the trained model in terms of the number of components. compared to partially memorizing. fully memorizing, and ground truth denoisers, with SNR the 6. 6.704. Right: A plot of the test loss Ls, estimated over a hold-ou set. the same setting, We observe that there is a phase transition from generalization to memorization, visible through the behavior of the memorization ratio and loss plots; the model stops generalizing (i.c., having an acceptable test loss) only when it starts memorizing (i.c., generating samples which approximately contained in the training data).
representative
0.3
4.1 Experiments for an Isotropic Gaussian Mixture Model
First, in Section 3 consider the target measure т, be an isotropic Gaussian mixture model, namely, n. = (1/ K) Σ Nir, of As emphasized in the rest of the work, we will study what happens when we use denoiser Øy corresponding to an isotropic Gaussian mixture model with different number of components M, with the parameterization given by [II
Existence of a phase transition. The first and guably most critical property of the trained denoisers in our laboratory is that there exists phase Iran- sition from generalization to memorization as the model size increases, which we show in Figure / Namely, we observe in the rightmost panel that ini- tially, heavily underparameterized trained denoisers exhibit statistical generalization (i.e., similar train and test loss the ground truth denoiser F.);; then, as the model size M increases, we observe rapid transition to memorization where generated sam- ples are memorized (nearly) 100% of the time (center and left panels). We emphasize that this replicates the central observations of many empirical studies of memorization in diffusion models, such as Zhang et al. 2023
Loss Weighting
0.8
0.6
0.4
0.2
10
10
10 10a
10
V
Figure We can accurately predict phase transition using our loss approximations. The optimal loss weighting per (15) using normalized approximate losses. Train and errors < 2 10
when the regression targets are 10º.
Kadkhodaie et al. 2023
Predicting the phase transition. Next, we show (via Figure 3) that the phase transition is pre- dictable using only the approximate training losses derived in Section 3 namely, it serves as the "crossover point" of the integrated approximate training loss CN >) for particular timestep weight- ing A(t), and this crossover point is a linear function of the number of training samples N. To show this, compute x(t) follows. First, create grid of (N, d, K) tuples and train several denoisers in each setting in order to estimate the location of the phase transition Mpr d, K), the first M such that the trained denoiser &e is memorizing
average. Then, solve the following optimization problem in the loss weighting À to make Mpc (N, d, K) close to the crossover point:
Mp (N, d, X) N
Mp d, K)
min Σ (N.d.K)
(15)
N
8

---

@ T. Figure 4: Our memorization laboratory enables modeling of natural image distributions with latent low-dimensional structure. Some sample synthetic images from our simple image model (bottom) visualized alongside their corresponding monochromatic image templates (top).
where (pc(NV d, K, X) the nearest M the crossover point with loss weighting X, i.e.,
Mp (N, d, K, arg min M
Σ X(t))Cx, (N.dK)) Lx,
The optimization and normalization details are postponed to Appendix H.3 In Figure we report that the train error and test error (evaluated on holdout set) are less than 10 signifying that we are able to compute the location of the memorization phase transition within one or two M's s on average, making our predictive model extremely accurate. Moreover, the recovered Mр is always linear function of N, namely Mpa d, K, X) (4/5)N. Therefore, Mpr (N, d, K) z Mpt (N, d, K, X) = (4/5) demonstrating experimentally that the consequences of our hypothesis (e.g., (14)) do indeed hold. Overall, the cetplliiiatiio-mmmizzatti phase transition is effectively predictable via our hypothesis and subsequent theoretical characterization of the loss.
4.2 Experiments for Simple Image Model
In the previous Section 4.1 we considered data which belonged to an isotropic Gaussian mixture model. To showcase the versatility of our Gaussian mixture model formulation, construct (low-rank) Gaussian mixture model whose samples resemble natural images. Namely, we consider a flattened and monochromatic d x d image ("template") x, E Rdd Now, we endow x, with color sampled randomly where @ is the Kronecker product. Defining the matrix Ag == @ x we have = Az. c. The colored image Y E Roxdxd is obtained by reshaping y. Since N(u,, I) and y = Ag. c, holds that y ~ ((A,, u. A.. A.,A))... Note, in particular, that y is low-rank Gaussian random variable. By combining multiple templates and color distributions, we obtain a mixture of low-rank Gaussians for our data distribution, namely, y T. (1/K) > i= W(A "a,,"'''244.
N(u.,ofI) R The colored output y E Redd is given by y
A
This an instance of Equation (8) and so we can compute its denoiser via Lemma 2.1 In Appendix [..4] we discuss some ways to efficiently implement this class of low-rank denoisers. We visualize some samples in Figure 4 with FashionMNIST templates (Xiao et al. 2017).
Our main result demonstrates that our framework is still expressive enough to model this facsimile of real-world image distributions, and that exhibits similar phenomena to isotropic case, enabling it to be studied in detail via our laboratory. Notably, we show in Figure 5htht we can still identify phase transition phenomenon, which looks qualitatively similar to the isotropic Gaussian mixture model studied in Section 4.1 and throughout the work.
5
Related Works
Understanding memorization and generalization properties of diffusion models crucial for practi-
tioners SSomepalli et al. 2023 Ren et al.] 2024, Rahman et al. 2024 Wang et al. 2024a Chen et al.
2024b
Stein et al.[2002 Indeed, concerns about privacy (Ghalebikesabl et al. 2023] Carlini et
2023
Nasr et al. 2003 and copyright infringement (Cui et al.) 2023 Wang et al. 2024a Vyas et al.
2023
Franceschelli and Musolesi 2022 are key issues as these models are deployed. Several factors
can in infuuece the memorization capabilities of diffusion models such as duplication [Carlini et al.
2023
Ross allaal.2220 2024 Webster 2003) or model architecture [Chavhan et al., (2024). We refer [Gu
al.
2023
Kadkhodaic et al. 2023 for an in-depth experimental investigation of those issues.

---

Memorization Ratio
Truining Loss, v 6.704
Test Loss, V 6.704
1.0
10
100
0.8
0.6
Lxte
Lu (apoeap
60
Ce (aFppmenni
0.4
(Buum
40
L.(
0.2
Lus(
Ca(mo)
0.0
0
0.2
0.4 0.6 0.8 L.0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
M/N
M/N
M/N
Figure 5: A phase transition persists in the low-rank Gaussian natural image model. Left: The memorization ratio the number components of the trained denoiser increases while everything else is fixed. Middle: The training loss of the trained model in terms the number of components, compared to partially memorizing. fully memorizing, and ground truth denoisers, at representative t z 0.3 with SNR ab as 6.704. Right: The test loss, computed over hold-out set, in the same setting. We emphasize the identical qualitative picture to Figure 2 wherein we observe a phase transition from generalization to memorization from the memorization ratio and loss plots. Transient "jaggedness" may explained by larger variance in the loss estimation.
On the theoretical side, memorization in diffusion models has been investigated through the lens of statistical physics leveraging the concept of phase transition [Biroli et al. 20244] Li et al. 2023 Ambrogioni 2023 Ventura al. 2024 Raya and Ambrogioni [204 (aakamoto et al. [024 Pavasovic et al.] 2025 In particular in [Biroli et al.] 2024], J. the authors identify three critical transitions for the diffusion model generative trajectories assuming that the score assumed to be perfectly learned. George et al. 2025 investigated generalization and memorization in trained random features neural network denoisers (nonparametrie models) in the case where the data a standard Gaussian. Our work is complementary to the theories of "creativity" and generalization of Kamb and Ganguli [2024), Nicdoba et al. (2024), Vastola [2025), which suggest in different contexts that generalization in diffusion models arises from the implicit bias of underparameterizee denoiser (Vastola 2025
also considers the landscape of the training objective, see [Bertrand et al.) 2025 for a rebuttal). Our work disentangles the competing factors in the implicit biases discussed in these works and captures the essential features into our theoretical laboratory. While in this work we only study in detail the simplest and most interpretable instantiation of the laboratory. further connections from framework to such previous work may be possible and enable us to broaden the scope of practical settings for which we have robust theoretical results.
6 Conclusion
In this paper, we have introduced a theoretical laboratory for measuring and predicting memorization and creativity diffusion models. Focusing on data drawn from K K-component Gaussian mixture models and Gaussian mixture denoisers with number of components ranging from K to the number of training samples N, our laboratory disentangles diffcrent factors contributing to memorization and generalization, enabling to compute tight theoretical approximations for the losses of representative denoisers within this model class and thereby predict the onset of memorization in trained models at inference time.
While our current framework allows us study generalization and memorization behavior in a rigorous and testable setting, we highlight several avenues of improvement. First, our model can be extended to capture additional properties larger and more realistic datasets such as intrinsic dimensionality or partial data replication. In future work, we plan expanding the memorization/- generalization laboratory to cover those cases and refining the asymptotics our training losses. We envision the laboratory growing to encompass framework for understanding memorization and generalization purely in terms of the geometry of the data, with a robust and extensive experimental apparatus to test hypotheses and verify predictions, and a rich theoretical toolkit that reduces statistical questions of sampling in trained diffusion models to geometric questions about the data itself.
Acknowledgements. YM acknowledges support from the joint Simons Foundation-NSF DMS grant #2031 1899, the ONR grant 100014-22-1-2102,, the NSF grant #2402951, and the startup fund from the University of Hong Kong.
10

---

References
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 35:36479-36494, 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22665- 2023.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Aher, Andrew Borst, Robert Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089- 1100, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06115, 1(2):3, 2022.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:2302. 2023.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035.
2023.
Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, Yue Xing, and Jiliang Tang. Diffusionshield: watermark for copyright protection against generative diffusion models. arXiv preprint arXiv:2306.0400 2023.
Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, and Yi Yang. Replication in visual diffusion models: A survey and outlook. arXiv preprint arXiv:2408. 00000, 2024a.
Nikhil Vyas, Sham M Kakade, and Boaz Barak. On provable copyright protection for generative models. In International Conference on Machine Learning, pages 35277-35299, PMLR, 2023.
Giorgio Franceschelli and Mirco Musolesi. Copyright in generative deep learning. Data & Policy, 4: e17, 2022.
Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emer- gence of reproducibility and generalizability in diffusion models. arXiv preprint arXiv:2310.05264. 2023.
Zahra Kadkhodaie, Florentin Guth, Ecro P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptive harmonic representation. arXiv preprint arXiv:2310.0255: 2023.
Mason Kamb and Surya Ganguli. analytic theory of creativity convolutional diffusion models. arXiv preprint arXi::2412.20292, 2024.

---

Matthew Niedoba. Berend Zwartsenberg, Kevin Murphy, and Frank Wood. Towards mechanistic explanation of diffusion model generalization. arXiv preprint arXiv:2411.19339 2024.
Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:23 10.02664, 2023.
TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest K Ryu. Diffusion probabilistic models generalize when they fail to memorize. In ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling. 2023.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memo- rization in diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse C Cresswell, and Gabriel Loaiza- Ganem. A geometric framework for understanding memorization in generative models. arXiv preprint arXiv:2411.00113 2024.
Hanyu Wang, Yujin Han, and Difan Zou. On the discrepancy and connection between memorization and generation in diffusion models. In ICML 2024 Workshop on Foundation Models in the Wild, 2024b.
Yunhao Chen, Xingjun Ma, Difan Zou, and Yu-Gang Jiang. Towards a theoretical understanding of memorization in diffusion models. arXiv preprint arXiv:2410.02467 2024a.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs Douze. A self- supervised descriptor for image copy detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14532- 14542, 2022.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40); 24652-24663, 2020.
Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. Advances in Neural Information Processing Systems, 36:9422-94555 2023.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metries and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312.14550 2023.
Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Mare Mézard. Dynamical regimes of diffusion models. arXiv preprint arXiv:2402.18499 2024.
Yu-Han Wu, Pierre Marion, GÄšrard Biau, and Claire Boyer. Taking a big step: Large learning rates in denoising score matching prevent memorization. arXiv preprint 3XXi:22502.03435 2025.
John Vastola. Generalization through variance: how noise shapes inductive biases in diffusion models. arXiv preprint arXiv:2504. 12532, 2025.
Peng Wang. Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02226, 2024c.
Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Denoising score matching with random fea- tures: Insights on diffusion models from precise learning curves. arXiv preprint MrXiv:2502.003340 2025.
12

---

Jascha Sohl-Dicksttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pages 2256-2265. pmlr, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Advances in neural information processing systems, 33:6840-6851, 2020.
Ruiqi Gao, Emiel Hoogeboom, Jonathan Heek, Valentin De Bortoli, Kevin P. Murphy, and Tim
Salimans. Diffusion meets flow matching: Two sides of the same coin, 2024. URL https ceiffussionflo..ithhbiie
Herbert Robbins. An empirical bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, Vol- ume 1: Contributions to the Theory of Statistics, volume 3.1, pages 157-164 Uni- versity of California Press, January 1956. URL http:://projjetteuclldd.oor
@ebooks/
Proceedings- the An http:///..a......oooooooooooaa/a
and/
Binxu Wang and John J Vastola. The unreasonable effectiveness of gaussian score approximation for diffusion models and its applications. arXiv preprini arXiv:2412.099726 2024.
Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the DDPM objective. arXiv [Cs. July 2023.
ab//237.001178 01178
Yochai Blau and Tomer Michaeli. The perception-disstrtion tradcoff. arXiv [cs.CV], November 2017. URL http:://rxix......///// org/ abs 06077
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance. Bernoulli: official journal of the Bernoulli Society for Mathematical Statistics and Probability, 25(44))2620-2666 2648, November 2019. ISSN 1350-7265. doi: 10.3150/18 15500/1--EJ1066 1065. URL httt:pss//projjecttuullidd org/
Khashayar Gatmiry, Jonathan Kelner, and Holden Lee. Learning mixtures of gaussians using diffusion models. arXiv [cs.LG], April 2024. URL http:://tax:iii/ www.//g///////////o///.//...ee// /ebbs/2444.. 200..1414868 18869
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Valentin De Bortoli, Alexandre Galashov, Swaroop Guntupalli, Guangyao Zhou, Kevin Murphy, Arthur Gretton, and Arnaud Doucet. Distributional diffusion models with scoring rules. arXiv [C), February 2025. /ttt:///aaxx hatt::///arra..oor///////5..
22480
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint 17Xiv:1708.00747, 2017.
Jie Ren, Yaxin Li, Shenglai Zeng, Han Xu, Lingjuan Lyu, Yue Xing, and Jiliang Tang. Unveiling and mitigating memorization in text-to-image diffusion models through cross attention. In European Conference on Computer Vision, pages 340-356. Springer, 2024.
Aimon Rahman, Malsha V Perera, and Vishal M Patel. Frame by familiar frame: Understanding replication in video diffusion models. arXiv preprint arXiv:2403. 19593, 2024.
Chen Chen, Enhuai Liu, Daochang Liu, Mubarak Shah, and Chang Xu. Investigating memorization in video diffusion models. arXiv preprint arXiv:2410.21663 2024b.
Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint arXiv:2305.08694, 2023.
Ruchika Chavhan, Ondrej Bohdal, Yongshuo Zong, Da Li, and Timothy Hospedales. Memorized images diffusion models share a subspace that can be located and deleted. arXiv preprint arXiv:2406. 18566, 2024.
13

---

Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36:2097-2127, 2023.
Luca Ambrogioni. The statistical thermodynamics of generative diffusion models. arXiv preprint arXiv:2310.17467, 2023.
Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni. Mani- folds, random matrices and spectral gaps: The geometric phases of generative diffusion. arXiv preprint arXiv:2410.05898, 2024.
Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Kotaro Sakamoto, Ryosuke Sakamoto, Masato Tanabe, Masatomo Akagawa, Yusuke Hayashi, Manato Yaguchi, Masahiro Suzuki, and Yutaka Matsuo. The geometry of diffusion models: Tubular neighbourhoods and singularities. In ICML 2024 Workshop on Geometry- grrouddy- Representation Learning and Generative Modeling, 2024.
Krunoslav Lehman Pavasovic, Jakob Verbeek, Giulio Biroli, and Mare Mezard. Understanding classifier-free guidance: High-dimensional theory and non-linear generalizations. arXiv preprint arXiv:2502.07889, 2025.
Quentin Bertrand, Anne Gagneux, Mathurin Massias, and Rémi Emonet. On the closed-form of flow matching: Generalization does not arise from target stochasticity. arXiv preprint arXiv:2506.03719, 2025.
Sanjoy Dasgupta and Leonard Schulman. A probabilistic analysis of EM for mixtures of separated, spherical gaussians. Journal of muchine learning research: JMLR, 8(7):203-226, 2007. ISSN 1532-4435,1533-7922 1533-798.. URL/https: jmlr.. /tgp/ppper/////assspptaaaaa/ html
Rong Ge, Holden Lee, and Andrej Risteski. Simulated tempering langevin monte carlo II: An improved proof using soft markov chain decomposition. arXiv [cs.LG), November 2018. URL
http://arxiv.on
Marvin Li and Sitan Chen. Critical windows: non-asymptoric theory for feature emergence in diffusion models. arXiv [cs.LG], March 2024. /tt/attp://arxi http:::///axiv.oo/abbbbb wwwg////////eeeeeeeo 2403. 01633
Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In 2011 International Conference on Computer Vision. IEEE, November 2011. ISBN 97814577 1022997779999977777111 1015. doi: 10. 10.1109/ic..2011.6622777 http:// doi org/ 10. 1w11/119ccccccc 6126278
Stéphane Boucheron, Gábor Lugosi, and Olivier Bousquet. Concentration inequalities. In Summer school on machine learning, pages 208-240. Springer, 2003.
Pál Erdös and Alfréd Rényi. On a classical problem of probability theory. A Magyar Tudományos Akadémia Matematikai Kutató Intézetének Közleményei, 6(1-2):215-220, 1961.
Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alex Dimakis, and Adam Klivans. Am- bient diffusion: Learning clean distributions from corrupted data. Advances in Neural Information Processing Systems, 36, 2024.
Niva Elkin-Koren, Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy? arXiv preprint arXiv:2305. 238. 2023.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, et al. The elements of statistical learning, 2009.
Peter L Bartlett. Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117749):30033-30070 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bari-nar trade-off. Proceedings of the National Academy of Sciences, 116(32) 2019.
14

---

Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-2653 2022.
Druv Pai. Diffusionlab.
2025.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565-55588 2023.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper is a theoretical paper aimed at providing theoretical framework for the study of memorization in diffusion models. It provides the full study of a partially memorizing model within that framework and show that the crossover that is identified can be mapped to a phase transition behavior. These claims are clearly presented in the abstract and demonstrated in the rest of the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the current work are discussed the conclusion. In particular, we highlight the limits of memorization laboratory well as the current limitation of our predictive framework.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All results are rigorously proven in the supplementary material as announced in the introduction. In order to guide the reader, we provide a sketch of proof in
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are given the appendix. The details of our memo- rization laboratory are also discussed in the relevant Section 2 Overall, the two datasets consider (Mixture of Gaussians and the stylized image model) are described at length. Our training setup. including losses and parameterizations is also fully described. In addition, we will open source the code.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be released with the supplementary.
15

---

Guidelines:
reproducible, they The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guide
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. www.cppllccccccccccccooc guwdes/iiee///e Sttps///SoooSuSississiieeemmmmee ssoliiolioiyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the hyperparameters needed to reproduce the results are specified in Appendix H In addition, in our released code, we provide training and evaluation scripts in order to reproduce our main experiments.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have statistical experiment in the paper which correspond to the regression to estimate the scaling between the memorization//enerrlizaaooo phase transition and the crossover of the training losses between the partially memorizing model and the fully generalizing model. In this case, we provide train and test loss estimated on holdout set. In Appendix H provide test bars for this regression experiment retrained on number of additional seeds.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: All the compute resources are described in Appendix H
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All authors have reviewed and agreed to the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
16

---

Answer: [Yes]
Justification: We strongly believe that our work is step forward a better understanding of the memorization and generalization in productionized diffusion models. Understanding on a deeper level the memorization of those models have a far-reaching impact from copyright infrigment data privacy. We believe that work could be the basis of further study with positive societal impacts, ensuring fairness in media generative models. We do not foresee negative societal impacts of our research.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper does not use existing assets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
17

---

Organization of the Appendix
In Appendix A we recall some basic calculations around Gaussian Mixture Model denoisers and pro- vide additional discussion of our setting. Justifications about our model class are given in Appendix [B]
Next, we recall some results on Gaussian concentration bounds in Appendix In Appendix [ present key results for approximating softmax operators. We combine our concentration bounds and softmax approximation results in Appendix B in order provide approximation of the denoisers in generalizing and memorizing scenarios. Our main results regarding the training loss approximations such as Theorem 3.1 are proved in Appendix F Finally, full experimental details are provided in
Appendix H
A Gaussian Mixture Model Denoisers: Basic Calculations
Justification of Gaussian Mixture Models. Gaussian mixture models represent a canonical model for assessing learning and sampling algorithms in theoretical computer science, including in the
context of diffusion models (Dasgupta and Schulman 2007 Ge al. 2018 Shah et al. [023 Gatmiry
itt al. 2024], and they have the appealing ability to model data with geometric structure, including hierarchical structure [Li and Chen 2024 and low-dimensional structure natural images (Zoran
and Weiss 2011 Wang et al., [024c] Most importantly, the task of training and sampling with diffusion model on a Gaussian mixture target n, via (5) represents an ideal test-bed for investigating issues of memorization and generalization, because as the number of components in the mixture varied, and in particular is as made as large as the number training data samples N, one can simultaneously represent the true distribution (8) and the memorizing denoiser 7 within the same class of models.
Discussion around the denoisers. The key technical challenges we investigate in our laboratory setting are the characterization of the denoiser &y minimizing (6) (or, equivalently its parameters 0), and using its properties to prove that its associated sampling measure Å either memorizes or generalizes. For the first challenge, typical theoretical studies of regression over a parametric class of models (as in (5)) rely the model class being well-specified with respect to the true model parameters, here those corresponding to ... In (5), this the case when M = K, but as M K, existing studies of the loss landscape for diffusion model training with Gaussian mixture model data/denoisers break down [Wang et al. 2024c| Shah et al. 2023). With regards to both N, the model class
challenges, a clear complication that have alluded to previously is that M
becomes expressive enough to represent the optimal, memorizing denoiser 7 which is suggestive behavior of
that at intermediate values of M, parameters that lead to seneralizing-denoistr are no longer prevalent. To overcome these challenges, we adopt the hybrid decoreticllempiiii methodology outlined in the main body, which our laboratory setting enables.
Basic denoiser calculations. We recall Lemma
Lemma A.I: Assume that To (1/M) ii= N( D.).
Then, we have that
M
Bo x.)
xt of Σ
a + + -xx, aspi)) sottm)offoaxtttioo wit, 21))s
at
where for all E [M] (t,
1 log det det(a,) oi' 2 / +
1 (x, (z1-o(p')) " 0; '2x; aqp²)
2
In the rest of this section, we are going to prove Lemma 2.1 This property is well-known and can be found in [Peluchetti] 2023 Biroli et al. 2024 Kamb and Ganguli 2024 for instance but we include
its proof for completeness. First, we compute pt. Recalling the noising process, ( we get that for any E [0, 1] and æ; e d
18

---

pr(xx)) 1 (2
(1/M) Σ J Prol=s/2o)\((' (')
M
= (1/M) Σ Naag "
Therefore, we get that for any / [0, 1] and ae E ERd
v (x,s) = /
Σ/ M 10, a (ap'' - (Z) of az
M E V20 of + ai
Σ (o + - (arp" xx) softmax (x(() (2x4)):
(16)
with uo defined as the lemma statement. Finally, using Tweedie's identity we get that for any t [0, 1] and Iz E Rd
V lo (a, (f, æ., =z)/o²
Therefore combining this result and (16), we get that for any t E [0, 1] and E gd
M
1
Et(t, xx.)
0c
of Σ
CajE + -0;1) -aepe) softmax(w((x:))
04
i=
which concludes the proof.
B Nesting of Model Classes
Here we will sketch a rigorous proof of the claim that the loss (6) associated to any M M-parameter GMM denoiser can be represented, in a limiting sense, by that of (M + +1)-prammete GMM denoiser. For simplicity, we will treat the case where Σ' = Σ for E [M]. i.e. all components have the same variance. Given such M -paarmmeerr GMM denoiser following the form in Lemmal 2. show that the M + parameter model given by the parameters (Hr', E...., HM Σ, m44 M Σ). for m E N, provides a suitable loss approximation as that it suffices to show a suitable degree of approximation of the softmax autoregression
co. Comparing the denoisers in Lemmal 2.1
[
M
myx] softmax (1 s [Az'
/ MM coftmax(wst(
where the weight vectors subscripted to denote the number of parameters. Because concerned with loss approximations, it suffices to show this approximation for æ; given by a noisy sample from the GMM u., following Equation (I). We have
1
n
M
mp M softmax (=+1((x)))
e (maquu x.) T [af (mo, -x.)
mm
M
-==1 e + e 1 (moqu - 21) T (mo M -ws)
MM ewee + e (maqp m. E+ - (imaauM-aa)
where write Wi for the k-tth entry of WM+1 above. From here, event (using Gaussian concentration) on which Xi is bounded for any sample x' initializing the noising process (coming from (6)), and then it follows by an application of Gaussian concentration that on this event,
construct probbbioyy
lim e M -x)) (a/ T m-000
(maas -w.)
19

---

whence by the previous expression
M
mpr"] (1 mm
[eet
n nM] softmax(ws)) (W (2t))).
Given that loss approximations for ( only evaluate on X; for which we construct the aforemen- tioned high-probability event, this establishes the claim that (M + -aarameterr models' losses can achieve any M-pamt p-raamettrr model's loss.
C Concentration Bounds
C.1 Chernoff-Cranee Cramér bound for X2
In this section, give some basic results regarding the concentration bounds of X2 random variables. Those lemmas will be key to establish our main sparsity results in Appendix First, we recall the Chernoff- Cramér bound, see [Boucheron et al. 2003, page 21] for instance.
Theorem C.1: Let X be a real-valued random variable, Let M(I) = E exp]tX]]] Then, we have that for any E I. P(X < <a) S inf1so (1) expl-ta]. In particular if X x2(p) with N,
we get that for any E (0, 1)
P(X & (1-=)p) S exp [L + log(1 -5)
(17)
and
P (E log(1 + E))
F(X 2 (1+ E1+) & exp
-
(18)
As consequence for any & E [0,1]
pE²
P[[ pl ž sp)
S 2 exp
(19)
8
Proof. We have that 17 and 18) are direct consequences of the Chemoff-Crammr bounds. Then for
/2 / 23 /6. Hence, any [0, 1], we have that log x) + x S -z² and log(1 + z) S x=²
we have that for any E [0, 11, we have that log - r) + S -x² and log + x)) x S - 1²² /8
which concludes the proof of (19) using an union bound.
One of the main application of Theorem C.I is establish the concentration of the squared norm of Gaussian random variables.
Lemma C.2: Let 9 ~ of be isotropic Gaussian with mean μ and covariance o² I. Then for any 0 < < 1, one has
P
|| - (1|m112 + ozdd) 2 sovalova [1ml1)] I de"pp_de²/8))
Proof. We have d μ + ow, where NNO,
| á || owl 2 = |||| o² ||||| + 20 ( w) d ||| + o² | 2 + 20 20( w)
|||² oz || + |0||||||
(20)
where the last line follows by rotational invariance of the Gaussian distribution, and therefore in particular
E || [ = ||||| + do².
By Theorem we have for every 0 5 8 5
IP [[]l|ur w 1|
2 ds} < 2 -dsp
and by Gaussian concentration, for any 2 0, have
P|[w1] < 2 exxpl- 12/2],
20

---

so in particular
evd
s 2xppl ds2//)
2
Thus, using a union bound, we have for any E [0.
P
(11a)1 + d2d) 2 covdlova+ || [1]l]]
P
w 2 + 2 201|u11 o'd| 2 covalovd [1l)
P
0² || || d 20 =oVdlovd (|
PP
0² |||||| d de] 2011|a1|
|w
N 0
4 expl -de//)2)))
which concludes the proof.
The following result will be used in the proof of Theorem
E3
Proposition C.3: Let (g)) be i.d Gaussian random variables ( a Id) and (w) a collection of positive random variables. Then, for any e E (0, 1), we have with probability at least 6n expl ds² /2]
do (1 32) s
Σ softmax(w),st x²
I 2do 3z).
Proof. First, we have that
(ottmax(w)jat x.
softmax(w)) (x) - x²
(itmma((w),((t+ -x=1
Therefore, using that x x is a Gaussian random variable (00 2o Id), get using union bound and Theorem C.1 that with probability enpeppp(edd)
softmax(w)) ( x
2do + +
For the second part of the proof, have that
softmax(w) f x
x
softmax(w)) x 2
softmax(w), (w) (x',"")
||=0"||2
softmax (a) (2)) a.")
||
Σ
softmax(wx) |x/ 1=1
n-1
soft -||=]1 *|*||<>>*||>
21

---

We have that with probability least 1 2n expl-de2/8 that |a|| I s.evé Combining this result with Theorem C.I and a union bound we get that, on an event of probability at least 1
4n expl-de de² /8)
Σ (atmaa((w)aa" x²"
softmax(w)))(l==]]
| | <
softmax (w) [1](|<==1 ev x
2 dd -)) -0 =) > d(1 -3s),
which concludes the proof upon using union bound.
C.2 Coupon Collector Bounds
In order to derive our results in the case of the partial memorizing denoiser, we will consider the following result from the coupon collector's problem.
Proposition C.:: Let K E be the number of means. Consider the distribution (I/K) Ek=1 N( I) and let (xr) be a collection of i.i.d samples from и. For any x denote E [K] the index of the associated mean, i.e. x' .. + o.w,, with w NO, I). Let S (ar") such that l. 2 (1 + log(d)) log(K) For any k [K], denote A the event such that there exists i1, 12 E [f] such that kig = ki Finally, denote A N Ag. Then
P[A] 2 K lox(d) (1 log( log (d)).
This result is a simple control of the tails of the coupon collector problem. In fact, much more precise estimates could be derived, see [Erds and Rényi [961 for instance.
Proof. We only deal with the case K 2 2. The case K is trivial. First, note that we have for any E [K]
t
2-1
P[
K
K
K
1
l
14
K
K
K-1
< expl-I/At
[+ K
Note that / + exp[-I/K]
R-T is decreasing on [K, +oo) and therefore we get that
R-I
P|A < expl- + (dg( [og(K)) (1+
K log( log(d)
K -
Using union bound and that K/(K - -1) 5 get
P|A 2 1 - 2K Ing(d)) (1 + (Kg(()) log(d)),
which concludes the proof.
D Softmax Approximation
low-Temperature Behavior. The following elementary lemma is useful. It shows that the key quantity controlling 1-sparsity the softmax the scale of the temperature relative to the gap between the largest and second- element of the softmax weight vector. For completeness
22

---

recall the definition of the softmax operation. For any a E R" we have that v E R and for any i [I....,n] n
exp(us] softmax(v); Ek=1 exp U4]
Lemma D.I: Let E Rd be such that vs Vj for j. Let k: arg ((...) UKe denote the index of the largest element of v. and define
min (tk v2),
as the "gap" between the largest and second-largest element of v. Then for any p 2 1, one has
[|soltma((o/T))>>>>>))))) - exllp S 2(n 1-en-11--71 1)e-
Proof. The idea is to notice that taking the ratio between elements the softmax has a simple expression, namely
softmax v vv/T); = e -v2--11)/1
softmax V/ (//T); where k is as in the statement of the lemma. By definition of k. we then have for every + k
softmax(v/T); S -/"" softmax(()) // v/TT) S Se>/
and thus
softmax(v/TT) 2 1 - (n 1)e 7/T
This gives
(roftax(v/T)) - ex1 s (n 1)ee .n/T + (n - 1)e-m/7 -//T
so in particular
|softmax(v/T)) - exl1,
-xx1,, $ (n - 1// 1 + (n - 1)-1+1/0
2(n 1)e-//1
Note that guarantee approximation in 8 in the worst case (reflected in the proof) it is necessary that the temperature depends logarithmically on the number vector elements n.
This proof operates in worst-case regime where every non-maxinne maximaxiaal element of the vector v may have the same magnitude. In reality, if there is more structured distribution non-maximizers, the estimates improve correspondingly. The proof could be improved to capture this by using different, more precise measure of the distribution of entries of v. For example, if some precise rate of decay of the entry distribution could be asserted, it seems reasonable that this, rather than the vector dimension n, would force the ultimate dependence of T for l approximation. seems reasonable that something like this should obtain for weight vectors of distances between random vectors.
The following lemma is a slight extension of Lemma
D.1
Lemma D.2: Let v E RD such that v + Un for i # j. Let k = arg max.r((.....) Up denote the index of the largest element of v. Let be a subset of [n] such that and denote
7S
min
(vk - uv),
#gS
as the "gap" between the largest of and the largest element not in S. Denote
exp(u./T]
softmax(vjs//T)
Eies explux/TT TT
if l and softmax(ns/T)) otherwise. Then for any p 1, has
|softmax softmax(us//T]), = (1 1S)) [S]) expl-ns//T
23

---

Note that Lemma D.1 is a special case of Lemma D.2 where S = {k}.
Proof. Let l 5, we have that
softmax(w/ softmax(0/T)) expl(-up + v;)/T] s expl-7s/TT
In addition, we have that for any l E
explue/T]
exp explve/T]
soflmax(M soflmax(1/
Eies exp 0i (T
Eies exple://0 w//T] Eigs exp(u:/T]
Eigs exp hcppet//1 /T] Esss exple; 02 Eies explu:/T1 + Eigs explu;/T [T]
explue/T]
softmax (aftmax(e/T)) (v/T): (n- [S|) expl /T/
Therefore, get that for any 2 (softmax(o/T)) softmax /T)sTT] - [ - |S| expl_>sppTT] + (n - (S]) cxpp-7
S (n - S)}² cxpl->ss//T[[( + (n |S1)-1+1/p)
(n S))² exp[-nsp/7]( + +S])
which concludes the proof.
E Results on High-Dimensional Denoiser Behavior
In Appendix E. we present some results which will help us control softmax approximation in Appendix
E.2
where leverage those results obtain controls on different denoisers.
E.1
Gaussian Mixtures and Softmax
The following lemmas are the key to establish our main results. They allow us to use our softmax sparsity results in the context of diffusion denoisers. In particular, Lemma E.1 will be used in Lemma E.4 while Lemma E.2 will be used in Lemma E.3
Lemma E.1: Given vectors (
satisfying
pr 4. 27
min kek
consider the distribution = (1/K) DK and let a ogg, where g ~~ ~ (O, is independent from x'. Let i; denote the index of the (uniquely defined) cluster centroid μt associated to x:. Then for any 0 S S satisfying the coupling condition
===1 I o For E [N], let x' ~ f 1,
85
4
Wd(azo?+ +o
one has with probability at least 1 - 4K expl exp/-de2/80
7²a²
lapp 1| kek,
min
Iaa
2
Proof. Start by writing x² L 1 to, w. where k; [K] is unique and w ~ ~N(0 I) is independent, then write, for any & E [K].
| - x || ||a:(ut - x') - - ||a: μ -. - =.gg
This equal distribution to the squared 8² norm of a Gaussian random variable with mean ac(p ) and covariance (afo? + +f))I. Applying Lemma C.2 it follows
P P|llaan x1|² -aaillk0 - - "/''l" + (afo? + df) 2 e=* S <3expl-
24

---

where for concision E +o
/ - + +a μ=: II). Taking a union
bound over E []] the above control holds for all simultaneously with probability at least 1 3K exp[-de² /8. More precisely, we have that with probability at least. 1 3K expl exp]-de2// /8). for any l E [K] with l +
(af0) +o))d) -eE", Hoou k x: -a -a²=(aai|lm² 1²
(21)
and
lasp - =l1 $ (1 E)d(a?E + +
Using (11). we have that for any t E [K] with + ki
| =||² (1- =)d[aaa +f) + aîl u k: 1 Ear eary d(azo? + o,,) k 1.
Consequently, we have on this event that, for each & E [K] with ks + ki
|ooeee x: 2 1oap - -=ll a² 1 11
2d 2d(aao? oz)) as
+ /
11:11)
To simplify this bound, notice that we have for all k E [K] with K + k;
of 2 Ilue - w.:1ll - eary
corydloo Mafoo? of) - w
a 1 11 11
4
if and only if
ar7 2 d( +
Similarly, we have for all + h;
a?
2 |l
u 1 2ed(o + +o+) 1 a
and only if
a17 2 /8d((af02 +
So is enough to enforce
max (e, Vej
a+7
4 /dlaço? oā)
get that on the aforementioned event, for every [K] with + kj
20|2 -laaa. x 20|2 7²07
min laap kgk,
2
which concludes the proof.
The following lemma is more involved than Lemma M Lemma E the softmax weights are investigating involve not only the means (+t) k= but the datapoints (x') > which are random.
The main reason is that contrary to
25

---

Lemma E.2: Given vectors (u satisfying
min Rik
Iut u 2 7 0,
consider the distribution x= (1/K) Ek= E [N] and I S 1, and let x ~ ax o.g, where g ~ NO I) is independent from (x²) )==1-
o. For euch j E [N], let x² n. Fix
Then for any 0 S satisfying the coupling conditions
04
2/
o + oof)
and
8² S
1 E 3 20
one has with probability at least 1 4N exp - d
min |aaad - zeI? - |1a00 - x0 =x0l 200?(1 - =))d.
j#l
Proof. We start by writing nf d per + a for each j E [N] where k; [K] unique and w (0, I) independent. We will first argue that is enough to consider only those indices for which the class assignments agree: that is, k; = kūs where we recall that æ ayæ' + d:g and therefore æ is associated with k; We have for any j E [N]
a a,o, aww ww') + are th 1 o.g,
so by rotational invariance
||aer 1/| d ad
- -/
11att 1| + |aaoo -w') - o.g'|
+ 2ar u w (aro - -wi) - -.g1).
(22)
In what follows, we consider the case + i.
first lower bound in the case of different means. First, suppose that ky * k; The random variable (w)
(wf wi) 0zg1 is equal in distribution to a Gaussian random variable with mean zero and variance 20 of. Call this random variable X; by Gaussian concentration, for any t 2 0, we have
P| 2 + s 2expl +o,))
so in particular
TP | /d(2afo +of)
2expl_de
get that with probability least 1 - 21(i [M] ky + k.)l exxp|-de2/80 dee//], we have for every E [N] for which ky * k; that
Combining this result with 22
1axx - x |2
k2
u/: 2
Jaro(ww w0) rlgll Eas ue," ky
u ||| d(2afo2+
2
a
μ
Given that
min k
F 1 2
if ouy seVd(2afo' + of), have the previous event
|or00 - 20 1-1² > of &s
1. I|² + Ilaso= - -w)) 0:g
uu
2
> [ - w²) ---0 -oigl?
26

---

Now, note that for those j for which k,- = kis we have as above
| - x 2 |laro, w') aigl?
Therefore, we get that with probability 22N expl-dee" (dee2/8) /8] we have that
|aaaa
x | w') agg||
(23)
Lower bound on the difference (first stochasticity level). We are going to give lower bound for j & on the quantity
I w²) -o.gll ||a - x:
We have for #
[0:00 (wo' - ww) - -oi9l - ||ar - z01² |1,00, - w²) - -0:9l' - |lag11
= a w01 20,0,0, (wi w w''g)
a,,0.11u' w 20,000091 /wo w'
where the last line uses rotational invariance of the Gaussian distribution. Once again using Gaussian concentration, we have that with probability least exp(-de" de /8] that 1 I eva It follows from a union bound that, on an event of probability at least 2NN exp(-ds"//) /8], it holds
2
|aaoa, - w') - o.g - -|aaar - ==|² 2 af w w ev evdo,a,,,1ww - -w'l.
Lower bound on the difference (second stochasticity level). Now, as before, if it holds for all such j
250 V a,0,
1 w0ll
then the preceding bound can be simplified to
año² 1/w w w''l?
lanalww' w0' -og)- ||ara -ax a0||
This leads to consider the lower tail of the random variable min i / will. which was studied in Appendix C A coarser approach will be sufficient for our purposes: when j + i, the random variable |wo w" 1|2 is distributed (2(d) random variable, so Theorem C. implies that for
any 0 E' 1,
d(e))²
P
P[l
w
w²| 2 2(1 - (()dd 2 exp
8
can simplicity simply enforce s' E. In this case, if we add the additional condition
c²
af
1-E
20
then by union bound, it holds with probability at least - N expl-de2/08 that for all such j.
afo? 2
|aro, (w" w²) oig || ||a, - =||²
1 - ww'l =''|222 af - =)dd
(24)
Finally combining (2
23 and union bound, we get that with probability at least 1
4N -Neppl expl-de2/
we have
min a æel2 - ||aræ* - = 22a002 E)d, ##1
which concludes the proof.
27

---

Lemma E.3: Given vectors (u satisfying
min Rik
Iut u 2 7 0,
consider the distribution x= = Ek= E [N] and I t S 1, and let x ~ axx o.g, where g ~ NO, I) is independent from (x² )
o. For euch j E [N], let x² n. Fix
Then for any 0 S satisfying the coupling conditions
04
2/ o + o
and
8² 0f
1 1 20
one has with probability at least 1 4NV expl de2 2
|| - x 1| aaxx - 2 1
og?²
min
JES:
2
where S,, where j E S, if ky ka, where k- is the (unique) index of the mean in (pb) associated with x².
The proof of this lemma is similar to the of Lemma E.2
Proof. We start by writing x & n.," +o,w" w for each j E [N], where ky E [K] is unique and
us ~ M(, I) is independent. We have for any j E [N]
a a awo - ww') + aelu - - - - a.g.
so by rotational invariance
|1aoo -od--an 20|2 d az u / || laaoos w²) - w
+ 20 ur by
/ (ara wi) - -.ggi)-
(25)
In what follows, we consider the case j * First, suppose that k, # hG- The random variable (wr - wi) oggn is equal in distribution to a Gaussian random variable with mean zero and variance 20 of Call this random variable X; by Gaussian concentration, for any / 2 0, have
P| 2 t] S 2expl /2200 + +,)),
so particular
sy d(22foo of)
P
X
2xxppl_d=")
we get that with probability at least 1 - 2/f E [M] ky + k:}l cxxp[=ee/8)] we have for every j E [N] for which k; + k; that
Combining this result with 25
20
2 2
a
μ.
11
(ww w') orgl| sat ue ks
u ||| /dlad(222 +z)
Given that
min Riglle
7
if ouy 22 / d(2afo? of), have the previous event
|1,000 - x 2112
at
-
ks u 1|² + [la -w0) a.g
2
2 (w. w')- -oool a?2?
2
28

---

Therefore, we have that with probability at least 1 2NV expl-de?/ / for any j $ S
- |a,x - ==|² 2 ||aro, w") org of??
(26)
2
Lower bound on the difference (first stochasticity level). We are going to give lower bound for j # i on the quantity
1aro -w') - -o:911 - ||ax0 - x
We have for j *
| (w² w') vo')-ooch -oi9l - 1|arx - Ø²|² l|a,o, (w - w') 0.g11 - |o
= a00021 100 w² (0nav,0(("* w a, w
2 20,000 1 w'
where the last line uses rotational invariance of the Gaussian distribution. Once again using Gaussian concentration, have that with probability least 1 - 2 cxp[-d= that |0i eva It follows from union bound that, event of probability at least 1 2N exp (oppppd=//1 /8], it holds
|ato. "eca,.w''. - w²) - o 2
|laa - ==|² 2 af
w w
w Ev EV doraro, / - will.
Lower bound on the difference (second stochasticity level). such j
Now, before, if holds for all
20a, v a,0
| will
then the preceding bound can be simplified to
año²
|araa w²) o I|ara - 2=|2²
1/00 w w''l
This leads us to consider the lower tail of the random variable min /u w'll. which was studied
in Appendix C A coarser approach will be sufficient for our purposes: when j بf the random
variable [w w is distributed as a (d) random variable, so Theorem C.1 implies that for
any S S 1,
P P[lw>
w''l 2 2(1 e'ya] 2 exp
-dd()))
We can for simplicity simply enforce e' In this case, if add the additional condition
E²
0f0²
1-8
20?
then by union bound, it holds with probability at least - N expp expp-de??
d=² dE2 that for all such j.
año?
|ovo,(w² / w²) oig||" ||a,æ" =||²
2 1/ -wwll w/|1 ajo0 =edd,
(27)
Finally combining (77 266 and union bound, we get that with probability at least 1
4N exeepp-ai exp[-de2 / we have
min lawo - m|| - |a, -x =0122affof1 - 8)d +d+++++ jes:
at?
which concludes the proof.
E.2 Denoiser Approximations
Finally, we prove Lemma B.4 and Lemma E.5
Those results control the approximation the true
denoiser in Lemma E.4 and the memorizing denoiser in Lemma E.5 Those approximations express that under certain conditions the true denoiser can be replaced by Gaussian denoiser and that under certain conditions the memorizing denoiser can be replaced by point in the dataset.
29

---

Lemma E.4: Given vectors (u satisfying
min Rik
jut u 27>0, 0,
consider the distribution π (1/K) Ek= N( o. For E [N], let x² NT. and let ot a,æ' + 0gg, where g ~ NNO, T) is independent from x² Let hi denote the index of the (uniquely defined) cluster centroid μl: associated x'. Define the nearest-neit neiiett-neighbor (one sparse) denoiser znom (t, xi)) associated to (t, æf) (recall Lemma 2.1 by
O407 añ + oj
of
no (t, x/)
at
+
ai oi k
Then for any 0 < E S satisfying the coupling condition
O
4 odd(a/o? + off)
one has with probability at least
4K expl-de?1)
2K max |
7² 4(a7o² + of)
(t. x (1, xi)ه
af of
exp
Proof. We apply Lemmas D.I and E.1
Our hypotheses let us apply Lemma d
E.1 which gives that
with probability at least 4K exp exp
2a
min lefk.
- - zill" 2 2
-1aaaa
Recall the expression for the denoiser (Lemma 2.1
12
The weight vector
1
wh (.)x))
2 |o.p
/a + +
is related to the gap condition have asserted: in particular with probability at least 1 4K exp/-de2/88
2²0z
min Wk, (x)) we wx(xx)) 2
4(afo2 + of)
Hence, an application of Lemma D.I gives that for any 2 1,
7 a² a
softmax(wwxx:)) - ek. 11, 2 - exp
exp
4(afo? + +o))
Using this result, we get
of
@.. (t, x/) - x nomi x;)
M(ex, - w(xi))
0/ + of
of
S 0.0? + 2
Tllex, w(xx))l ||
s 2 1)0
7²
0.0? + o;
exp
A(a + +
where we have defined
M [u]
E k
r
max
|
kE[K]
which concludes the proof.
30

---

Lemma E.5: Given vectors (u satisfying
min Rik
1ut u 2r>0, 0,
consider the distribution ग = (1/K) Ek=1
k= , I. For E [N], let x' 5xx S N (0, , 1) independent from x². Then for any 1
and let va a + ag where 9 satisfying the coupling conditions
2v
12 + +
and
8² S
0/
1 1 3
20
one has with probability at least 1 8NN expl de /8]
afo? - 2)d
@mem (t, xx)) æ'll < 2NT, exp
20
with
V / + ca. vdlvulluu11 (1 + E)O3d. kE[K]
r.
max
Proof. The proof is similar to that of Lemma E.4 we apply Lemmas D.I and E.2
Our hypotheses
let us apply Lemma E.2 which gives that with probability at least 4N exp de"
=i| - ||a,x' - xill 2 af a)d.
min
Recall the expression for the denoiser (Lemma O, 0, so the weight vector
12
The memorizing denoiser corresponds to
wy((x)) - | /0//
related to the gap condition we have asserted: in particular with probability at least 4N expl-de²/S
min wy (x)) wy (mx)) 2 a302(1 EE)d
207
Hence, an application of Lemma D.I gives that for any p 2 1,
af - 8)d
(x))) eill, 5 2 - exe
20
Using this result, we get
@mem (t, x:) x'|| |M w(xx))]1
S T ex. w(x{) |
a a2 s)d
s 21 - 1) exp
20
where we have defined
M x
x
Rax
r max
| ||
JENN
Finally, using Lemma
C.2 have that for any <
| 1 (1 2|2 + oad) 2 so, Vo Vd |1 I0]
< $3expl-de2/8)
so with probability at least - 4N expl- we have
<
max KE[K]
|| + to, Vdl put|l + (1++)07d
31

---

By union bound, we conclude that with probability at least 1 8N expl
d=² dd2///
/8]
that
af e)d
Zmem (1,, mz)) æ²1 2T 1) exp
20
for each i, which concludes the proof.
Finally, we derive similar results for the partially memorizing denoiser. We recall that o pmmem, M (t, xt) given by
ppmom M M1,,, (1, xz)) =
Σ
soltmax(w))a²²
jC
where wj
207 |la,
afl12.
Lemma E,6: Given vectors ( satisfying
1ut u | 12700
min Regeke
consider the distribution π = (1/K) Ek_, k=l o For E [N]. let x' ~ fix 1 1,
and let æt axæ' + Jgg. where g satisfying the coupling conditions
NO, 1) is independent from x'. Then for any 0 < <
2 d(2 + +o²
and
2² 0f 2
1 S
20
We consider two cases, first assume that E [8] then one has with probability at least 8N expl-de2//1 dc2/88)
afoo E)d
a pmem, M (t, 2'1 s 2NT, exp
20
with
r. max kE[K]
soo + (1 + s)o
Second, assume that & [C], with probability 1- 8NN expp [8] K log(d) (1 (l+log(As) log(d)),
S, = (j E, kj- k;s is not empty and
af?2 20
Epmem, M (t,
Σ jeS:
softmax(wls.),32
<T,(1+ exp
Proof. The first part of the proof where i E [4] is identical to Lemma E.5 i É [f]. First, using Proposition C.4 we have that on an event with probability 1 K log(d) (1
We now assume that
log( () log(d)) S, not empty. In addition, using a union bound we have that with probability at least 1 4N exp[-de' de? K log(d) (og(K) log(d)). S. {j l, ky k2} empty and
a??2
min | - 20 || aa, x 185;
2
Therefore, using Lemma D.2
we get that
af??
|softmax(w) (w) softmax(w (w|s 1)1 <1 (s.)) - [St]) exp
20
0272
< (1 + NY 1/N exp
20
32

---

Using this result, we get
Apmeem. (M (1,
Σ
sollmax(w0s)))
|M (softmaxx(ulss. softmax(w))
jES,
[|lsofmaxx(wss))
softmax w)
S (N + 1) exp
20
where we have defined
M
x
x²
E
Rdxt
max JEIN]
|
Finally, using Lemma C.2 have that for any 0SESI
|| | (1 A || + +o?d) 2 so.volo.va+ (|x [D] s 33expl_de2/8]
so with probability at least - 4N expl- d=? /8] we have
TST. = max K
+oll + + e)ofd.
By union bound, conclude that with probability at least K (og)(g + log(K) log(d))
8NV exp[-de? dd] that
a77²
pmmmm Apmem.M xt))
Σ
(s))
TT. N)2 exp
20
jES:
which concludes the proof.
F Training Loss Approximations
In this section, we give proofs of results that imply our main results stated in the main body, namely Theorem 3.1 and Theorem (32., In the proofs, we will use a convenient shorthand for the training loss defined in 6 when specialized to the Gaussian mixture model denoisers of interest. Namely, since in this case the loss is effectively a function of the learnable mean parameters and the shared learnable variance parameter in the model, we will write
B[CN(H',
u K o.)]
to denote the loss of the generalizing denoiser L (āx),
E Nx' ...., x .O)
to denote the loss of the memorizing denoiser (# and
E (x ..... x .]]
to denote the loss of the partially memorizing denoiser LN e pmmem,
33

---

Theorem F.1: Given vectors (pt) satisfying
min jut u 2イ>0, 0,
N( consider the distribution π = (1/K) E "[p,,,70 I. For E [N], let x² ~J, fix and let o a,x' + ogg, where 9 (0,, 1) is independent from x². Consider the denoiser H. (t, xx) associated to the true distribution n.. Then for any 0 satisfying the coupling conditions
047
83
2 daaoo? +
and
2²
I
1-8 20
do,07 0/ 02
( 1. .o.3 R [ (m)
x ,0]]
$ (E,,,,,)
where the residual =\t is explicit in the proof.
We denote 1/v = =o//o? and & its inverse function.
Proof. The proof will be an application of Lemma E.4 Lemma E.5
Nominal value decomposition. First, we define for any E [N]
?o50 a²o? + of a.o? + o, k.
of
nom (t, x2)
x +
This nominal value noom (t,, .z) will be erucial for the rest of our analysis. Using 13). we have that
do
LE (N .... K [o.]] - [x ....,,NNN .o]
af +
do 2
L. (t, x1)) Emem (t, x [|]
N
a/o² + o
|| (t. xz)) nt, xi) + zuom (t, 2t,
2; + x; 2mem (t, xx) 2
do of
a o + o
In what follows, for simplicity, we denote A
A (t, x;)) ntm
(t, (t, + Zmem (t, x;).
34

---

We have that
d
| ([CN(uu..... u,, .o)] EE[N((' ,N.,,,,,,.. xN,,,, _0]
(28)
año? + o
N
do
[|A + + x;)) - x211?]
E
09 + oo
1
E
|| nom (t, xi) -
-x:11" 2=.1²]
do
N
a² + Fo?
1
N
E
|
N Σ E
2 N
1/2 [|A+||]]
B
2 (t. - -z,0 1/2
N
N
do
nom (t, x/) - x [1
E
afo? + of
N
2
E || x (t,, xx)) (t x;) ||@] + E ar; memm (t,, x;;)) [1]
N
4
N
a.. (1, x xm (t, mz)3 ||]]
1R. [|:2 a mem (1, 21:)1 1/2
N
(1, x1)) - mm =1]]]1/2
x
E
In the rest of the proof, we control each term in (8
Control of generalizing denoiser. First, we are going to control E [| a (t, x/) - wuo 2z)) First, we recall that by (11
ago? af 2 + oi
of
M
a. (t, x
z +
H softmax(w):
a
o
o
In what follows, we denote In addition, we have that
r max | ||. =E[K]
क a h: + a,o. + o.g a + +afo? + oz?
where k; is the index of the mean corresponding to x'. Finally, we have that for any p N
B||||||| < B |0 1/2 < 2n/
< /n/ 2 (d/2 p/
Combining those results, we get that for any p E N
E |==:||||
< 2n
+ ""+
Similarly, we have that for any p EN
E
a nom (1, 110] < 220 (Tr + (af0? + (2))
For any event A such that || @t. (t,x)) -
- noom nt, x))l s C, we have that
S CTP(() + 2P/ E [B||at|| || 1/2
+ E 1l (L, (111)) 1/2)
E ||| | (1) - x (1,m(1,2:))1
SC 16 (T2 + (afo ( (d/22 1/2
35

---

Now, combining this result and Lemma E.4 we get that
E aT. (1, a:))
x nmo (1, mz)
2Kož max |
2
a²
afo? + oo3
exp
3 + of))
+ 64
max kE[K]
|| + (af of) + 4) expl-de2/100 /16]
Therefore, there exists a numerical constant Co 2 0 such that
E
B. (1, (1, wz) x noo =1,
a²a
I Co
|1 + + o?)ld K² exp 2(a k
max
of)
+ expl_dee (odee///6))))
Control of memorizing denoiser. Second, we going control IE | t mem (t, )1 The
proof is similar to the control of E i.. (t, æt) - -oom (t, <.,))
We recall that we have
C Zrem (t, x1) = Σ x softmax(w);
where
wi(xx) = 2 $layoo - =0/12/0²
Similarly before, we have for any N
E
max
< 2T + (d/2 + p)2/2).
[E[N]
Therefore, we get that
E|| N
E[ (t, (t, S 2
Note that this upper-bound is rather loose but given the of growth of N with respect that will consider we won't need tighter bound. For any event A such that we have that
|| @mem (1, xz)) - x: < C.
1] 1/2
+ E || (t. x;) x;)1| 21)11") (1=2))))))1²
E @mem (t, xx)) ='1]] S CP[A] + 1//2
x
S 16 (T 2 4))
P[A911//
Therefore, combining this result with Lemma
E.5 there exists numerical constant C₁ such that
E
2 mem (t, x x x0(1²]
S Ci
|1 + (1 0.)d) N2 KE[K]
max
exp
a, s)d
+ exp[_de? de? /16)
20
Control of the nominal expectation. nominal denoiser's definition in Lemmal
For the nominal error ||a (1, xx)
x²1,, we have by the
E.4 and the distributional assumption x LLidd.
of + o k
moom (t, x:))
x
x
ofo? + of
a70?
0,??
F:g' of
-_30
0f of
af
36

---

where w' ~ N, I) is independent of g¹ ~ (O I). In particular, this is equal in distribution to the squared E norm Gaussian random variable, which has zero mean and isotropic diagonal covariance with diagonal elements
0f 0f
0. ai oi ai + of
0g00
of
+
An application of Theorem C.I then gives that with probability at least 1 - 2 expl dd
do
do do
moom x;) x01
S E 0.00 + o,
a7 of
For any event A such that on that event, we have that || nomm (t, x/)
do:o3
am (1,22;) =0'l2
SC,
0f + o
?o
do
xx1
nom (t, xxt) 2z11
0i 07 +
oz
0 07 + of
1/2
nom (t, 2 1/2)
<C E
+ E
Therefore, there exists C2 2 0, numerical constant, such that do aF + o
E nom x;) xx1 111
do 2
E 0, oi
+ C2
max kE[K]
|| + (1 + o.)d expl- exp(-=2d/16).
Therefore, there exists C3 2 0, a numerical constant, such that
do
E [[ (M), ..... 1 [. E E[CN(at (
(
..... N No,O) [0]] afo? o7
do
E af +
+ +C3
max kE[K]
| !1
(1+ 02) (K² + +22
oi
?²?
afo7 s)d
exp
2(aço3 + o
+ +exp
20
+ exxp(=2d//16)
Let us further simplify the bound. We have that
dofo?
([CN(H..........
.K.0 E[[N(x',
N [NO]
aio? +o²
do?o?
dojo3
Se
af 2 + of
+33
1+
max kE[K]
02 Poly(d)
af + oo
(1/0² + +1)
7²
aFo2 - 8)d
exp
2(a + +o
+ exp
207
+ exp - 2/166
where we have used that
do
(1/0 + +1)) 21,
of + of
with 2i a./
Proposition F.2: Assume that N = poly(d), 72 = (d)) max ((1). Let k(d) = ((((log(d)//))) We have that uniformly on E [0,x(d)
|1 = (dd) and 0² =
do 2
E C[CN 1.t o0)] - B[CN
.)] = afo of
The claim will then follow by an appropriate choice of constant in the definition of k(d), which will make the residuals the previous expression of the correct order of magnitude to be nontrivial, and guarantee the approximation for I within the claimed interval.
37

---

Proof. First note that (llog(d)/d' satisfies the coupling condition. Next, if we assume that is large enough so that E _ 1/2. We have
do:0i
B(CN(M...... ................................
Mx k
o.
[[[x(x' x. .0]]
ai + +
d
| 1 / o3 Poly(d) do,00
+C1
+
max kE(K]
(1/0² + v)) af + of
0 + of
1
3/4/ 1 + 0f )
exp
min
log(d)) 24
16
Theorem E.3: Given vectors (u+)k=, satisfying
min kEk
p, u. 1 0,
consider the distribution (1/K) / N For E [N] let x' N. nxx 1, and let xi axæ² + o.g, where 9 ((,, is independent from x'. Consider the partial memorizing denoiser Fmmem (t, x.). Then for any $ & S 1 satisfying the coupling conditions
a+
E E
2
+ o d(a
and
8²
aj
1 8
20
we have that
(1- L
do - t(),t,d)
E LN ,..., x..0) - E EEILNv (2w(w x.....,,000
2 1 N do + E(E, d),
where the residual explicit in the proof.
Proof. The proof will be an application of Lemma E.6 and Lemma E.5
Softmax value decomposition. Using 13 we have that
E
(N ,...,,"' _0] - (C x 0)]
N
T ppmem M (t, xt) - &mem (t, (t, x;)1
N
N
E
pmmmm M (t,, 2x) x, + x. Tmem (t, xz)) 113
N
We have that
E [ (x) ...., 2..00000 [)] E[CN(x' ,..., .o0)
N
1
EE
(ppmem, dsl x/) xz + x, Zmem (t, =3
N
N
1
E
@pmem, M (t, (t,zt)
x [|]] + E @mem (t, x;) - -2(11²] 21
N
38

---

where
| < R Bmem (1, m²) xr'] "=]""] 1/2
pmmmm, (1, (1, x0 2011]]12 1/2
18
In the rest of the proof, control each term in (28
Control of memorizing denoiser. Second, we going control E [z) &mem (t, x The
proof is similar to the one of Theorem E.1 but we reproduce it for completeness. We recall that we have
N x softmax(w)i;
mem (t, xz) =
where
-
wa( =
2 EIlooaat =0/
Similarly before, we have for any p
max [E[N]
S 2 N (d/2+ p)
Therefore, we get that
B|[| [P] S N
[[ (t)))))))) (t,
S +
Note that this upper-bound is rather loose but given the rate of growth of N with respect that
will consider we won't need tighter bound. For any event A such that Zmem (t, xi) - x $ C, have that
E o memm (t, (t,x)) - x011"] - CP[AI + /2
"1 1/2
+ E [| xx x1)) (122)))))))
C6+16 16 oo 4)) P
Therefore, combining this result with Lemma E.5
there exists numerical constant C₁ such that
@mem (l, x - x=1
a af ()- + expl expp-de²// de2/16])
Ci max | + +11 +1++ o2)d
N2
cxp
KE]K]
20
Control of the partial memorizing denoiser.
Third, we are going to control [||00 1 Tpmem, (t, First, similarly before, have that
E B|[a [P] 2 oo +d.(d/2+p))/
E (1,
For any event A, have that
2 pmmmm M (t, xx) x=-1|3] -E
x -2011211 pmem, M (t, (1)) ppmmemmmemmae
< 2IP 1// [=[|a=[]]
1/2
+ +E +B[1@pmmem Moaeemmmm/i 2t) [1]] 11]1/2)
< 16 (T2 (d/2 + 4)) 2/2
(29)
Hence for E [0. using Lemma E.6 we have
E || M(t, (t, xz)
x -x11
2 21 e)d
< 2NT, exp
16 (T2 + o3(d/2 + 4)) (8N) exp[-de2 -
o7
39

---

Now, for i & [2], we have that
T Mmm (1, x²6 x2
I pmem, M (1, 2;)
Σ
softmax s
soflmax(wwls.)) x.
jes.
JES.
ppmem, (t, (t,
Σ
softmax w )
JES;
+
Σ
softmax ( (wss)ss )
jES;
+ T pmem, M (1,
Σ
(s Σ
(oftmax((rls)))) xx
JES.
jES,
Therefore, we have that
@pmem. (1,
(1) - -0011=
Σ
softmax(w/ jo x²
JES,
<
8 @ (t,
Σ
softmaxx w )s)
JES.
+
E Dpmem. Mt, xt)
Σ
softmax(w|s,)..
Σ
softmax(wx) (w),)) x
JES:
JES.
Therefore, we get that
pmmmm Ns (1, (1, - 22l/²
Σ
softmax s x²"
JES.
softmax(u0) (wl )ya
jES.
+ Spmem.Nr (t))
Σ
softmax(wls.),rr
Σ
(eltmax(wll.)). ) x²
jES,
JES.
Hence, using this result and Lemma E.6 we get that with probability at least 1 - 8.N expplds=211 K- log(d) (1 log( K) log(d))
E
i P M (1, 2:1
Σ
softmax(1ols.))
r 1
JES.
0,721
< 81 + 2 exp
N(T + (d/2 (d/2+ 11)
o
40

---

where A is the event of Lemma E.6
Hence, combining this result and (2) we have
27
| ppmem, (1, xx) -xal)
E
Σ
softmax(ur|s)) xx
JES.
a
$ (1,(++N) exp
07
+ 16 (I2 + (d/2 4))
(8N) 1/ expl expl-de +K (0-og(d)/2(1 (1+ + log(K) log(d))
of72
+ 88. + N) exp
N(T + 0x 1)
of
Finally, in order to control TE Ejes, softmax(uols,)jat x² [2 concludes the proof.
we use Proposition C.3 which
Proposition E.4: Assume that N poly(d), 72 (d)), max
|| ((d) and of
((1). Let sdd) = (log(d)//d)) We have that uniformly E [())
B[CN ,...,"......."....... ;..., x 0)] -E E[[N((a' s....
x0 0)] θ (11- 8
do?)
Proof.
The proof of this result similar to Proposition F.2 by letting & = (log(d) //22)
G Additional Related Work
Memorization in diffusion models. Understanding memorization and generalization properties
of diffusion models is crucial for practitioners [Somepalli et al. 2023 Ren et al., 2024 Rahman
ett al. 2024 Wang et al. 22224a] Chen al., 2024bb Stein et al. [2024] Indeed, concerns about
privacy Ghalebikesabi et al. 2023 Carlini et al.
2023 Nasr et al.] 2023] and copyright infringement
[Cui et 2023 Wang et al. 2024a] Vyas et 2023 Franceschelli and Musolesi 2022] are key
issues as these models are deployed. Several factors can influence the memorization capabilities of
diffusion models. Duplication and oui-i--distributiin samples have been shown lead to replication
in diffusion models [Carlini et al. 2023, Ross et al. 2024| Webster 2023) and solutions have been proposed by curating the dataset [Chen
2024a] introducing dummy data and adapting diffusion models to unseen data [Daras et al. 2024 Yoon et al., 2023). Neural network architectures
have also been shown to play role in memorization with [Chavhan et 2024) identifying specific neurons causing memorization and [Wen et al.) 2024 Wang et al. 2024bb analyzing variability in the prediction function identify problematic prompts in image models. We refer [Gu et al.) (2023) for an in-depth experimental investigation of those issues. Finally, we highlight the work of [Kadkhodaic] et al., 2023). in which the authors investigate the inductive bias image denoisers to investigate the generalization properties of state-of-the-ar diffusion models.
Definitions of memorization. Note that our definition of memorization (Definition 2.2 is connected to the one of Eidetic Memorization as introduced by Carlini et al.) (2023). Similarly, to [Carlini et al.) 2023], we acknowledge that this notion of memorization strong as implies that an image is memorized if there exists near perfect copy in the training dataset which does not fully capture
copyright infringement or data privacy issues. We refer FElkin-Koree al. 2023 for an in-depth discussion of those issues.
Memorization and overfitting. A priori, may not clear why memorization in diffusion models is substantially different from overfitting which has been a long-standing and well-studies property of machine learning systems Hastie et al. [2009) Overfitting characterized by a large gap between training and validation losses. However, overfitting is not directly connected to memorization. For a small number of parameters, the partially memorizing denoisers defined Section 3 have large training and validation losses which ultimately have small gap, meaning that such models not
41

---

significantly overfit; however, every single sample they generate is memorized. Meanwhile, the model corresponding to the fully memorizing denoiser defined in Section Diatiionnssss is both very overfit and does memorize. It is also worthwhile to discuss the connection of diffusion model memorization with benign overfitting (Bartlett et al. 2020) and double descent [Belkin et al., 2019). In particular, we wish to clarify why the setting of diffusion models may be separated from benign overfitting. The core difference is as follows: in the usual double descent setting, one studies an overparameterized learning problem, such regression or classification, and there are many models at fixed parameter count which obtain the minimal training loss; then is up to (implicit) regularization to choose the best solution, which benignly interpolates the training data. As the number of parameters increase, the training loss of the trained model never increases, and the validation loss eventually also decreases (after an initial increase for the purpose of overfitting). Mcanwhile, in the case of diffusion, there is exactly one model which obtains the minimal training loss, which is the memorizing denoiser, allocated. Certainly the memorizing
and this does not change no matter how many parameters denoiser does not benignly interpolate the training data. As the number of parameters increase past a certain point, the training loss never decreases, and in fact may increase, showing that double descent or benign overfitting indeed do not apply straighfforwardly to the case of diffusion.
H Experimental Details
We run all experiments on several Nvidia A 100 80GB GPUs using Jax 0.6.0 and Equinox 0.12 (Kidger and Garcia 2021]. Each training/evaluation job occurs on single A100 and the results saved to file to be aggregated later. Aggregation, analysis, and visualization occur on a single A100.
H.1 Datasets, Optimization, and Initialization Details
In our experiments in Section 4.1 we generate synthetic Gaussian mixture models, by generating the K means yi, uniformly on the sphere of radius vd and setting the ground truth variance of 1, as prescribed by the results in 3 The experiments conducted in Figures I and Zand Appendix H.. are all under the setting N- = 200, d 50, and K 12. For the sweep in Figure 3 we take (N, d, K) tuples from [50, 100, 150, 200] x [30, 40, 50, 60) x [3, 6, 9, 12), obtaining a total of 64 (N, d, K) tuples. For each setting of (N, d, K) (ineluding the sweep Figure 3 we train 20 models at different model sizes M: starting from M = |N/10] and moving in increments of [N/10] to M = 10 [//N/ then, training 10 more models where M is equally spaced between the M where the phase transition starts and the M where it ends (using the empirical criterion in Section
In our experiments in Section 4.2 we generate synthetic colored FashionMNIST data by sampling K FashionMNIST (Xiao et al. 2017) images uniformly at random to use as "templates", then using the PIL (Pillow) utility to reshape them 15 x 15 resolution. For each of the K templates (components) we generate a color vector using Gaussian with ground truth mean and variance (u(,o3) of) = (0, 1) E 3 x IR. We take the Kronecker product of the color vector and the template as described in Section 4.2 in order to form the sample. Figure f juuess the setting N = K = 8 while the experiment in Figure 5 uses the setting N 100, K 4, and color dimension d = 3. Here we train 10 models at different model sizes M, starting from M = [N/10] and moving in increments of [N/10] to M = 1/O0/
For all experiments, we use the "variance preserving" process which yields at = -I- and or = for & E [0, 1]. We use the objective (5) to train our model denoisers. For training, we always train with the loss weighting A(t) := of/oz. which is equivalent to using noise prediction where we use L = 25 decreasing timesteps tz 0.01 - 0.999 0.9988 L 10. In lieu of computing the (obviously intractable) inner expectation in CN.e. we use Ndup := 100 Gaussian noise draws for each of the N samples to estimate the expectation. We use full-batch Adam for Nepochs epochs (also, iterations) to optimize the objective; for experiments in Section 4. we have Napochs = 50, 000 and for experiments in Section 4.2 we have Nepochs = 100, 000. We use a "warmup-decay" learning rate schedule: for
( (Karras et al. 2022 D. and t ~ (tz)
N Nwarmup Nepochs/ 10 epochs the learning rate linearly increases from 0 10-3, for the remaining Nacray- Nepochs Nwarmop epochs the learning rate linearly decreases from 10-3 to 10-6
For all models we train, we use a "partial memorization initialization" along with the Adam optimizer We use this initialization because the loss truly random initialization is extremely high, in many
42

---

cases often at least eight orders of magnitude larger than the loss at optimum, and Adam is often unable to learn effectively given this massive conditioning. The partial memorizing initialization, in the context of the isotropic Gaussian mixture model, sets the initial o² to 10- and each mean ja' to a random sample. In the context of the simple image model, sets each initial template parameter Agt to the template which generates sample, the corresponding initial color vector to the unique vector which generates the (not identically zero) sample given that template, and the initial color variance o² to 10- Notice that in the experiments such Figure 2 even models initialized with this partial memorizing initialization end up learning (nearly) generalizing solutions - unless of course it is more favorable to memorize, which occurs with very large M.
H.2 Formal Description of Sampling Scheme
We use the implementation of the DDIM sample prescribed in De Bortoli al. [025 i..., using the above notation and given a denoiser æb
Oeerl De, + ate.
Otes ats
D1s+1
x(te; at:);
T ~ N),,
Ote
As previously stated, we use increasing timesteps t, 0.999 for L 25 and E 10. L). Notice that we use the same timesteps for sampling for training. We implement this iteration using the Diffusionl.at PyPI package Pail 2025
H.3 Formal Description of Loss Weighting Regression
Recall that in Section 4.1 we predicted the phase transition using the loss approximations derived in Section 3 To do this, solved a regression problem (15). Here, will discuss how solve this problem efficiently by smoothing and regularization.
Namely, bilevel semi-discrete quadratic program, the problem (15) is hard to optimize outright via gradient methods. As result we parameterize the distribution over via a temperature-weighted softmax with high temperature softmax output to ensure that the learned distribution over M is sparse. The overall problem (using the notation from (15)
1/20, placing an entropy penalty (Separeity 10 10-3) the
M (N, Mpt 2
min (N.
(30)
N
Peporsity
E Σ
p(N, d, K, M. d, K.M
(N.dKK M
where
Mpa(N, (,,,K, E M
M KN,,
and 5(N, d, K ;,) softmax (Eto (ttt K)) Lx..0.((N.d,
In all cases, when we report the loss refer to just the MSE component described in 30
H.4 Simple Image Model Calculations
In this section we simplify the computation of the colored image denoiser in Section 4.2 Recall our setup in Section 4.2 along with the notation in Lemma 2.1 Under the settings μ! = A u² and Σ O? (A it holds that
/ = A.u. = ( = u x
Σ = oi (TTzz)) oz) o (T) (z)
43

---

Also, simplifying the inverse (a²Σ; + 0² obtains
using the Sherman- Woodburry identity
a²
-
02
+
Σ
σ²
0²
o²o²
AA(A)T
+
Ø²
g²
0²
I + (A.) 4: T [d:)7]
A
(A:)TA:
σ²
a²o?
Calculating the interior term first, we obtain
(A:) T A
(I @ x
[I @ xx) =
- ( (1) = ((x)) (x!)]
@ |||= |2] | 12
which yields
(a + 022 1)=
1
1
AA(A
g²
o²
+ |
a²
1
I
a²0² 1+ AA(A
a²o²
+
Another part that requires elaboration is the action of (A))T on a (block) vector, say 0 which has
(at)
( = (I (I (xt)
(art) 0.
Finally, the last part that can do with simplification is the log-determinant, which obtains
log E + I² log det (22 I
a²
Σ
log do II) log det
I 0² Σ₁
a²o²
2cd log(o) + leg
I
A (A))
a'o²
2ed log(o) + log det
I
(A)) A:
o²
2nd log(o) + log det
I
|| |a=
{
+1+
a²07 0?
| {p]1)) 1
2ed log(a) + log det
0²
a²0?
2nd log(o) + clog
+
|
o2
=c 2d log(o) + log
a²o?
})
1
|
1|
02
These terms all simple to compute, and therefore so is the true denoiser (as well any parameterize).
H.S More Experimental Results
Interpreting the learned models. In this section we take advantage of the white-box nature of our denoiser and specifically how relates to the data generating process, to try to uncover some mechanistic aspects of memorization in this setup. There two questions wish to answer:
44

---

Learned Variance
Dist. Nearest Sample
Dist. to Nearest True Mcan
7.0
6.5
Distanes
9
Distance
6.0
5.0
4.5
3.5
0.2
0.4
0.6
0.8
1.0
0.2
0.44
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
M/N
M/N
M/N
Figure Left: A plot of the learned variance as function of the model size M. Middle: The average distance of a learned mean to its nearest sample in the training datá, as a function of M. Right: The average distance of learned mean to the nearest ground truth mean, as function of M. All plots include the start and end of the phase transition. While the variance eventually decays 0, it surprisingly only does linearly, and for every M before the memorization ratio increases and the phase transition occurs, the average distance from a learned mean to the nearest sample decreases linearly as function of M. Mcanwhile, the average distance from learned mean to the nearest true mean increases. Surprisingly, this behavior happens during the generalization phase as well. Note that the provided example representative among our trained models.
the phase transition the ground truth variance does collapse to O. Similarly, as the
Memorization Ratio
Training Loss, & = 6.704
Test Loss, 1 = 6.704
1.0
10
100
80
60
Le (.FFmmmm,.. M)
Exallfmam)
Le (Tmmm)
0.2
(x.l
Cx.((s))
20
(..)
C. (((ās)
0.0
0.2 0.4 0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
M/N
M/N
M/N
Figure 7: Our loss calculations and memorization trends are stable under different random seeds. observe the same behaviors as Figure 2 when re-attempting the same experiment with three separate random seeds; we provide error bars but note that they are extremely small, indicating tiny variance.
Is the learned variance Prain an effective proxy for memorization or generalization?
How do the learned means Pirain behave in memorized and generalized models?
Recall that the ground truth (generalizing) denoiser has variance of and means μú, and the memo- rizing denoiser has variance 0 and means æ'. Thus, one intuition would that smaller variance implies propensity of the trained model to memorize, and learned means closer to samples would imply the same. To verify this behavior we plot the learned variance and the average distance of the learned means to training samples and ground truth means, respectively, in Appendix H.5 We can confirm that our basic intuition is true, with a twist: while the learned variance o decreases as the model size M increases, does linearly, and even in the generalization regime. Similarly the average distance each learned mean to the closest point in the training dataset decreases linearly to 0. In this sense, these and other mechanistically derived quantities may serve continuous proxy for the rapid phase transition, behavior also observed in large models with far more complicated "circuits" [Nanda al. 2023 Schaeffer al. 2023
Assessing the variance of different seeds. In Figure 7 we examine multiple runs using different random seeds to their effect the loss and memorization plots (akin to Figure TT. The shading on the memorization plot, which showcases the minimum and maximum value of the quantity across three seeds, amounts error bars on the regression experiment in Figure 3 since the approximated losses will be the same (as they are computed deterministically). and the only remaining variation is the regression target, i.e. the location of the phase transition.
45


Reviews:
[{"review_id": "qaXDRcJbu8", "reviewer": "Reviewer_vocx", "summary": "This paper studies when diffusion models memorize their training data versus when they properly generalize. The authors study a setting in which the data distribution is a $K$-component Gaussian mixture in $\\mathbb{R}^d$. Diffusion models are trained to reverse a standard forward noising process, and the learned denoiser $\\tilde{x}_\\theta(t, x_t)$ is constrained to be the minimum mean squared error (MMSE) estimator implied by a Gaussian mixture model with $M$ components.\n\nThe parameter $M$ controls model capacity. For $M = K$, the model can represent the true data distribution and generalizes. For $M = N$ with zero-variance components at training points, the model memorizes — its denoiser effectively returns a nearest neighbor from the training set.\n\nTwo types of training losses are defined and analyzed:\n\n1. $L_N(\\mu^1, \\dots, \\mu^K, \\sigma)$: the expected training loss when the denoiser is derived from an assumed $K$-component Gaussian mixture model. The authors derive a closed-form expression for this loss using Tweedie’s identity.\n\n2. $L_N(x^1, \\dots, x^N, 0)$: the training loss of a fully memorizing denoiser that places a zero-variance Gaussian at each training sample.\n\nThey show that these two losses intersect at a critical model size $M^\\star$. This crossover point predicts a transition between generalization and memorization: for $M < M^\\star$, the generalizing model has lower loss; for $M > M^\\star$, the memorizing model does.\n\nTo relate this theoretical crossover to practical training outcomes, they introduce a time-dependent weighting function $\\lambda(t)$ over diffusion times. They fit a single weighting function $\\tilde{\\lambda}(t)$ across multiple training setups to align the theoretical crossover point with the empirically observed memorization threshold. This calibration is used once and held fixed in subsequent evaluations.\n\nThe experiments are conducted on two synthetic settings:\n\n1. Isotropic Gaussian Mixture: A dataset drawn from an isotropic $K$-component Gaussian mixture. The authors train denoisers of increasing size $M$ and observe a sharp phase transition in both test loss and a \"memorization ratio\" (the fraction of generated samples classified as memorized based on proximity to training data). They show that the predicted crossover point matches the empirical onset of memorization.\n\n2. Low-Rank Image-Like Data:  Each image is generated by applying a low-rank color embedding resulting in a mixture of low-rank Gaussians. They repeat the same experimental setup and observe a similar memorization transition.\n\nTogether, these experiments demonstrate that the theoretical loss crossover — once calibrated using $\\tilde{\\lambda}(t)$ — predicts the generalization–memorization phase transition observed in trained diffusion models.", "questions": "Can you explain why optimizing the weights $\\lambda_t$ actually isn't \"cheating\"? I'm not claiming that it is, but it does feel fishy when one of the main claims of this paper is that the theory \"predicts\" a transition point and yet this prediction is made with some sort of explicit training.\n\nEdit: re-evaluated post rebuttal.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: this paper provides a simple model where memorization can be easily characterized. It also gives explicit derivations of the expected losses (rather than bounds) which allows the loss difference comparison (i.e. memorization loss vs loss from using the true data distribution) to be accurate computed. \n\nWeaknesses: this paper claims a \"phase transition\" without really giving a good definition of what that is. It is sort of obvious that if you increase the memorization capacity of your model that your model will memorize more and generalize less (this is observed ubiquitously in machine learning) so merely providing plots and claiming that it is happening doesn't seem very compelling. Furthermore, claiming that the theory above can \"predict\" this transition point is a bit less impressive than it sounds because 1. the predictions are in a very restricted theoretical setting and 2. the weights $\\tilde{\\lambda}_t$ appears to be optimized to make these predictions as accurate as possible. \n\nI also found this paper a bit difficult to read. The notation used throughout the first few sections feels rather messy (there is an excessive use of bolding, subscripts, \\tilde, \\overline, \\hat, etc.) and it took me a while to understand what the Loss $L_n(x^{(1)},\\dots x^{(n)}, 0)$ meant. I strongly recommend thoughtfully cleaning up this section and adding some definitions in (like a definition environment) that might help more cleanly explain what is going on here.\n\n\nEdit: see comment for update on review.", "was_revised": true, "final_justification": "This is all clearly laid out in my comment to the authors.", "rebuttal": "We thank the reviewer for their thorough review of our submission. We respond point-by-point below. \n\nGiven the clarifications below around the soundness of our methodology, we hope the reviewer will consider raising their score. We look forward to discussing any remaining issues further. \n\n> this paper claims a \"phase transition\" without really giving a good definition\n> of what that is. It is sort of obvious that if you increase the memorization\n> capacity of your model that your model will memorize more and generalize less\n> (this is observed ubiquitously in machine learning) so merely providing plots\n> and claiming that it is happening doesn't seem very compelling.\n\nWe respectfully yet strongly push back on the notion that we are “merely providing plots and claiming that [the phase transition] is happening”. The existence of this phase transition and even its universality across different models and hyperparameters has been very well established by prior empirical work [1–2]. A main contribution in our work is to isolate the factors underlying this phase transition and reproduce it in a synthetic setting amenable to theoretical analysis, and to empirically link the precise point at which it occurs *in trained models* to easily-computable properties of the loss landscape. This sheds light into the mechanisms underlying the phase transition in a novel way with regards to previous empirical studies, and can act as a foundation for further finer-grained studies in the future.\n\nIn general, we can clarify the writing to more clearly introduce the concept of a phase transition. This term is prevalent in physics/statistical\nmechanics and their connection to machine learning [5–6], and phase transitions between\nmemorization and generalization have been empirically observed in prior\nworks [1--2], which we have discussed in the introduction and related work sections of\nthe submission. Quoting from [1], a foundational work in this area:\n> We show empirically that diffusion models memorize samples when trained on small sets, but transition to a strong form of generalization as the training set size increases… [t]he amount of data needed to reach this phase transition is very small… and depends on the image size and complexity relative to the neural network capacity.\n>\n> --- [1], p. 9\n\nWe respectfully note that the reviewer's objection (“it is sort of obvious…”) could be identically levied\nat any of these foundational prior works.\n\nThe fact that this phase transition has caused\nsurprise and research interest in the context of diffusion models (in contrast\nto its \"[ubiquity] in machine learning\") may be attributable to the fact that\ncertain diffusion models seem to memorize more than equivalent generative models [3],\nand that the relationship between the trained model (a denoiser for\ndifferent times $t$) and the algorithm for generating samples (a multi-step\niterative refinement of a noisy image with the trained model) obfuscates the\nlink between \"overfitting\" behavior of the denoisers on a given timestep's loss\nand the ultimate memorization behavior of generated samples.\nOn a more abstract level, note that in the context of diffusion\nmodel training (i.e., Gaussian denoising), *the unique optimizer of the training loss\nis the model that memorizes the training data*, as we describe in\nSection 2 of the submission. This should be\ncontrasted with the situation in other machine learning settings, e.g.\nclassification, where there are typically an infinite number of models that\nachieve minimum training loss, and the key question is to understand which model\nis learned, given one's specific training algorithm [4].\n\n> Can you explain why optimizing the weights \\lambda_t actually isn't\n> \"cheating\"? I'm not claiming that it is, but it does feel fishy when one of\n> the main claims of this paper is that the theory \"predicts\" a transition point\n> and yet this prediction is made with some sort of explicit training.\n\nIn short: Optimizing the weights $\\tilde{\\lambda}$ is not “cheating” because it is a standard and accepted scientific methodology. It corresponds to hypothesizing that an empirical phenomenon follows a certain mathematical law, then fitting the free parameters of the law based on empirical data, and verifying whether the resulting law correctly predicts the phenomenon on held-out data. This is the same methodology that underlies the modern practice of scaling laws for training large models [7].\n\nBelow, we provide a discussion with full details in order to completely clarify the role of $\\tilde{\\lambda}$ in our methodology.\n\nIn our submission, we hypothesize that the\nmemorization-generalization transition occurs when the weighted relative loss\nof two surrogate models (a generalizing model and a memorizing model) switches\nfrom positive to negative (the \"crossover\"), for *some* weighting. In this way, our experiments effectively fit a predictive model $f(N, d, K;\n\\tilde{\\lambda})$ (defined in eqn. 19) with parameters $\\tilde{\\lambda}$ for the\nnumber of model parameters $M_{\\star}(N, d, K)$ at which memorization becomes prevalent\n(lines 242-251), **and validates the accuracy of this predictive model on a set of\nheld-out simulation data** (lines 284–288). \n\nWe can also demonstrate that the effective predictive model implied by this parameterization is a simple linear function of the number of samples $N$, so the reviewer should not be worried about any hidden capacity to overfit.\nThe two surrogate models’ loss approximations that we use in our predictive model are justified by Theorem 3.1, namely\n$$ \\mathcal{L}^{\\mathrm{gen}}(t, M) = \\mathcal{L}^{\\mathrm{mem}}(t) +\n  \\frac{d\\sigma_\\star^2}{\\psi_t \\sigma_\\star^2 + 1},$$\nwhere $\\psi_t = \\alpha_t / \\sigma_t$ and $\\sigma_\\star^2$ is the variance of the\nground-truth model (we use $\\sigma_\\star^2 = 1$ in our experiments); and Theorem\n3.2 (and Figure 1), namely\n$$ \\mathcal{L}^{\\mathrm{pmem}}(t, M) = \\mathcal{L}^{\\mathrm{mem}}(t) + 2 d\\sigma_\\star^2 \\left( 1 - \\frac{M}{N}\\right). $$\nFor integrable weights $\\tilde{\\lambda} : [0, 1] \\to \\mathbb{R}$, the weighted\ncrossover point formally\nentails the study of the value of $M$ at which\n$$ \\int_{[0, 1]} \\tilde{\\lambda}(t) \\mathcal{L}^{\\mathrm{gen}}(t, M) \\mathrm{d} t \n-\\int_{[0, 1]} \\tilde{\\lambda}(t) \\mathcal{L}^{\\mathrm{pmem}}(t, M) \\mathrm{d} t = 0. $$\nBecause $\\mathcal{L}^{\\mathrm{pmem}}(t, M)$ is a constant function of $t$,\nwhen we plug in the concrete expressions for $\\mathcal{L}^{\\mathrm{gen}}$ and \n$\\mathcal{L}^{\\mathrm{pmem}}$ above to this expression, we end up with the\nequivalent expression\n$$ \\int_{[0, 1]} \\frac{\\tilde{\\lambda}(t)}{\\int_{[0, 1]}\\tilde{\\lambda}(t)\\mathrm{d} t } \\left( \\frac{\\tfrac{1}{2} + \\psi_t \\sigma_\\star^2}{1 + \\psi_t \\sigma_\\star^2 } \\right) \\mathrm{d} t = \\frac{M}{N}. $$\nThis shows that *fitting weights $\\tilde{\\lambda}$ to\nminimize the predictive accuracy on a dataset is equivalent to fitting a linear\nmodel $M_{\\star} = w N$*, in the language of the discussion above.\n\nIn spite of the simplicity of the model for the location of the phase\ntransition, note that our experiments fail to reject this hypothesis.\n\n> Furthermore, claiming that the theory above can \"predict\" this transition\n> point is a bit less impressive than it sounds because 1. the predictions are\n> in a very restricted theoretical setting…\n\nWe emphasize that our setting is no more restrictive than what every theoretical work on diffusion models considers. In the theoretical literature on diffusion models, the Gaussian mixture model (GMM) we study actually sits at the ‘frontier’: almost any algorithmic research question one can formulate around diffusion models remains open for GMMs. More precisely, [8] proves that the minimizers of the training loss over a properly-parameterized class of GMMs generalize (in our notation, analogous to the number of components $M$ being equal to the number of classes $K$), [11] proves the same for models learned by gradient descent for two-point mixtures of Gaussians, and [9–10] establish guarantees for learning properly-parameterized GMMs with non-gradient-descent algorithms. To the best of our knowledge, no prior work has studied the setting where the number of components $M$ may be significantly larger than the number of classes $K$, which is the essential setting to understand memorization in GMM diffusion models as they require $M \\gg K$ parameters to memorize with low loss. We provide further references in our response to `DzmC`.\n\n> I strongly recommend thoughtfully cleaning up this section and adding some\n> definitions in (like a definition environment) that might help more cleanly\n> explain what is going on here.\n\nWe apologize for the issues of presentation the reviewer mentioned. In the revision, making use of the expanded page limit, we will streamline the presentation in Sections 2–4.\n\nPlease let us know if you have other questions or hang-ups, and we will be happy to resolve them. We look forward to a productive discussion during the rebuttal period.\n\n# References\n[1] Kadkhodaie et al., Generalization in diffusion models arises from geometry-adaptive harmonic representations.\n\n[2] Zhang et al., The Emergence of Reproducibility and Consistency in Diffusion Models.\n\n[3] Carlini et al., Extracting Training Data from Diffusion Models.\n\n[4] Zhang et al., Understanding deep learning requires rethinking generalization.\n\n[5] Amelunxen et al., Living on the edge: phase transitions in convex programs with random data.\n\n[6] Maillard et al., Phase retrieval in high dimensions: Statistical and computational phase transitions.\n\n[7] Kaplan et al., Scaling Laws for Neural Language Models.\n\n[8] Wang et al., Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering.\n\n[9] Gatmiry et al., Learning Mixtures of Gaussians Using Diffusion Models.\n\n[10] Chen et al., Learning general Gaussian mixtures with efficient score matching.\n\n[11] Shah et al., Learning Mixtures of Gaussians Using the DDPM Objective."}, {"review_id": "5WtLEFQBjg", "reviewer": "Reviewer_DzmC", "summary": "The paper introduces a theoretical framework to investigate the conditions under which diffusion models either memorize their training data or generalize. The authors model both the data-generating distribution and the denoiser using Gaussian mixtures, which makes the problem analytically tractable. The primary contribution is the theoretical characterization of a crossover point in model underparameterization at which the loss of a simplified, partially memorizing model becomes lower than that of an ideal, generalizing model. The authors demonstrate empirically that this crossover point accurately predicts the onset of a phase transition from generalization to memorization in trained models. Experiments are performed both with a simple isotropic mixture and a low-rank setting designed to emulate the structure of images.", "questions": "To strengthen the claim that your framework captures a more general principle, have you considered an experiment with a simple neural network (e.g., an MLP) as the denoiser? While your analytical predictions for the crossover point would not hold, demonstrating that a standard network trained with gradient descent exhibits the same qualitative phase transition as a function of the number of neurons would partially reinforce the paper's relevance.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper studies the timely and practically relevant problem of memorization in diffusion models.\n2. It provides a sound and rigorous theoretical analysis that appears correct.\n3. It is well-written, with clear explanations and figures that support the central claims.\n4. Numerical experiments are provided to validate the theoretical predictions.\n\n**Weaknesses** \n\n5. The simplifications required for theoretical tractability (i.e., the GMM assumption) create a substantial gap between the analyzed model and those used in practice.\n6. The novelty of the core insight feels somewhat limited. The finding that increased model capacity leads to memorization is well-established. This work's contribution is to formalize this intuition and provide a predictive formula within a specific, very simplified setting, rather than offering a fundamentally new conceptual understanding of the phenomenon itself.\n\n*Minor*\n\nEquation (8) contains an extra $I$.", "was_revised": true, "final_justification": "I am maintaining my rating of borderline accept. The paper is technically sound. However, my primary concern remains the very synthetic nature of the studied setting.", "rebuttal": "We thank the reviewer for their careful review of our submission, and for recognizing our work’s “sound and rigorous theoretical analysis”. We respond point-by-point below, and look forward to discussing further.\n\n> The simplifications required for theoretical tractability (i.e., the GMM assumption) create a substantial gap between the analyzed model and those used in practice.\n\nThis is an unavoidable artifact of developing rigorous theory for a system as complex as diffusion models. Moreover, in spite of the apparent\nsimplicity of this data model and its associated denoisers, even basic\ntheoretical questions about, for example, the number of parameters at which\nmodels learned from the class of GMM denoisers tend to memorize, and the relationship between these trained models and the sampling-time behavior of the diffusion sampling algorithm, remain completely open. \n\nMore precisely, in the theoretical literature on diffusion models, the Gaussian mixture model (GMM) we study actually sits at the ‘frontier’: almost any algorithmic research question one can formulate around diffusion models remains open for GMMs. Specifically with regards to the behavior of trained models, [1] proves that the minimizers of the training loss over a properly-parameterized class of GMMs generalize (in our notation, analogous to the number of components $M$ being equal to the number of classes $K$), [2] proves the same for models learned by gradient descent for two-point mixtures of Gaussians, and [3–4] establish guarantees for learning properly-parameterized GMMs with non-gradient-descent algorithms. To the best of our knowledge, no prior work has studied the setting where the number of components $M$ may be significantly larger than the number of classes $K$, which is the essential setting to understand memorization in GMM diffusion models as they require $M \\gg K$ parameters to memorize with low loss. In addition, because existing theoretical works apply only to the ‘properly parameterized’ setting where $M = K$, they are able to guarantee that trained models generalize at sampling time by applying existing results on diffusion model sampling under accurate score estimates (e.g., [5–6]). In our setting where $M \\gg K$, **it is generally intractable to exactly characterize learned models**, which means that characterizing the sampling-time behavior of learned models (i.e., memorizing or generalizing) is a completely open problem – even the Gaussian case (i.e., $K=1$) is highly nontrivial [7]. Our work paves the way towards a rigorous theoretical understanding of this non-trivial question via the hybrid theoretical-empirical approach we employ, which correctly predicts the location of the phase transition between memorization and generalization in terms of simple properties of the loss landscape.\n\n> The novelty of the core insight feels somewhat limited. The finding that increased model capacity leads to memorization is well-established. This work's contribution is to formalize this intuition and provide a predictive formula within a specific, very simplified setting, rather than offering a fundamentally new conceptual understanding of the phenomenon itself.\n\nWe respectfully disagree with your characterization of this work’s contributions and novelty, and would phrase it as follows. As a theoretical work, our work’s main contributions are proposing the GMM “laboratory” as an environment for studying emergent phenomena in diffusion models, and demonstrating its utility in rigorously characterizing the phase transition between generalization and memorization. Although this is a “simplified setting”, we note above that the theoretical study of diffusion models in the GMM setting is completely open as far as memorization behavior at intermediate model sizes is concerned. In this sense, our work aims to build a *theoretical foundation* for studying complex phenomena in diffusion models, rather than simply adding another explanation in a general setting, and we demonstrate the viability of this in characterizing the phase transition.\n\n> To strengthen the claim that your framework captures a more general principle, have you considered an experiment with a simple neural network (e.g., an MLP) as the denoiser? While your analytical predictions for the crossover point would not hold, demonstrating that a standard network trained with gradient descent exhibits the same qualitative phase transition as a function of the number of neurons would partially reinforce the paper's relevance.\n\nWe thank the reviewer for this suggestion. Firstly, we would highlight that the GMM denoisers we train in our work (eqn.s (9) and (12)) are in fact primitive MLPs, as inspection of these formulae reveals, with a softmax nonlinearity and a specific weight-tying structure. With regards to more practical architectures, we note that this experiment has been performed in prior work in the literature, and the reviewer’s suspicions confirmed. Specifically, in [1, Figure 4], the authors demonstrate under a slightly different metric for memorization (which measures similarity using a fixed neural feature descriptor model instead of comparing $\\ell^2$ distances) that U-net models trained on GMM data undergo a rapid phase transition as the number of training data is varied. The existence of these results led us to focus on theoretically characterizing the phase transition, as in the submission.\n\nPlease let us know if you have other questions or hang-ups, and we will be happy to resolve them. We look forward to a productive discussion during the rebuttal period.\n\n# References\n[1] Wang, P., Zhang, H., Zhang, Z., Chen, S., Ma, Y., & Qu, Q. (2024). Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering. arXiv preprint arXiv:2409.02426.\n\n[2] Shah, K., Chen, S., & Klivans, A. (2023). Learning Mixtures of Gaussians Using the DDPM Objective. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).\n\n[3] Gatmiry, K., Kelner, J., & Lee, H. (2024). Learning Mixtures of Gaussians Using Diffusion Models. arXiv preprint arXiv:2404.18869.\n\n[4] Chen, S., Kontonis, V., & Shah, K. (2024). Learning general Gaussian mixtures with efficient score matching. arXiv preprint arXiv:2404.18893.\n\n[5] Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., & Zhang, A. R. (2023). Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations (ICLR 2023).\n\n[6] Benton, J., De Bortoli, V., Doucet, A., & Deligiannidis, G. (2024). Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization. In The Twelfth International Conference on Learning Representations (ICLR 2024).\n\n[7] Hurault, S., Terris, M., Moreau, T., & Peyré, G. (2025). From score matching to diffusion: A fine-grained error analysis in the Gaussian setting. arXiv preprint arXiv:2503.11615."}, {"review_id": "Y1uZuzVzc7", "reviewer": "Reviewer_1zED", "summary": "The paper investigates memorization in (unconditional) diffusion models from a theoretical perspective. More specifically, the paper explores a so-called crossover point, where the training loss landscape of a diffusion model starts to prefer memorizing training data over generalizing across them. Based on this concept, the paper defines the difference between memorization and generalization. It offers a framework to identify the phase transition of a diffusion model's critical capacity when the model starts to favor memorization. This crossover point is theoretically motivated and empirically shown for experiments on Gaussian mixture models and (small) monochromatic images.", "questions": "Regarding the 3rd point stated under weaknesses, is it correct that the model always requires multiple training runs with different capacities to identify and predict the crossover point? Or is it possible to predict the number of parameters for, e.g., a model trained on ImageNet?", "limitations": "The paper mainly discusses future work and potential extensions in the conclusion section but lacks a clear statement of the method's limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper lays down a well-motivated theoretical foundation for memorization in diffusion models. The core of the analysis -- the crossover point -- is derived in detail, and important, relevant research and foundations are extensively cited. The proposed framework offers fascinating insights into understanding memorization in diffusion models.\n- The paper is, in general, well-written and presents all findings appealingly.\n\n**Weaknesses:**\n- My primary concern is that the theoretical concept of the crossover point is only shown for particular synthetic cases. While the paper promises a generalization to more complex datasets, empirical proof is missing. While I understand that training diffusion models can be costly, I would have expected to see at least some experiments on small scales, such as CIFAR-10.\n- The paper writing is based heavily on mathematical symbols. While Tables 1 and 2 provide a good lookup, it is sometimes hard to follow without frequently looking up symbols from previous sections. Repeating symbol names or using names instead of symbols would improve the readability of the paper. This also includes figures, e.g., Figures 1 and 3, where stating the name of the symbols on the axis would improve their readability.\n- If I understood the method correctly, it seems like the method requires reference models / multiple training runs with different model capacities to identify the phase transition in practice. Assuming one wants to train a larger diffusion model without a valid reference, one cannot a priori define a sufficient number of parameters to avoid memorization.", "was_revised": true, "final_justification": "After reading the paper and going through the reviews and rebuttals, I am not 100% convinced that the findings generalize to more complex and capable diffusion models. However, I think the paper provides interesting insights from a theoretical perspective. Therefore, I support a (weak) acceptance.", "rebuttal": "We thank the reviewer for their careful review of our submission, and for recognizing our work’s “well-motivated theoretical foundation” and “fascinating insights”. We respond point-by-point below, and look forward to discussing further.\n\n> My primary concern is that the theoretical concept of the crossover point is only shown for particular synthetic cases. While the paper promises a generalization to more complex datasets, empirical proof is missing. While I understand that training diffusion models can be costly, I would have expected to see at least some experiments on small scales, such as CIFAR-10.\n\nIn our view, the memorization-generalization interplay in diffusion models is a holistic phenomenon with numerous interrelated causes (as evidenced by the diversity of explanations proposed in the literature [1-4]), and to develop a mathematical understanding of it, it is necessary to study it in a setting in which these different causes can be disentangled, precisely varied, and their effects examined. In this sense, our work aims to build a *theoretical foundation* for studying memorization in diffusion models. Due to this scope, we do not attempt to rigorously mathematically analyze real data and real networks in our work. However, we note (here and in the paper) that experimental results in our simplified laboratory setting reproduce phenomena observed in previous empirical work studying memorization and generalization in real diffusion models trained on real data (i.e., larger than CIFAR10 scale), see e.g. [5].\n\n> The paper writing is based heavily on mathematical symbols. While Tables 1 and 2 provide a good lookup, it is sometimes hard to follow without frequently looking up symbols from previous sections.\n\nThank you for the feedback; while our work is intended to be theoretical in nature, and therefore involves significant amounts of mathematics, we agree that the notation indeed needs improvement, and we commit to streamlining it in the revision. With the additional content page, we will also add descriptions of the relevant variables before using them, e.g., “the number of components $K$”.\n\n> If I understood the method correctly, it seems like the method requires reference models / multiple training runs with different model capacities to identify the phase transition in practice. Assuming one wants to train a larger diffusion model without a valid reference, one cannot a priori define a sufficient number of parameters to avoid memorization.\n\n> Regarding the 3rd point stated under weaknesses, is it correct that the model always requires multiple training runs with different capacities to identify and predict the crossover point? Or is it possible to predict the number of parameters for, e.g., a model trained on ImageNet?\n\nIndeed, in our methodology, we need to build a predictive model by learning the loss weighting $\\tilde{\\lambda}$.  However, as we show in the paper, we can use the learned loss weighting to extrapolate to different (held-out) training configurations, and predict the location of the phase transition from generalization to memorization with high accuracy. Notice that this is similar to the current literature on *scaling laws*, where a simple law relating the size of the model to some property (such as the performance on some benchmark) is fit using a number of trained models, and then used to predict the given property for different model configurations without training those models [6]. In this context we can reframe a contribution of this paper as: within our laboratory, we derive a hypothesized scaling law form for memorization and demonstrate that it holds to high accuracy in practice. We humbly point out that scaling law work conventionally requires this curve-fitting and cannot give a principled hypothesis as to the form of the scaling law. As a result, we do not view this limitation as a major weakness, since in fact it constitutes an improvement (a prediction about the form of the scaling law) compared to numerous works in this domain.\n\nPlease let us know if you have other questions or hang-ups, and we will be happy to resolve them. We look forward to a productive discussion during the rebuttal period.\n\n# References\n\n[1] Kadkhodaie, Z., Guth, F., Simoncelli, E. P., & Mallat, S. (2023). Generalization in diffusion models arises from geometry-adaptive harmonic representation. In arXiv [cs.CV]. arXiv.\n\n[2] Kamb, M., & Ganguli, S. (2024). An analytic theory of creativity in convolutional diffusion models. In arXiv [cs.LG]. arXiv. \n\n[3] Finn, E., Keller, T. A., Theodosis, M., & Ba, D. E. (2025). Origins of creativity in attention-based diffusion models. In arXiv [cs.LG]. arXiv.\n\n[4] Niedoba, M., Zwartsenberg, B., Murphy, K., & Wood, F. (2024). Towards a mechanistic explanation of diffusion model generalization. In arXiv [cs.LG]. arXiv.\n\n[5] Zhang, H., Zhou, J., Lu, Y., Guo, M., Shen, L., & Qu, Q. (2023). The Emergence of Reproducibility and Consistency in Diffusion Models. In arXiv [cs.LG]. arXiv.\n\n[6] Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J. and Amodei, D., 2020. Scaling laws for neural language models. In arXiv [cs.LG]. arXiv."}]

Author rebuttal:

Meta-review: The paper presents a theoretical framework for understanding memorization in diffusion models, focusing on the point at which these models transition from generalization to memorization. The authors introduce the notion of a “crossover point,” where the training loss landscape begins to favor memorizing training samples rather than generalizing across them. The framework thus provides a theoretical grounding for the phase transition in model capacity that is responsible for memorization. The authors further validate this crossover point empirically, showing that it reliably predicts the onset of memorization in practice.

**Strengths**: The paper provides a well-grounded theoretical analysis of memorization in unconditional diffusion models and provides interesting insights on the memorization-generalization behavior in diffusion models which were previously unstudied theoretically.

**Weaknesses**: The main concerns that remain after the rebuttal are about the transferability of the findings in the simplified setup to state-of-the-art diffusion models, which might limit the actual impact of the work. While it would have been nice to extend the evaluation in the paper towards more real-world datasets, the argument from the rebuttal that these experiments are already conducted in prior (empirical) work seems convincing.

During the rebuttal, concerns about the choice of $\lambda_s$ were resolved and the authors committed to improve the presentation to make it more clear. Additionally, concerns about novelty of the finding were adequately addressed in the rebuttal by arguing that while the phenomenon is "expected", theoretical assessments were missing. Thereby, the paper closes a gap and makes a valuable contribution.
Decision: Accept (poster)