Provable Separations between Memorization and Generalization in Diffusion Models
Authors: Zeqi Ye
Qijie Zhu
Molei Tao
Minshuo Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Memorization and Generalization
Diffusion Models
Statistical Estimation
Network Approximation
Primary area: generative models

Abstract: Diffusion models have achieved remarkable success across diverse domains, but they remain vulnerable to memorization---reproducing training data rather than generating novel outputs. This not only limits their creative potential but also raises concerns about privacy and safety. While empirical studies have explored mitigation strategies, theoretical understanding of memorization remains limited. We address this gap through developing a dual-separation result via two complementary perspectives: statistical estimation and network approximation. From the estimation side, we show that the ground-truth score function does not minimize the empirical denoising loss, creating a separation that drives memorization. From the approximation side, we prove that implementing the empirical score function requires network size to scale with sample size, spelling a separation compared to the more compact network representation of the ground-truth score function. Guided by these insights, we develop a pruning-based method that reduces memorization while maintaining generation quality in diffusion transformers.

Published as a conference paper at ICLR 2026
PROVABLE SEPARATIONS BETWEEN MEMORIZATION AND GENERALIZATION IN DIFFUSION MODELS
Zeqi Ye
Qijie Zhu
Northwestern University
Northwestern University
zeqi htw.:///www..tttttetttteeeeeead
Molei Tao
Minshuo Chen
Georgia Institute of Technology mtao@gatech.ed.
Northwestern University minshuo. www.ehhrrttwwsttee..cc
ABSTRACT
Diffusion models have achieved remarkable success across diverse domains, but they remain vulnerable to memorization memorizaiion-xeprrddcccc reproducing training data rather than generating novel outputs. This not only limits their creative potential also raises concerns about privacy and safety. While empirical studies have explored mitigation strategies, theoretical understanding of memorization remains limited. We address this gap through developing dual-separation result via two comple- mentary perspectives: statistical estimation and network approximation. From the estimation side, show that the ground-truth score function does not minimize the empirical denoising loss, creating a separation that drives memorization. From the approximation side, we prove that implementing the empirical score function requires network size to scale with sample size, spelling a separation compared the more compact network representation of the ground-trute -ruth score function. Guided by these insights, we develop pruning-based method that reduces mem- orization while maintaining generation quality in diffusion transformers.
INTRODUCTION
Diffusion models have emerged as one of the most powerful families of generative models, achieving -fte-offthhe-art performance across a wide range of tasks (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2020a;b; Kong et al., 2020; Mittal et al., 2021; Jeong et al., 2021; Huang et al., 2022; Avrahami et al., 2022: Ulhaq & Akhtar, 2022). Applications span image synthesis (Nichol et al., 2021; Yang et al., 2024), molecular design (Weiss et al., 2023; Guo et al., 2024), and time-series modeling (Tashiro al., 2021; Alcaraz Strodthoff, 2022), where diffusion models consistently generate samples of high fidelity. Their remarkable empirical success has established them leading paradigm in modern generative modeling.
Despite these advances, diffusion models have raised critical concerns. A central is memo- rization, where trained models reproduce training data instead of generating genuinely novel sam- ples (Gu et al., 2023; Stein et al., 2023; Webster, 2023; Kadkhodaie al., 2023; Rahman et al., 2025; Chen et al., 2024). Such behavior undermines the creative potential of generative modeling and threatens the promise generalization (Somepalli al., 2023; Carlini et al., 2023). Memoriza- tion also leads to serious risks for data privacy and intellectual property, as training datasets may include copyrighted works or sensitive information (Ghalebikesabl et al., 2023; Cui al., 2023; Vyas et al., 2023).
growing body of research has attempted characterize and mitigate memorization in diffusion models. Empirical studies have explored its correlation with data duplication, training procedure, and model architecture and capacity (Somepalli et al., 2023; Gu al., 2023; Stein et al., 2023), and proposed defenses such as dataset de-duplication, modified training objectives, improved sam- pling strategies (Wen et al., 2024; Ross et al., 2024; Wang et al., 2024). These methods provide
"Equal contribution.

---

Published as a conference paper at ICLR 2026
valuable heuristics yet leave principles underneath their success underexplored. In parallel, theoret- ical investigations have begun to analyze memorization from a statistical perspective. For instance, asymptotic analyses, where both sample size and data dimension grow proportionally, have provided insights into the interplay between data availability, model complexity, and generalization (Raya & Ambrogioni, 2023; Biroli et al., 2024; George et al., 2025). However, these analyses do not fully explain memorization in practical, finite-sample regimes, leaving open a fundamental question:
Can we disentangle memorization from generalization in practical regimes and mitigate it?
In this work, we take a step toward addressing this question. We develop a non-asymptotic analysis that theoretically explains the emergence of memorization through the dual lenses of statistical esti- mation and neural function approximation. Our analysis reveals that memorization is fundamentally tied to the statistical properties of the training objectíve-tth denoising score matching loss, and the approximation capacity of score neural networks. More specifically, from the statistical estimation side, we show that the ground -ruth score function does not minimize the empirical denoising score matching loss, leading to an inherent gap that drives memorization. From the approximation side, we establish results demonstrating that the empirical score function demands network size scaling with the sample size, whereas the ground-truth score admits compact representation. Guided by these insights, we explore empirical consequences and mitigation strategies. Our experiments not only validate the theories but also introduce a pruning-based method that reduces memorization while maintaining generation quality for diffusion transformers.
Our contributions summarized follows.
We show that the denoising score matching loss admits an inherent gap between the ground-truth score function and the empirical score function (Proposition 4. Furthermore, for mixture models, we provide a lower bound on the gap in Theorem 4.3, which provides a formal characterization of how memorization arises from statistical perspective.
Statistical separation theory:
We establish bounds on neural networks approximating both ground-truth and empirical score functions in Theorem 5.1. Our results reveal that the ground- truth score function admits a compact neural representation, whereas approximating the empirical score function requires the network size to grow with the sample size.
Neural architectural separation theory:
Guided by our theory, we conduct experiments in Section 6 that (a) validate our insights regarding memorization and generalization in diffusion models, and (b) propose mitigation strategies that reduce memorization while preserving generation quality.
Notations: For a vector z, we use ||æ||e denote its Euclidean norm, ||=||1 to denote its l noror, and || to denote its -orm. For a matrix A, ||Al|ee and ||A||F denote its spectral norm and Frobenius norm, respectively, and ||A|]] max,, AAgl-- We use O--) to suppress multiplicative constants in upper bounds, while O-) further suppresses logarithmic factors. Similarly, ?(-) sup- presses multiplicative constants lower bounds, and ((-) suppresses constants both upper and lower bounds.
RELATED WORK
Memorization and generalization in diffusion models have drawn increasing attention in recent years. In this section we provide overview of progress on both empirical and theoretical sides.
From an empirical perspective, memorization is significant issue observed across various settings, raising practical concerns about privacy, copyright, and model generalization (Ghalebikesabi et al., 2023; Cui et al., 2023; Vyas et al., 2023). This phenomenon widely identified in different do- mains, and researchers have revealed several contributing factors, such as training dataset size and score network size, and have proposed corresponding general mitigation methods like data augmen- tation and data de-duplication (Somepalli et al., 2023; Gu et al., 2023; Stein et al., 2023; Webster, 2023; Kadkhodaie et al., 2023; Rahman et al., 2025; Chen et al., 2024). More targeted mitigation methods have also been developed recently, including tracing memorized samples to network chitectural activations for pruning- based remedies (Chavhan et al., 2024; Hintersdorf et al., 2024), excluding trigger tokens (Wen et al., 2024), and penalizing manifold memorization (Ross et al.,
2

---

Published as a conference paper at ICLR 2026
2024). Interested readers may refer to recent survey (Wang et al., 2024) for more comprehensive exposure of contributing factors and mitigation methods for memorization.
From theoretical perspective. memorization in diffusion models has been analyzed from sta- tistical physics perspective, with focus on phase transition phenomena (Biroli et al., 2024; Li et al., 2023; Ambrogioni, 2023; Ventura et al., 2024; Raya & Ambrogioni, 2023; Sakamoto al., 2024; Pavasovic et al., 2025). For example, Biroli et al. (2024) relate the sample generation pro- cess memorization and generalization of diffusion models by identifying critical transitions in generation trajectories. George et al. (2025) use asymptotic analysis of random-feature denoisers, which are functionally equivalent to score networks, to characterize learning curves and reveal the inherent trade-offs between generalization and memorization. Bonnaire et al. (2025) provide an asymptotic analysis of the training dynamics of random-feature denoisers, identifying generaliza- tion-memorization phase transition and examining how network architectural regularization miti- gates memorization, with their theoretical findings supported by extensive numerical experiments. Baptista et al. (2025) also investigate the dynamics of empirical score matching through dynamical- systems lens, identifying a training-time imneralizationmmmmriiina how various forms of regularization help prevent memorization. Other lines of work emphasize the role implicit bias underparameterized denoisers (Kamb & Ganguli, 2024; Niedoba et al., 2024; Vastola, 2025) and how dataset statistics shape model's generalization behavior (Lukoianov al., 2025).
transition and demonstrating
During the preparation of this manuscript, 2025), where memorization and generalization properties in well-separated Gaussian mixture distri- butions are studied. By considering a specific type of denoiser parameterized by Gaussian mixture, they demonstrate a sharp transition from generalization to memorization as the capacity of the net- work increases. Different from their study, our analysis holds for generic uub-Gaussian distributions and establishes a statistical separation theory. In addition, we analyze the representation power of general score neural networks and show another separation for approximating empirical and ground- truth score functions. Based on our theoretical insights, we further develop mitigation methods to improve generalization.
aware of closely related work (Buchanan al.,
DIFFUSION MODEL AND DATA DISTRIBUTION REGULARITY
In this section, we briefly review the continuous- time formulation of diffusion models and introduce the structural assumptions on the data distribution that will be used throughout our analysis.
Score-based diffusion model A score-based diffusion model aims learn and sample from an unknown data distribution Paata by estimating the score function (Song & Ermon, 2019; Ho et al., 2020; Song et al., 2020a;b). It consists of coupled forward and backward processes. We adopt a continuous-tine description, where the forward process is
1 xtdt dX2 = -
X,dt + dB, for Xo ~ Patta and B, is standard Brownian motion.
The forward process gradually corrupts the data distribution by Gaussian noise injection. Here Paata represents the ground-truth data distribution. We denote P; the marginal distribution of X at time L. and pt the corresponding density function. In practice, the forward process terminates at a sufficiently large time 7'.
The backward process reverses the noise corruption in the forward process- referred to as denoising for new sample generation. Mathematically, the backward process is
xxx;=
2 X, Vlogpr- dt dB, for Xo Pr.
where B, is another Brownian motion and V log is the function. To simulate the backward process, one needs to estimate the score function using samples from the data distribution.
Score estimation. We collect i.i.d samples D {x1 (2,,-22-00 ..., ..., from the data distribution Panta. we estimate the score function by minimizing the following denoising score matching loss:
[(s) = _ 4 Ei (x,, s) with l(F,, 8) = Ex,1Xo=R [||
X.-axm: s(x,, 1) (3.1)
3

---

Published as a conference paper at ICLR 2026
where og = and of = =1-e Note that tu is an early-stopping time to prevent score up and secure numerical stability (Song et al., 2020b; Ho et al., 2020). The estimator s is parameterized by a large-scale neural network such UNet (Ronneberger et al., 2015) transformer (Peebles & Xie, 2023).
Empirical and ground-truth score function. Although the primary focus of optimizing (3.1) is to estimate the ground-truth score function V loggr Pt. the use of finite collected samples introduces bias towards the so-called "empirical score function". More specifically, we denote Patta Ei=1 s, the empirical data distribution. Let P. be the marginal distribution of the forward process if the initial state Xo follows Pdnta. In fact, E.=, N(auTa, of is Gaussian mixture with mean and variance dependent on time t. Consequently, Pe induces the empirical score function defined as
V (1:)) - E Ei (x(2))(xx+ ay3:),
where w. (x;)) is weight function; see detailed derivations in Appendix A.2.
An important property of the empirical score function is that it is the global minimizer of (3.1). Moreover, using the empirical score function, diffusion models only reproduce training data points instead of generating novel tomples-known as memorization. Our theory in the sequel focuses on distinguishing statistical behavior and representation requirement empirical and ground-truth score functions, providing insights on the emergence of memorization.
Data distribution regularity To study different properties of empirical and ground-truth score functions, consider sub-Gaussian data distributions with Hölder smoothness. These com- monly adopted regularity conditions in statistical literature and recent advances in the theory of diffusion models (Wasserman, 2006; Fu et al., 2024). We introduce Hölder regularity first.
Definition 3.1 (Hölder norm). Let 3 = + 0 be smoothness parameter, with integer and E [0, 1). For a function f IRd +
[8] an
R, its Hölder norm is defined as
10th (xx f²)
|| (ure)) =
sup |8 [(x) + max sup =| xマッ
a:I|alli<s x
| v16
where a is multi-index. We say f is 8-Hölder if [[ < oc. The Hölder ball of radius B is defined
H B) (f [f: - lff (R)))) < <).
We now specify class of Hölder density functions that exhibit sub- Gaussian tail behavior.
Definition 3.2 (Sub-Gaussian Hölder density). Let C. and ey 0 be two positive constants. For any Hölder index 3 > 0. let f H (1Rd, ( B) for constant radius B 0 with inf, f(x) 2 cz. density function p is sub-Gaussian Hölder if
p(x) = exp( =xxp[=((|<=(//]] f(x).
Since f is uniformly upper bounded, it holds that p(z) B Bexp((CC|<=11//3 which encapsu- lates sub-Gaussian densities widely studied in classical statistical literature (Wasserman, 2006). The lower bound on ensures the regularity of the ground-truth score function, it is well-known that the regularity the score function can arbitrarily bad near low-density regions (Vahdat et al., 2021; Song & Ermon, 2020). Definition 3.1 is adopted in Fu al. (2024) for establishing min- imax optimal rate conditional diffusion models. Yet our analysis tackles a more fine-grained understanding of the generalization capability of diffusion models.
STATISTICAL SEPARATION: GROUND-TRUTH SCORE DOES NOT MINIMIZE DENOISING SCORE MATCHING
In this section, we systematically show that the ground-truth score function does not minimize the denoising matching loss (3.1). In particular, there exists gap in the loss evaluated at the
4

---

Published as a conference paper at ICLR 2026
empirical score function and at the ground-truth score function. The gap, perhaps surprisingly, may not vanish with polynomially many training samples. To begin with, we define
Loss-Gag; = Σ
(4s (zi, Vlogre) le (zi, V logg:))
==1
as the gap between the score matching loss at time L.
4.1 Loss-Gap; IS FISHER DIVERGENCE
We relate Loss Gap; the well-known Fisher divergence (Johnson & Barron, 2004; Holmes & Walker, 2017; Yang et al., 2019; Yamano, 2021). Fisher divergence has a fundamental connection to classical central limit theorems (Johnson & Barron, 2004) and has been widely adopted in machine learning and Bayesian inference (Hyvärinen & Dayan, 2005; Hyvärinen, 2007; Yang et al., 2019), change detection (Moushegian et al.. 2025), and hypothesis testing (Wu et al., 2022). We state the formal result in the following proposition.
Proposition 4.1. For any time S T. it holds that
Loss-Gapa Fisher P.,, Pt),
where the divergence Fisher Pe. P.) x
P. V log (X) log px
The proof is provided in Appendix A. Loss is analogous to the generalization bound of the empirical score function log Pt. but fundamentally different. A generalization bound evaluates the deviation of V log Pt from V log Pt under the ground-truth data distribution Pt. Here, Loss- -Ga,, evaluated under the empirical distribution Pt. Interestingly, Fisher divergence is not symmetric and Fisher( Pt. P:) coincides with the generalization bound of V log ft. Existing literature presents fruitful studies on the generalization properties of diffusion models (Oko et al., 2023; Chen et al., 2023; Wibisono et al., 2024). Yet, the established analyses cannot be directly applied to our setting. Indeed, bounding Loss Gap, can be much more involved due to its intricate dependence on the empirical score function and the loss evaluation over the same empirical data points. In the following section, we show a lower bound on Loss-Gaps under mixture models.
4.2 QUANTIFYING THE LOss GAP IN MIXTURE OF DISTRIBUTIONS
We instantiate Paata to a mixture of K components with an equal prior, namely Σx
Paata K k- p(k),
(Mixture Model)
where each component p(k) admits density (A)), and we denote by ( p(k) random variable drawn from the k-th component with mean (*) = (a(*) and covariance Cov[X(()] = Σ. Mix- ture Distributions align well with real-world datasets, which often exhibit multi-modality. For exam-
ple, image datasets may contain distinct categories, such as cats and dogs in CIFAR-10 10 (Krizhevsky al., 2009), that correspond to different components. For each component in the mixture model, we impose the following assumption.
Assumption 4.2 We represent (*) as (A) = +((k) and assume & is a unit variance, entry- wise independent sub-Gau GoosGaussian vector with ||E|| O(), where || llw. denotes the sub- Gaussian norm (see Definition 3.4.1 in Vershynin (2018)). We also assume that ||E||2 = 0(1),|/E] = (Vd)), and _ admits the sub-Gaussian Hölder density defined in Definition 3.2. Additionally. we assume [| 112 O(va).
Assumption 4.2 ensures samples generated from the mixture are well separated with high proba- bility when log(n) = (d). We define the minimum component separation distance as Amin = minjæ: (| - μ (+)) Equipped with these, we are ready to state a lower bound on Loss -Gapt-
Theorem 4.3 (Lower bound on Loss-Gap;). Suppose Pasta takes the form (Mixture Model) with each component satisfying Assumption 4.2. Further assume the separation distance Amin = ( For to and / verifying log(ot.) ()(d) and log(at,) O(- log d) and sample size log (d), it holds that
= 2(d6,² + +r(()) for all / E [l,,4 41],
Ep
5

---

Published as a conference paper at ICLR 2026
where Ep denotes expectation with respect to the dataset D. The proof of Theorem 4.3 is provided in Appendix A.2. We present several discussions.
Small t and large variance amplify the gap Theorem 4.3 says that for polynomially many training samples, Loss-Gap, is not negli- gible in the small-/ regime. We visualize Loss-Gap; in a Gaussian mixture setting in Figure 1. The do, term arises from the Gaussian noise injected during data corrup- tion, while the tr(E) term originates from the within-componen variance. The effect of larger variance on increasing the loss gap can be understood through the Fisher divergence be- tween Pe and Pt. For the same number of sam- ples, larger within-com within-component variance makes the samples sparser in space, leading larger Fisher divergence between the Gaussian mix- ture P. formed by the samples and the true dis- tribution Pt. Although the divergence vanishes as n + co, the convergence rate n -1/d is subject to the curse of dimensionality shown in Weed & Bach (2019).
50
LossCap
0.25
t=
0.75
20
1.0
10
0
10P Sample Size (n)
10"
Figure I: Smaller leads to larger Loss-Gapi- When sample size n is not sufficiently large, the gap is non-negligible.
Gap leads memorization Using Theorem 4.3 and revisiting (3.1), we can derive
Ep (V log - (oVlog = T
Ep [Loss- 2 log(1/to) d + (41 - to) tr())
This highlights an important mechanism of memorization: the training loss gap between the ground- truth score and the empirical score is non-negligible. Therefore, strong optimizers, c.g., Adam and AdamW, tend to drive a sufficiently expressive score network to learn the empirical score rather than the ground-truth score during training. This effect is more pronounced in higher dimensions.
Extension to bounded support Our analysis also applies mixtures of well-separated compo- nents with bounded support. The key step establishing Theorem 4.3 is to prove a reduced-form approximation the empirical and ground-truth score functions, respectively. More specifically, for given noisy state X Pe generated by injecting Gaussian noise into the empirical data points Ti we argue that V log (X) 2 -0, 2 (X a,z;). Similarly, the ground-truth score function is dominated by V (X)
V logg (k) (X), where x, sampled from the k-th component and Pi is the density of the marginal distribution via applying diffusion process to the p(k), These ap- proximations valid thanks to the separation among the components. Bounded support naturally ensures this separation and hence the result follows.
5 ARCHITECTURAL SEPARATION: GROUND-TRUTH SCORE ALLOWS COMPACT REPRESENTATION
Section 4 establishes that Loss-Gap, does not vanish in the small-t regime, implying that train- ing sufficiently expressive neural network with a strong optimizer can bias the training towards the empirical score function. Yet, it remains unknown whether a network is expressive enough. In this section, we investigate the representation requirements for the ground-truth and empirical score functions using ReLU networks and identify another gap in the complexity of the network architecture.
For simplicity, we focus on feedforward ReLU networks, while extending to other network ar- chitectures does not impose substantial challenges. We define ReLU network architecture as F(W, L, NN,, where W. L and N are the width, depth, and non-zero parameters of the network. More specifically, we have
F(W, N) { f()) A ReLUUAA ReLU ReLU(AA + b.)...)
.... + bL-1) 1) + by.

---

Published as a conference paper at ICLR 2026
where A E Rd with di S W for 1 L. and Els ||Aill + lbbi Nj.
Here da represents the data dimension and du represents the output dimension. The following theorem establishes approximation guarantees of the ground-truth and empirical score functions.
Theorem 5.1. Suppose that the density function of Patta satisfies the sub-Gaussian Hölder density condition in Definition 3.2 with Hölder index B. For any sufficiently small > O, choose the early- stopping time to satisfying log to = (log e) and the terminal time T = (log Then there exist network architectures Fi L1. N1) and F2 L2, N2) giving rise to
$1 (W,. NI) and 82 F2 L2, N2).
such that for any E [to, TJ. holds that
Ep E ExirA
[| (X,,
V
Plog
[/]
Σ e
and
(5.1)
o%
Vlogp: Ep E Ex ||s2(X(,))
[1]]
6
(5.2)
0,
The configurations of Fi and F22 are
W1 (n log c = 2lo
Ni = Ön log" c and
(5.3)
W2 ō (e # log E t L Ō(log* 2 :
N2 = Ō ( - log E e.)
(5.4)
The proof is provided in Appendix B. The key idea of the proof is rewrite the score function as v Pr = (x)(2)/pc (x) and then construct ReLU networks for approximating the numerator and denominator separately. Note that (5.1) is equivalent to the denoising score matching loss (3.1). Thus, minimizing (3.1) over sufficiently large network identified in (5.3) using strong optimizer will bias training toward the empirical score function. Probing the network size upper bounds and the corresponding approximation error, we make the following interpretations.
Network size depends on sample size The configuration of the network architecture L2,11 depends on the sample size n and the desired approximation error c, whereas the configuration of the ground-trutt truth network 82 depends on c $ More specifically, as n increases, the required width W and the total number of parameters N for Fi will increase. This distinction highlights the potentially greater complexity involved in approximating the empirical score function, as it corresponds to a Gaussian mixture distribution with components.
Sample Duplication and Memorization Empirical observations, such as those Somepalli et al. (2023), show that training sample duplication plays a significant role in memorization. From our insights in Theorem 5.1, sample duplication reduces the complexity required for the network to represent the empirical score function, thereby making memorization more likely. As stated in (5.3) and (5.4), when the dataset contains n i.i.d. samples, approximating the empirical score requires both the network width and the number of non-zero parameters to scale with n, while the corresponding quantities for approximating the true score do not depend on the sample size. However, if m samples are duplicates of the remaining n i.i.d. samples, then from a theoretical viewpoint, the dataset effectively has size n m < n. This implies that duplication makes the empirical score easier to represent and requires a less expressive network. Consequently, duplication makes memorization more likely to appear.
Different sensitivity to time We also observe that the approximation errors in (5.1) and (5.2) exhibit a distinction in the dependence on variance of The empirical score function reproduces the empirical training data distribution Paata which does not have smooth density function. Con- sequently, the empirical score function becomes highly irregular when t approaches 0, making substantially more difficult to represent. On the contrary, the ground-trui -trtt score function possesses better regularity as the data distribution satisfies the sub- Gaussian Hölder condition. We dive deeper into this regularity contrast in the sequel.
7

---

Published as a conference paper at ICLR 2026
Lipschitz continuity of score functions We investigate the Lipschitz continuity of score functions by computing the Hessian matrix of log density. As shown in Lemma C.1 in Appendix C, we have
V log = 1
Cov[Xo|X, =z1].
oi
The same result applies to the empirical density Py by replacing Cov = 21] with the em- pirical counterpart induced by training samples. For small time t, we show that the Lipschitz coefficient the supremum operator norm of the Hessian of the empirical score is bounded by aa, min ||=; - z IlB), which depends on the separation of the training samples and vari- ance o.. contrast, the Lipschitz continuity of the ground-trut score of sub- Gaussann Hölder distribution in Definition 3.2 behaves much better. As a concrete example, for Gaussian distribu- tion Pdata [N,, E), denote Amin (E) as the smallest eigenvalue of Σ, have
v2 logp: ll2 = of + Amin (2) = (()) for any t.
Weight decay effectively control the Lipschitz continuity Weight decay controls the Lipschitz continuity of neural networks by penalizing the Frobenius norms of the weight matrices (Krogh & Hertz, 1991; Loshchilov & Hutter, 2017: Zhang et al., 2018). It has been implemented widely for training large-scale complex neural networks. Motivated by the separation in Lipschitz coefficient, we demonstrate the effectiveness of weight decay for mitigating memorization in Section 6, as the score network can hardly represent the empirical score function with well-controlled smoothness.
6 NUMERICAL RESULTS
We conduct experiments on both simulated Gaussian mixture dataset and CIFAR-10 (Krizhevsky al., 2009) to validate our theoretical insights and evaluate the effectiveness of our proposed theory- driven memorization mitigation strategies.
6.1 EXPERIMENTS ON GAUSSIAN MIXTURE DATASET
We explore how network size, training sample size and data dimension affect generalization and memorization. Additionally, demonstrate that weight decay and network pruning effective remedies for memorization, which validates our theoretical insights. For the purpose of evaluating memorization numerical experiments, following Buchanan al. (2025); Yoon et al. (2023), we identify memorization as follows.
Given a training dataset {a:);_1 and trained diffusion model M, we say that sample Fnew generated by .M is memorized if nnew Ta) 112 < fll -x 12. where T((k) is the k- th nearest neighbor Euclidean norm to Fnew in (m4)²_ Further, call the proportion of memorized samples within a batch of new sam- ples drawn from M the memorization ratio.
haterg Doca Gemented Eat (Mamoraed)
1 N Ia), where (k))) k E [K] are well-separated. As teaser, we set d K = 4 to visualize how network size affects memorization, which is shown in Figure 2.
We specify Patta
Figure Learning 2D Gaussian mixture with varying network sizes. Increasing the network size leads to a clear progression: from failing to capture the underlying distribution, to partial generalization, and eventually to memorization. Memorized samples generated by the largest net- work are highlighted in red.
In the following experiments, we set K = 8, and draw (k) independently from V(0,4la) We first examine the relationship between memorization ratio, training sample size n, and data dimension d. The results are shown in Fig- ure 3a. We initially fix the data dimension at d 32 while varying the training sample size
8

---

Published as a conference paper at ICLR 2026
and network size. The results indicate that larger networks exhibit stronger memorization capacity, while more training samples reduce memorization ratio. We then fix the network size (12M parame- ters) to analyze the effects of training sample size and data dimension. The results show that higher dimension leads to lower memorization as data is harder to replicate.
We then leverage theoretical insights explore potential remedies for memorization. Motivated by the theoretical insights in Theorem 5.1, we conduct further experiments to investigate the effects of network width and weight decay. The results are presented in Figure 3b. With sufficient sample size (n=10K), memorization is less likely and increasing network width promotes generalization (measured by mean likeliiooooo where higher is better), while strong weight decay is harmful. However, with reduced sample size (n=3.2K), wide networks and light weight decay both lead to a high memorization ratio and severely impair generalization, while proper network width and weight decay prevent memorization and improve generalization. These findings validate that choos- ing appropriate network widths and applying weight decay during training are effective strategies to mitigate memorization.
100
Sample
100
-2.5
Momuraatoon Rorb 196
halo
10K
32C
100K
Dara Dimenson
40
40
Sample
25
1:K
Networt Stme
Sample
Network Witth
Wugll Ducay Rote
(a) (Left): fixed data dimension with varying sample sizes and network sizes. (Right): fixed network size with varying sample sizes and data dimensions.
(b) (Left):: fixed network depth with varying widths and sample sizes. (Right): fixed network width with varying weight decay rates and sample sizes.
Figure 3: Comparison of experimental results on Gaussian mixture data, In (b), solid lines show memorization ratio, dashed lines show mean log-likelihood.
6.2 EXPERIMENTS ON CIFAR-10
Motivated by our theoretical insights and results on the effect of network width from synthetic experiments above, we propose pruning method glug-and-play approach for trained diffusion models to reduce memorization.
Pruning to mitigate memorization Pruning has been widely adopted for trained diffusion mod- els, either reduce network size for faster inference while maintaining performance (Fang et al., 2025), or to remove specific memorized samples by identifying the responsible neurons (Hintersdorf' et al., 2024). We propose a one-shot pruning method for trained Diffusion Transformers (DiTs) (Pee- bles & Xie, 2023). In particular, motivated by Theorems 4.3 and 5.1, we identify and prune attention heads that contribute least in the small-t regime, followed by fine-tuning. This forces the remaining heads to represent the data with reduced capacity, which in turn encourages the model to learn the ground-truth score rather than overfit to the empirical score. The full procedure is summarized in Algorithm 1. We adapt importance score computation from Liang et al. (2021), with details provided in Appendix D.I.
Performance of our pruning method We evaluate our pruning method on the CIFAR- 10 (Krizhevsky al., 2009) dataset. First, we randomly select subset of 5,000 samples and train a DiT on this dataset. We then apply our pruning method with diffusion time step sampling distri- bution Beta(0.8, 2) and set the pruning ratio r = 20%. For comparison, we also evaluate the original model and random pruning baseline with the same pruning ratio. Besides memorization ratio and FID, we also use the precision and recall metrics of Kynkäänniení et al. (2019). Defini- tions provided in Appendix D.2, where recall measures diversity and generation coverage. The results in Table show both our method and random pruning reduce memorization, but our method achieves higher recall and maintains competitive FID, indicating improved diversity without sac- rificing much fidelity. See Figure 4 for a comparison between the images generated by the original model and our pruned model.

---

Published as a conference paper at ICLR 2026
Algorithm One-Shot Pruning for Diffusion Transformers
1: Input:
2: Dataset D, trained DiT model M with heads Ht fhi...... h
3: Time sampling distribution T, which shall put more density small L.
4: Pruning percentage rj [0, 1]. fine-tuning steps M.
5: Compute importance scores (1( KEE + IMPORTANCESCOREEMM, D, 7).
6: Identify the set Hpunne of [m H heads with the lowest importance scores.
7: Prune all heads E Hprune from the model M.
8: for 1..... M do
9: Fine-tune the pruned model M on a batch from D.
10: Output: The pruned model M.
Although pruning slightly reduces precision, this is expected, as a high memorization ratio can artificially inflate precision by replicating training samples. For completeness, also vary the pruning ratio and report additional results in Appendix D.3.
Model
Precision (1)
Recall (1)
Memorization Ratio (%) (4)
FID (4)
Original
0.39 #0.01
0.08
73.82+1.12
15.47
Our Pruning
0.33
0.12
68.58
15.07
Random Pruning
0.30 0..02
0.09
66.87 +0.94
17. 40.15
Table I: Comparison of the original model, our pruning method, and random pruning. Each value medd over 5 runs. Best results are in bold.
CONCLUSIONS AND LIMITATIONS
In this work, we present theoretical framework to explain memorization diffusion models, ex- amining it from the perspectives of both statistical separation and architectural separation. From the statistical separation side, we show that the ground-truth score function does not minimize the denoising score matching loss, and quantify this discrepancy for generic aub-Gaussian mixture models. From the architectural separation side, we establish theoretical bounds on the approxima- tion capabilities of neural networks for both the true and empirical score functions, demonstrating the separation of network size. Finally, we validate these theoretical insights through series of experiments and propose a novel pruning method to mitigate memorization based on our findings.
While our work provides valuable insights, it has few limitations. First, although we quantify the discrepancy for sub-Gaussian mixture models-a very common case-our theoretical framework does not yet extend to heavy-tailed distributions. Second, while our pruning methods are effective our experiments, we lack the computational resources to fully validate their performance on larger datasets and models. We hope that future work can address these challenges.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
MT thankful for partial supports by NSF Grants DMS-1847802, DMS-2513699, DOE Grants NA0004261, SC0026274, and Richard Duke Fellowship. The authors also thank Giulio Biroli and Ricardo Baptista for helpful discussions that improved this work.
ETHICS STATEMENT
This work follows the ICLR Code of Ethics. We do not involve human or animal subjects, and all datasets and models are obtained under proper usage guidelines without compromising privacy. Our study avoids biases and discriminatory outcomes, does not use personally identifiable informa- tion, and poses risks to privacy security. We are committed to conducting this research with transparency and integrity.
REPRODUCIBILITY STATEMENT
We take reproducibility seriously and provide all necessary materials support it. Theoretical results are stated with clear assumptions, and complete proofs are provided in Appendix A, B and C. Experimental settings and implementation details are described in Appendix D. These materials ensure that our claims and results can be verified and reproduced.
REFERENCES
Juan Miguel Lopez Alcaraz and Nils Strodthoff.. Diffusion-based time series imputation and fore- casting with structured state space models. arXiv preprint arXiv:2208. arX1v:2288.09399, 2022.
Luca Ambrogioni. The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability. arXiv preprint arXiv:2310.17467, 2023.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18208-18218, 2022.
Ricardo Baptista, Agnimitra Dasgupta, Nikola Kovachki, Assad Oberai, and Andrew M Stuart. Memorization and regularization in generative diffusion models. arXiv preprint arXiv:2501.15785 2025.
Giulio Biroli, Tony Bonnaire, Valentin De Bortoli, and Marc Mézard. Dynamical regimes of diffu- sion models. Nature Communications, 15(1):9957, 2024.
Tony Bonnaire, Raphaël Urfin, Giulio Biroli, and Mare Mézard. Why diffusion models don't mem- orize: The role of implicit dynamical regularization in training. arXiv preprint 250::22555.7555 2025.
Sam Buchanan, Druv Pai, Yi Ma. and Valentin De Bortoli. On the edge of memorization in diffusion models. arXiv preprint arXiv:2508. 7689,, 2025.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX security symposium (USENIX Security 23), pp. 5253-5270, 2023.
Ruchika Chavhan, Ondrej Bohdal, Yongshuo Zong, Da Li, and Timothy Hospedales. Memorized images in diffusion models share subspace that can be located and deleted. arXiv preprint arXiv:2406. 18566, 2024.
Chen Chen, Enhuai Liu, Daochang Liu, Mubarak Shah, and Chang Xu. Investigating memorization in video diffusion models. arXiv preprint arXiv:24I0. 2666,, 2024.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low- oow-iienssional data. International Conference Machine Learning, pp. 4672-4712. PMLR, 2023.

---

Published as a conference paper at ICLR 2026
Yingqian Cui, Jie Ren, Han Xu, Pengfei He, Hui Liu, Lichao Sun, Yue Xing, and Jiliang Tang- Diffusionshield: watermark for copyright protection against generative diffusion models. arXiv preprint arXiv:2306.0440 2023.
Gongfan Fang, Kunjun Li, Xinyin Ma, and Xinchao Wang. Tinyfusion: Diffusion transformers learned shallow. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 18144- 18154, 2025.
Hengyu Fu, Zhuoran Yang, Mengdi Wang, and Minshuo Chen. Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory. arXiv preprint arXiv:2403. 1968, 2024.
Anand Jerry George, Rodrigo Veiga, and Nicolas Macris. Denoising score matching with ran- dom features: Insights on diffusion models from precise learning curves. arXiv preprint wwXXv:25022.00336 2025.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint arXiv:2302.13860 2023.
Xiangming Gu, Chao Du, Tianyu Pang. Chongxuan Li, Min Lin, and Ye Wang. On memorization in diffusion models. arXiv preprint arXiv:2310.02664, 2023.
Zhiye Guo, Jian Liu, Yanli Wang, Mengrui Chen, Duolin Wang, Dong Xu, and Jianlin Cheng. Diffusion models in bioinformatics and computational biology. Nature reviews bioengineering," 2 12):3361154, 2024.
Dominik Hintersdorf, Lukas Struppek, Kristian Kersting, Adam Dziedzic, and Franziska Boenisch. Finding nemo: Localizing neurons responsible for memorization in diffusion models. Advances in Neural Information Processing Systems, 01888236-88788, 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6857 2020.
Chris C Holmes and Stephen G Walker. Assigning value to power likelihood in a general bayesian
model. Biometrika, 104(2):497-503, 2017.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2595-2605, 2022.
Aapo Hyvärinen. Some extensions of score matching. Computational statistics data analysis, 51 (5):2499-2511 2007.
Aapo Hyvärinen and Peter Dayan. Estimation of normaaizddd statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104.014000 2021.
Oliver Johnson and Andrew Barron. Fisher information inequalities and the central limit theorem. Probability Theory and Related Fields, 129(3):391-40) 4099 2004.
Zahra Kadkhodaie, Florentin Guth, Eero P Simoncelli, and Stéphane Mallat. Generalization in diffusion models arises from geometry-adaptin harmonic representations. arXiv preprint arXiv:2310.025511 2023.
Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. arXiv preprint arXiv:2412.20292 2024.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint (XXi:2009.007660 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12

---

Published as a conference paper at ICLR 2026
Anders Krogh and John Hertz. A simple weight decay improve generalization. Advances in neural information processing systems, 4, 1991.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric assessing generative models. Advances in neural information processing systems, 32, 2019.
Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selec- tion. Annals of statistics, pp. 3302-133,, 2000.
Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties of diffusion models. Advances in Neural Information Processing Systems, 36:2097-2127, 2023.
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. arXiv preprint arXiv:2105. 12002, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled
weight decay regularization.
arXiv preprint
17Xiv:1711.05101,, 2017.
Artem Lukoianov, Chenyang Yuan, Justin Solomon, and Vincent Sitzmann. Locality in image dif- fusion models emerges from data statistics. arXiv preprint arXiv:2509.09672, 2025.
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint crXiv:2103.16009 2021.
Sean Moushegian, Suya Wu, Enmao Diao, Jie Ding, Taposh Banerjee, and Vahid Tarokh. Robust score-based quickest change detection. IEEE Transactions on Information Theory, 2025.
Ryumei Nakada and Masaaki Imaizumi. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1-38, 2020.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2112.10744 2021.
Matthew Niedoba, Berend Zwartsenherg, Kevin Murphy, and Frank Wood. Towards mechanistic explanation of diffusion model generalization. arXiv preprint arXiv:241 19339, 2024.
Kazusato Oko. Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distri- bution estimators. In International Conference on Machine Learning, pp. 26517-26582. PMLR, 2023.
Krunoslav Lehman Pavasovie, Jakob Verbeek, Giulio Biroli, and Mare Mezard. Classifier-free guidance: From high-dimensional analysis to generalized guidance forms. arXiv preprint 0724095002.0744,. 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
losif Pinelis. Exact lower and upper bounds on the incomplete gamma function. arXiv preprint 20X5:2000500684.. 2020.
Aimon Rahman, Malsha V Perera, and Vishal M Patel. Frame by familiar frame: Understanding replication in video diffusion models. In 2025 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2766-2776. IEBE, 2025.
Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion mod- cls. Advances Neural Information Processing Systems, 36:66377-66389, 2023.
Herbert Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
13

---

Published as a conference paper at ICLR 2026
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241- Springer. 2015.
Brendan Leigh Ross, Hamidreza Kamkari, Tongzi Wu, Rasa Hosseinzadeh, Zhaoyan Liu, George Stein, Jesse Cresswell, and Gabriel Loaiza Ganem. A geometric framework for understanding memorization in generative models. arXiv preprint arXiv:2411.00113 2024.
Kotaro Sakamoto, Ryosuke Sakamoto, Masato Tanabe, Masatomo Akagawa, Yusuke Hayashi, Man- ato Yaguchi, Masahiro Suzuki, and Yutaka Matsuo. The geometry of diffusion models: Tubular neighbourhoods and singularities. In ICML 2024 Workshop Geometry-grounded grounded Representa- tion Learning and Generative Modeling, 2024.
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6048-6058, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025022 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-12445 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, 36:3732-3784, 2023.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based dif- fusion models for probabilistic time series imputation. Advances in neural information processing systems, 34:248044222 14888004-4-16666 168866222 2021.
Anwaar Ulhaq and Naveed Akhtar. Efficient diffusion models for vision: A survey, arXiv preprint 1.X0:2211000922.. 2022.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling latent space. Advances in neural information processing systems, 34:11287-11302, 2021.
John J Vastola. Generalization through variance: how noise shapes inductive biases in diffusion models. arXiv preprint arXhv:2504.12532, 2025.
Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, and Luca Ambrogioni. Man- ifolds, random matrices and spectral gaps: The geometric phases of generative diffusion. arXiv preprint 4xX1v22400.05898, 2024.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Nikhil Vyas, Sham M Kakade, and Boaz Barak. On provable copyright protection for generative models. In International conference on machine learning, pp. 35277-35299, PMLR, 2023.
Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, and Yi Yang. Replication in visual diffusion models: A survey and outlook. arXiv preprint arXi::2408.0000) 2024.
Larry Wasserman. All of nonparametric statistics. Springer, 2006.
Ryan Webster. A reproducible extraction of training images from diffusion models. arXiv preprint NKKv:2305.08694 2023.
14

---

Published as a conference paper at ICLR 2026
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of em- pirical measures in wasserstein distance. Bernoulli, 25(4A):2620- 2248,, 2019.
Tomer Weiss, Eduardo Mayo Yanes, Sabyasachi Chakraborty, Luca Cosmo, Alex M Bronstein, and Renana Gershoni-Poranne. Guided diffusion for inverse molecular design. Nature Computational Science, 3(10):8738822, 2023.
Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference Learning Repre- sentations, 2024.
Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. The Thirty Seventh Annual Conference on Learning Theory, pp. 4958-44884 4991. PMLR, 2024.
Suya Wu, Enmao Diao, Khalil Elkhalil, Jie Ding, and Vahid Tarokh. Score-based hypothesis testing for unnormalized models. IEEE Access, 10:71936-71950, 2022.
Takuya Yamano. Skewed jensen-fisher divergence and its bounds. Foundations, 1(2):266-664 2021.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In Forty-first International Conference on Machine Learning, 2024.
Yue Yang, Ryan Martin, and Howard Bondell. Variational approximations using fisher divergence. arXiv preprint arXiv:1905.053 00.29005002.,.... 2019.
TaeHo Yoon, Joo Young Choi, Sehyun Kwon, and Ernest Ryu. Diffusson probabilistic models generalize when they fail to memorize. In ICML 2023 workshop structured probabilistic inference & generative modeling, 2023.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 12281, 2018.
15

---

Published as a conference paper at ICLR 2026
A PROOF OF PROPOSITION 4.1 AND THEOREM 4.3
A.1 PROOF OF PROPOSITION 4.1
Proof. The proof relies on rewrite of the score functions. For the ground-truth score function and any empirical sample F3, we have
at [(a; - mo) exp( - I|= - (razol)ddPaataaaa 2
Vlog (mt -(03))
Ti
o,
{ [exp(= 207 | 04.0 11 3dd danaaaata (2o)
(
at (a: (oje(ze));
of (at a,x,)
(A.1)
oi
where in equality (i), we insert OgTi, and in equality (fi). we denote
1 2n7 || - ar 3o (1 3) data (Io) (a)) = J zo exp( Recalling the definition of le (xaz :) in (3.1) and plugging in (A.1), we obtain
J exp( 277 2. ax [ daPa tata (zp)
12
& (zi, V loggt)
Ex
o 0f
(a:
Maje (o)((X))))
n
By analogously denoting
f Ty exp( 1 I| - a4 (1 (F)) 20?
Pojo(ar)= =
f Texp( 207 | - o4 B)d Paata (v0)
we have
Σ 4 (zi, V Vog
1
E Xe
||
at 1on (x: Pop (X+))
1=1
1=1
Combining them, we have
Ct (x; - Mope Moje( (xi)) 12
Loss-Cap
E Ex (m;
of
x=1
E
Ex,
of
(as - Mojo (x.))
(A.2)
i=1
To compare the terms in A.2, suffices to fix an arbitrary time E [to, 77. Starting with the ground- truth denoising loss, we have
ai 7 (r: Moje (X.))
E
Exll
a²
1
n
Ex,12, |||0; [11a;-Pojf: "oje((,)) + (x.) - (X))(X
07 n
o3 1
|| - (
E Ex,12,
of
2
Ex mi (
(ax - (x)) T (Roje(X)) - Moo(Xx)))
+
Ex,le
(A.3)
16

---

Published as a conference paper at ICLR 2026
We claim that
0. In fact, have
2 E Para Ex,1xo (Xo (oor(XX)) T (X)) Moje Mojo (Xi))
at
of
E x, ExoXx, (Xo -Noj((x))) (Tojo(Xe) molo
o Ex, ((00)((xi) -(oo)(())))o(((())))) (Moje(X))) -ooj(
where equality (i) follows from the tower property of conditional expectation. As result, compar-
ing (A.2) and (A.3) gives rise
of Ex las [I Hojc(Xx))12) Σ
Gass-Gap;
(A.4)
oi i=1
To further simply the expression, we apply Tweedie's Formula(Robbins, 1992) and have
X[Xo|X,= = x4] of logpe(s)) xt
which immediately gives us
a 1 Ex,
- (Mo
+20 Ex, ja
Voogp((X)) V Po
n2
Then we can conclude
1 n
Loss Loss-Gap;
Σ n
E Exalz ||log V Vlogpi(X.))
EEx Vlogft(X)) - Vlogp((X)
and complete the proof.
A.2 PROOF OF THEOREM 4.3
The proof Theorem 4.3 proceeds in three steps:
Step 1. After simplifying Loss Loss-Gap, the form in (A.4), and assuming Paata follows the mixture model (Mixture Model), we can express Moj Poje weights the contribution of individual samples, while Poje weights the contribution of mixture components.
(xz) (xx) and (oje(E)) as weighted sums:
Step 2. In the small-/ regime, on high-probability event for both the diffusion noise and the samples (where their norms lie in regular range), we identify the dominant weights in (xz) and (z)) If is the diffusion-corrupiee version of training sample æ;, then (1a) is dominated by the contribution of z;, whereas Moj (x:) is dominated by the contribution of the component that generated z,, also provide explicit lower bounds on these dominant weights.
Step 3. Separating the dominant and residual terms in the weighted sums yields lower bound on Loss-Gap,-
We now proceed with the proof step by step.
A.2.1 Step 1. SIMPLIFICATION OF (A.4)
For each E [K]. let pt denote the marginal density of the forward diffusion process at time Equipped with this notation, have simpler diserete version of (A.4).
17

---

for Published as a conference paper at ICLR 2026
For (2)) we have:
(oje(ar)) 21=1 27 exp(
20; 20 0,21
11
(x))
(A.5)
===1 exp(
2 aTT
20;
expl- 24ę (
where (t (x1)) (x))
for 1 = ,,2,
Σj=1 expl 2o7 li - a13
As for (1)) noticing that
(k) (zx) Pi
(2TO7)-3 / |exp(=
1
||
(20)))
20
we have
Σk k==l/ I 20 exp(-ppppp
|2 ax (20))
207
(o))(zz))
Σ leep(=
It 0, 11 p (k) dro
2n?
(2TO?)= - " I 2o exp(
- 2 I1 a (x0
Σラ (xxe)
K
(1) (x1)
Pt
(1) (Ix)
30
(2TO?)) expl
20?
| a, (220 (zo) (1) (x)
(xx)
doro
j= Pt
K
(8) (Ire Hole (xe),
(A.6)
I xo explp ane las in--a =0 (zo)
where we denote wi (8) (xx) = E
(z,)
Moje (xx)
(xx,)
=
I expl 20z 1i meeoo (mo
E [K].
After simplification, Loss-Gap, can be rewritten
Loss Gap; = of
Ex Exilz
Σ T
(X.)
K Σ k=1
w
(X:) (8) (Xx)
oi
For the sake of simplicity, we further denote
A; A Exil
Σ W (Xi)
wi
(X:)) (X+)
(1)
(8)
A.2.2 Step 2. BOUNDING THE DOMINANT WEIGHTS WITHIN CERTAIN EVENT
We first denote = E1/25, following the notations in Assumption 4.2. We can then write the decomposition of ((k as
(()) μ (k) e, Pe, B(c] O, Cov(X(()) Cov(c)= = Σ. And thus, under Assumption 4.2, there exist some constants C1, C2 C3 such that
||>||2 S C3. = /// B[E] = 0, Covig] = la. | < C1. || - C2vd.
(A.7)
We define mapping e: [n] [K. where e(i) maps the index of the component from which is generated. Equipped with this, we can write x, ((()) c. We now define a high probability event E for sample norm and their well-separation properties. Invoking Corollary A.3, we can specify a high probability event ε,, within which the samples are well separated, and their norms are in regular range. The statement in the corollary suggests that, for 8 (0, 1). with high probability at least 1 - 5, we have
min ||q - = 25n(5/2n) ijjmm]
d
1og(n2/8), and
C
CV
18

---

Published as a conference paper at ICLR 2026
(6/222))
vu(6/2m)) d.
S inf
||=;
1
sup
|| - م 1
v
EE[n]
kE[n]
C
Thus, the following event holds with probability least 1 - 8:
E
; Zn
c
2 n (2(6/2n)
4
C
d
C
V co
log(n?/0) /6),
F',A E [n]
ye (r(8/2n)
31 21....; Zn
S iE(n)
inf
||>; - / ((((6))
Yo (8(8/2m)
$ sup iE(n]
||0
(o(1))
13
v
Similarly, we can also define another similar high probability event for Z, the Gaussian noise intro- duced by diffusion. Invoking Lemma A.I, for dz E (0, 1), with high probability at least - 8z the following event holds
& a V d 2loog(2/22) < |21|2 S V d 2 2/doo((2/6)) + log(2/62))
First, for the sake of simplicity, we can take óz exp (-d/9) and analyze in a certain range such (66/22)) x
that
With such constraints, we can easily derive the following relationship:
8C
Qt
Yu (0(2/2 2m
at
m/862m1 2m
2 or o. d+ 2lllog(252) + (20g(2)/2)
(A.8)
2
C
2
C
Additionally, we can make first-step simplifications of the weights.
According to (A.5),
1
(Xi)
1 E, expl 1 (1 X2 - 0,5 1X, a,.;113))
207
According to (A.6),
qt(X, - (z)
wi
(n(1))
(X,)
1+
qe(X: -asjulott)
kxf(()
B
C(I|X, a ||2 ||x a (c(1)))
exp
Cj
2(a + Cof)
kx)((()
The second inequality invokes Lemma A.5, which provides upper bound the ratio of q evaluated at different points.
Consequently, from the first-step simplifications, the analysis of the dominant weights reduces to the comparisons of different distances. Within & n &2, we can easily conduct such analysis.
Distance analysis Conditioned on ε O Eg, discuss the following three kinds of distances for investigating the weight behaviors.
Case 1: The distance term regarding X, = a,æ; + o,Z and p (c()) We evaluate the distance
||, (1) According to the forward process, conditioning on F, write X, as X, a + J.Z, where ~ N(O, Ia) independent X,. Thus, we derive
||, (c())()))) < ||X, aga0il22 anl|2; (e(1))
19

---

Published as a conference paper at ICLR 2026
S o < + aiV Vu(6/2n)
C
where the second inequality leverages the fact that, within E the norm of the samples are controlled. Consequently, we deduce
yul6/2n)
|| - amp(((0)) ll2 I a. d
+ aiv 2/ (2 + 2log(2/62) d
C
sanva (v0l0
1
(((6/2))
yn
(A.9)
2
C
where the first inequality leverages the fact that, within &g the norm of the diffusion noise is con- trolled, and the last inequality leverages (A.8).
On the other hand, by the triangle inequality, have
| app(((0)) 2|2 2 max {oz 2112 aullæt - 1 arl1xx 11 21
For the first term in the maximum above, we have
yu (u(6/2n)
o arllz 12 2 aoyd d 2Vllgg(200) ar
d.
(A.10)
C
Similarly, we have
anp
(000) ,1|1Z|12
(6/2mm)
d - oi / d 2v (2og(2/62) + 2 log(2/02))
2 00
01 m((52m))
d
(A.11)
2
C
where the last inequality leverages (A.8). Taking maximum over (A.10) and (A.11) leads to
yu (u(6/2n)
yn (n(6/2n)
[x (
2 max
ao ΦIV d 2v dlogg(2002)
d
d
C 2
C
(1166/21))
C d.
(A.12)
Case 2: The distance terms regarding X, a. + o,Z and (((),, h + c(i).
We only need
lower bound on the distance ||X, - a)1 1|:
|| axp (R) 1|2 ||X, - (o(1) + aap (o(i) ()) 112
2 ar aellu (c(0)) 10 - ||, - (00
1
Amin - -avd
yul88/21
1
91(5/2m)
(A.13)
2
where the last inequality leverages the definition of Amin and the upper bound in (A.9).
Case 3: The distance terms regarding 24 and x... We have
|X, a,v,112 ||x, - 0x
= ||ar(2; - x.)) + o, ||,,211
> of =1y12² 20
If c(i) =((), then by the definition of E,, we have
||>; 22112 = ||| - c 11
2 2 (6 /m2
4 d
d
log(n2/0) 18),
C
c cu
20

---

Published as a conference paper at ICLR 2026
and if c(i) # c(j), we have
||0 1-2y12 2 A
sup |<:
min
iE(n]
y 22m)
A
d.
min
C
If we set
2 1u(66/2))
2 y0 (6/2m)
4
d
A min
d
C
d
C
V
cg
log(n? 18),
we can then have a union lower bound
2 2 22)
4
||:; 2z512
C
d
C
V
co
log
(ognn2/0) /8).
Thus,
ariti
|X, 0.2i112
(1(6/21) a
d. 0e CV Co
log(n2 /6) 20, 1
C
(2666//2)) 0
d - ai CV
log(n2/00 (6)
20, (d 2v 2 (log(222))) + 2 (2g((22)))
Co
(6mm2m) dg
2 2
d log(n2/1 /6)
1
2 (116222))
d of CV
2 1
d
CO
C
of 2C'
d log(n2 /6)
y1 (1(822n) -
4V
(A.14)
where the second inequality leverages the norm range control within Eg, and the third inequality leverages (A.8).
Lower bounds of dominant weights Thus, within & n Eg, we have
(0) (Xi)
1+ E+E$ exp( 207
(1 a 1X, axxx
2 1 (n 1)
(A.15)
-aad 2Cot
(r( (d(6/2m)d 4 da log(n2/00 /0) )
Leveraging Lemma A.5 and the bounds in (A.9), (A.13), and also setting
( yu (8/22)) + v/ 92(5/2m)
+ +1]va vd.
A Amin
>
we have
qx( (k)
(c(0) (Xi) (X,)
=
+
Ax) (X, asp(((11)
C(I[x, - (x) -||X, -ag ((()))
B
exp
of
2(a + Cof)
kxf(())
B + (K- 1) exp
C 2(a + Cof)
Cj
[aaAAmin -aain--vadll avvall
(u(0/22)
g1(6/211
1/2
C
21

---

Published as a conference paper at ICLR 2026
(6(622m) aV
mn(6/2m) C
V
)²)
(A.16)
C
where the last inequality leverages the bounds in (A.9) and (A.13).
To further simplify the expressions, we shall notice that if we take K = poly(d), and log(n) = (log()) + d), we have the conditions on Amia become Amin = ((Vd). and
yr(8/2n)d
4V VV
log(n2 22().
(u(6/2m)
(((6/2m)
(a,Amin a. a.va(y
+
22
)
C
C
(2((6/2n)
11(6/2n)
- -aad
1/4
1)² (d(d)
C
C
Thus, the bound in (A.16) can simplified as
Cafd
w (o))) (Xx) 2
1 exp
exp
(A.17)
2(a + Cof)
and the bound in (A.15) can be simplified
1
(A.18)
-afd 200?
+ nexp
A.2.3 Step 3. LOWER BOUND OF THE Loss GAP
In the sequel, to simplify the derivation, we denote 0, of Cot
We now further simplify the loss gap Loss-Gapa by extracting the weights of dominating sample and component. Within E, we can write
A 2 Ex, 1000 (Xe)x:
wi
(c(i))
(Xt) Maje
(c(1)) (X+)
+
Wi
(X+)z==
Σ wM (8) (X.) M (X,) [(1(8,)
(1)
2 E
2
Xe
10,
(1) (X,)
w/ (c(t))
(x.)
Molt
(c(1)) (x+)le (
Ex,
|> (1) (Xt)xx
Σ
w, (k) (Xi) Moje (x:) (x)))1 1(8)}]
l+i
where the last inequality leverages the fact that || - v112 2 1 - -1
Plugging in the expression of Majs in Lemma A.6 gives rise to Ex, [A1(E))
Ex,,e (T( (x.) - mi (o(()) (o(() (x.)0.)
- ( 0 (c())) (x) w/ (c(1) (X,)0, 01 (c(0) (c10) (X.) wi
ar.
2 Ex,la (1(8)) (x,) - w: (c(1)) (X.)0t)
- (010 (X+) 0, (c(i) wi (c(0) (X,)0,
(c11)
Tt | 21
at
2 Ex,la. (118))[(TT)) (x.) - w/ (o(1)) xx)0 m
22

---

Published as a conference paper at ICLR 2026
w -1) (c()) -
(1((0) (X,)0= -1 2|]]] 20(0)/a?)
at
(1182) 16 (X,) w/ (o(1)) (X:)0) xi - wi (o(1))
(Xx) (.))(((00113))
Ex.la,
( (X,)0. " z 3 1(2))
+ Ex,(
at
2
- -xxl2
(wr
(d(1)
(Xi)0: (X:)0,
_
z) (( (Xx) w. (c(1)) (X.)0t)
0n
w (c(9) ( (X,) - 0,) ((())) 1(62)
- -20(0
(A.19)
The first term in (A.19) can be simplified as
(1183)
|| ( (X₂) Dr
(c(1)) ( (x.)) 21 (c(00 (xi) (x)(( 0.) (1106
Ex,(u;
(x,) -w((()) (x:)) Iz w/ (x.) 0.)(x; alo(o))) (
Exx
1182} (X.,
1 Ex Exil
1112) |n (c(1)) (x,)(1 - 00) - =dlcons/1)]
(1,10(())) w! ( (xx)) I
-x,,,z
1 (1 0,)² 1|000- (olo0n) 1|2 -Ex, [1(0)))
(l, (Xx) - w((( (xi) |13 11::12
2
- 2V 11 0,) 1 2 [ exp
(1epp (-Cord) C.. 2
Glrd 2
R + 3a(6/2n) d)
I +
exp
where the second last inequality leverages the fact that in our range (the condition of Lemma A.6, at I 1/vd), wi ol)) (X.) 2 1. the last inequality leverages the lower bound of the weight in (A.17), and the fact that within &,, supse[M] ||=.||2 suPse[m]
(1 + ||G|| I
Rinax (8(22n)) d.
The second term in (A.19) can be simplified
(c(1))
(X,)0 Z
of aj
1
2
Ex,|=
1{82}
2 f
d
1 (-Cgra)
CB.d 2
a2
where the inequality leverages the fact that 2| 2 d/3 within E2, and the lower bound of the weight in (7.17).
The third term in (A.19) can be simplified as
(woc(()) (c(0) (X,)0, az) (()( (c(1) (X.)).))
E Ex, jm;
m (c()) (X,) ((()) 1(2))
w (X,))))))))))))))))))))) TT (xm()(xx)) (x,) Brus (o()) (x,)),, 112) )
02 dt E
Ex,
ai
0, Ex
wi
(c(t) (X.) 0.) z (((())) ((2)
We now decompose this expression by adding and subtracting the term 0 S1 Ex, 1zT 2((11- 0z)(2; -((()) 1(6)) This step is designed to isolate component that is provably zero due to symmetry, leaving us with a residual term that we can then bound.
(c(0) (x.)), 2t 222 ((((() (x.) w (c(+)) (x))))
Ex,e.
23

---

Published as a conference paper at ICLR 2026
m (x.00)().)(( -0:) )1(2) [(2))
Og 21 E Ex,
0 ((1 (1-0:)(2; ( 1{E2}
01 a Ex
z 3"((0-- () - -H(0(0)
+ mi (c(10) (10g (0) (Xi) 0,e0 (c(1)) (X1))
mn (c(1)) (x,) ( 0:)x (c(1)) 1{82}
The first term in the equality above is exactly zero This is because the expectation is over Z and the vector (1 - (x) إ is a constant. The event &g is symmetrie (it depends only on 2|2)) and the Gaussian density of Z is also symmetric.
Therefore, the original cross-term is equal to the second term. We now bound the magnitude of this remaining term.
Ex.ls (wo (x)), (10)) - wt (c(0) (X,)0))
w (e(t0) (X,) - 02))((((0)) 1(82)
9,04 Ex.lx z
(X,)u, (c(0)) (X1) (()(()
(c(i)) (c(())((x)))))
(X,)²0, + 0. 11)aa
(c(1) (X))3) 0z) 1182]
(1 - wi
< 9,0e
Ex,,x 11 Ex, la, [11 w. (Xx)) (x))) 0.) (1xr113 + 11 1000))1
exp( CBid
0.0e (1 Gr
0:)
2
R22
max
(8/22)) d
1 11+epp exp( C
where the second inequality leverages Cauchy-Sehwarz, and the last inequality leverages the fact that within ε,, supte(n) || < R Mm.60/2n) d by Corollary A.3.
Collecting all the terms have
of at
Ex. lm. [11 2 0²
d
exp
C8.d 2
(1==-((o()) (((())) |2
exp
CB.d 2
Bu(6/2n)
+ = (1 - 0:)2 (xz (zi -
R22
+
d
2
+ exp
C0id 2
max
O
0,01 (1 0z)
( exp (- (eep
CBid
B5((/22)
Raax
R2
(A.20)
at
1 1 exp CDid
C
Additionally, by the estimation
(k) derived Lemma A.6, within & E2,,, we have Moje
-02d n exx
B < 2(n 2(n-1
20
sup | jE(n)
-afd 20o2
1+ nxx
-CA,d
+ +2(K 1
exp
EX
sup Moje (x,)118)) kE[R]
1+exp
C8.d
24

---

Published as a conference paper at ICLR 2026
-ofd 2Cat
2
2
71
1 + exp
nxxp
-afd 220g
+ K
1+
exp exp
exp
(- C0.d
2
C0.d
Rmax +
Bu Bu((/22)
C
d
(A.21)
We can now summarize all the conditions we have imposed as
Aminy Riax θ (i
log(n) = ((log()) + d), K = poly(d).
We focus on t [tt, t1] where is chosen to satisfy log(oto) 2 -d, t is chosen satisfy log( S - logg d. With such conditions and time range, and by further noticing that when we take log(n) (log(d) d), we have yn (ō/2n),, 30 (5/2n) (1) (recalling their definitions in Corollary A.3), we shall have
-afd n eex
(- CB.d exp
2
2C
- O(oi),
n
1 + exp
aad 2Coo
1 + exp
C0.d
which makes B and the third and fourth terms in (A.20) negligible. Thus we finally have within E we have
Loss-Gap, = a. 1
n
A.
di i=1
a7 1
Ex
[A1(82]] - 1x.12.111(88
0
of
n
8
d
Σ 0,)31
10000001
aj
n
d
(())))
2
+
0
1=1
where we shall recall that 0,
a²
Finally, by taking = exp(-d/ d/cc) we have
Paloss-Gap/) ž ED
1{8)}
00 11
d
2 ED
1{8)
+
Ep [1 1(81))
||I, p (
i=1
d + trC(ov(()))
1
ED |||2 - p(
oo
n
d
+ tr(),,
where the second last inequality leverages Cauchy-Schwarz, and we complete the proof.
A.3 SUPPORTING LEMMAS
We first present the classical lemma of x2 concentration bound, to control the range of diffusion noise Z.
Lemma A.1 (Laurent-Massart bound for x" concentration (Laurent & Massart, 2000)). Suppose a random variable X xa with degrees of freedom d. Then for any > 0, it holds that
PP[X - d 2 2v di + 2(
S exp(- (sp(-(), (), < Exp((11)
PPd - x < 2v di
25

---

Published as a conference paper at ICLR 2026
We can next derive the following lemma control the range of
Lemma A.2 (Norm Concentration of E). Under Assumption 4.2 (e satisfies the conditions in A.7),
the following bounds hold:
1. Upper Tail: For any 1j > 1/C-1
B
d [C(1+ ( - - 1 log(C(1 m))1)
P (IlE| (1 (1+n)d)= - ef
exp
(-2
2. Lower Tail: For any (1 1/C; 1),
B
exp (-2 [C - n)) log(C(1 - n))1)
( d
P = 1)) v cf
Additionally, let
2B
-(6) log
Cs 8
(6) (1++(6)) + (()) (0) (1 (1+7(0)) v=(0)(++((0). Then, for any E 10,
P y (5) d S
||l|
2 1-8. 2=(0d C
A corollary induced by Lemma A.2 is that
Corollary A.3 (Sample Separation and Norm Control). Under Assumption 4.2 (e satisfies the con- ditions in A.7), let c En be i.i.d copies of Fix (0, 1) and define
2 d
AnB
(6/2m)
log
Cj
(((82n)) (1 + r(6/2n)) /=16/2n/2++(0/221))
Yu (6/22)) = (1 + (6/ )m)) + (2 ((((22n))
Then, with probability at least 1 - 5, the following holds for all pairs + j:
C d - - d where 0 is some constant depending C, C2, Cz. Additionally, within the same event, we have
2 (6/22)))
||G - 111 z
log(n2/0)
cu
(5(62m)
Vu (u(6/2n) d, for i = 1,2,
d S 11cil12
C
C
We defer the proofs of Lemma A.2 and Corollary A.3 to Appendix A.4.1.
We denote q: the density of oue a,Z. We then provide some useful results that help us derive
useful properties of ft-
Lemma A.4 (Lemma B.1 and B.8, (Fu et al., 2024)). Let
σi
a1
t (a + Co?)
ā af + Co.
2/2
1/2
under sub-Gaussian Hölder density assumption, we have
C||211
ge(x)=
(a? + Cof) d/2
exp
2(o + Co?)
h(2,t), t),
where
1
||= - 2
M(,, 1) [(x)
(27)
exp
2
de, and ej S h1(r1)) s B.
26

---

Published as a conference paper at ICLR 2026
Equipped with this, it is also straightforward to obtain the following:
Lemma A.5 (One sided upper ratio bound for the channel). For any 21, 22 E IR we have
qr (311 exp
B
C( - - 1
(-~ 2(a + Cof)
qt (x2)
cs
We finally present the following lemma to provide estimation of Poje (az4).
(k)
Lemma A.6 (Estimation of Mojs
(k)),
For any t satisfying 2t = 52( (2(vd), and 2t @(Vd), we have
(8) Moje (mx)
1e
at
(ot a./1 + O (o)//tt)
2 0f
+ Cot
where E, the emror term, satisfies || O
The proof is deferred to Appendix A.4.2.
A.4 PROOF OF SUPPORTING LEMMAS
A.4.1 PROOF LEMMA A.2 AND COROLLARY A.3
Proof of Lemma A.2. We define the function h(x) = x- ==1-log(z)) which is positive for x * 1. The proof proceeds by first bounding the moment gennertngnnn function (MGF) of [|]l and then applying a Chermoff bound.
The normalization constant Z is defined as % (xx Leveraging of S
S B, we can bound Z as
4/2
Z 2 / cj
1n
exp((C(|== = Cf
Rd
d/2
2T
7 <
B B.exp(-CC<==1/32
JRd
C
Let M()) E[e>Meel1]] be the MGF of ||c|2. For A > 0:
/
M(A)
(11=1lB
1
/
Z
gd
1
/
21)1 -22)013))) ((x)dr.
exp
Z
Rdd
For the integral to converge, we require C - 2X > 0. i.e., X < C/2. Using the upper bound
fx) I and the lower bound on Z:
B (
1 -0 2|)| dx
M(A) <
exp
7
2
gd
d/2
B
2m
<
2x
d/2
C 2X
Cj
d/2
d/2
B
C
B
1
cf
C
2X
Cj
2X/C
Part 1: Proof of the Upper Tail Bound, We seek to bound P([|e] 2 (1 + n)d). The Chernoff bound for upper tail is P(X 2 a) < infxzo e A
27

---

Published as a conference paper at ICLR 2026
First, we bound the MGF (A) B|c^ [cllā] for À 0. As shown above, this yields:
B 21
-4/2
M(A) - Cf C
for <><
Applying the Chernoff bound with (1 + n)dd
P([k 2 (1 + n)d) - inf
exp
(- n)d log(1 - 22/C))
Cj 0<X<0/2
Minimizing the term in the exponent with respect to À yields the optimal value X This choice is valid (i.e., X if 1/C-1
C 1
2(1+n)*
Substituting X back into the exponent gives:
d
C(1 n) n)
(og(C(1+m)))) (C((1
This completes the proof of the upper tail bound.
Part 2: Proof of the Lower Tail Bound. We seek to bound P([||| S (1 17)d). The Chernoff'
bound for a lower tail is (Px S a) & infxso eAaaEee -.
First, we bound the MGF for a negative argument, M = Ele A|cll for A > 0:
d/2
B
21
M(-A) S Cj
1 1+
C
Applying the Chernoff bound with (1 - n)d:
B
-2 log 1+ 21 ) C
d
P( s (1 - 1))d) - inf exp
((1 7)
Cf A30
1- - -c). This choice
Minimizing the term the exponent yields the optimal value X valid (i.e., X 0) n > 1 1 1/C.
Substituting this x back into the exponent gives:
d
d cC(1 - log(C(l n))] --5(0(L n)).
This completes the proof of the lower tail bound.
Part 3: High Probability Argument. We finally derive high probability argument for ||e|[2. Set
2
2B )
T(8) log
h(x) = x 1 loo x,
x >> 0.
From the one-sided bounds,
P(1e|12 2 (1 + n)d) S - exp
B
1. - h + +n))).
Cf
B
P(|e|| < (1 - -)) < exp
hC - n)))
ef
Imposing each tail to be most 5/2 is ensured if
hxx)) 2 T(6) with x == C(1 + 1)) (upper tail),
x= C(1 - 17)) (lower tail).
Using h(x) 2 (x-1)² for all x > 0, it suffices to require
2x
(x - 1)²
2 T(0) - (x - 1)2 2 2T(5)x - x² -21 +1 2 0.
2x
The quadratic has roots
(0) (1 + 7()) (())), ()) (1 (1+)()) v7(0)2+7(()),
28

---

Published as a conference paper at ICLR 2026
with 0 < (0) < (o) (since r(6) 0). Hence x² 2(1 + =()))z 2 0 is equivalent to
x E (-0) (8,,
Applying this to each tail:
Upper tail: with æ C(1 + 11), it suffices that C(1 + 7) 2 yn (0), i.c.
(o(
2
exp
- 1. C
Lower tail: with x C(1 - 17), suffices that C(1 7) S yr i.e.
())
Using a union bound with 8/2 on each side yields the two-sided statement
y (6)
vu(0) d ) 1-
|| S C
equivalently,
- - p
d v || v (1 + p) d.
with
(6)
(o(
2
2B Cj
exp
1
7
exp
C 1,
7(6) d
log
This finishes the proof.
Proof of Corollary A.3. The proof separately bounds the norms from below and the inner products from above.
From the statement in Lemma A.2, for each E (1....,m).
yu (5m222m)
< Yn ya(5/2m)
5
P
d S ||Ei112
d 5
C
C
2n
Let A be the event that 5x(5/2m)
||ea1|2 < yu16/2n) d for all i 1, ,,..,,
n. By union bound over
all n samples, the probability of failure at most n Here introduce another lemma:
8 2n
5. Therefore, IP(AA) 2 1 - 6/2.
Lemma A.7. Suppose satisfies the conditions in A.7. Let E,,Y be independent copics of Then for some universal constant oo which depends on C, C1, C2 Cz. we have
cot
2exp
d
The proof is deferred to Appendix A.4.3.
Let tn / co log(n2/8). Setting t tn makes the tail probability for single pair (f.j) at most Let B be the event that e, ej L, for all * j. By a union bound over all pairs, the T.
probability of failure is at most (z) < 3. Thus, P(B) 1- 8/2.
We now consider the event AnB, which holds with probability P(AnB) P(44) -P 2
1 - 8. On this event, for all i + j:
||e; - e1 = ||e.112 + ||, - -2ETe)
yn (6/2n) d (((622n)
2hp
C
C
2 nn( (5(6/2)) C
d -2 d
log(n2/0).
co
Since this holds with probability least
8, the claim follows.
29

---

Published as a conference paper at ICLR 2026
A.4.2 PROOF OF LEMMA A.6
Proof of Lemma A.6. By separating the mean and the random part of the original data zp, we have
(2) Poje (xx)) S Do exp( 1
203 || 2 (1)) (20)dro
Texp(= 2 11 a. (1) (k))))))))))))))))))))))) (2) (ro)
S(e + (a(a)) expl 20 a - ae + (1))) (1)p (e)de
expl 207 a| arle + (1
Plugging in the expression of Pe. we have
207 || arlc + n [1) pc(c)
exp
C マ |lel1 2 + log S(e)
10 - ar(e + n (1
11
exp
2
C
|10t
0 - 20x (k) To E aQlld11)
211e ||e1 + log f(()
exp
?
exp
1
+C)
+
||<| of
(x)
a))
1
12, app (1) log f(c)
2
of
20
exp
||e
mell
l l1
||
(1) log f(t)
20
exp(C(tt
24)) cxp
||c (-7
72 2
Fel12] f(c),
where
72 = ai
+ +C,
of
Te = 2 2
0t
(ot
(1))
o 11
Ctt, := 2
1|at - (1)
20
By substituting the simplified kernel back into the expression for Poje (xxe, the constant term exp(C(tt,x)) cancels from the numerator and denominator, yielding:
(M)
(k) Hoje (a:1) = [(e + (k))
(()) exp
- 22 ||e 7..11) f(e)de
Sexp - |le 1 f(e)de
22
I [cexp
(- 3 ||e 1.a133 f(c)de
= (k) +
ʃexp
(-
2 ||e - mill33 f(e)de
This expression is the expectation with respect to a new posterior distribution, whose unnor- malized density is given by q(5|2,, k) & exp (- m1 f(e).We provide a more rigorous justification for the approximation, starting from the exact expression for the posterior mean:
(wtt Poj (xxa) = (x()) Eewg(e] (*)) + er Ee-gle mc]-
Our goal is to analyze the term Eeogle - -.e].. Writing it as a ratio of integrals:
[(c The) exp (- 12 ||e - f(e)de 2
Ee-gle- - [ir]
exp (- 3 lle m.113) f(e)de
30

---

Published as a conference paper at ICLR 2026
Let (c), = exp - T2.13) denote the unnormalized Gaussian density, We apply mul- tivariate integration by parts to the numerator, which yields the exact identity:
J (e- (e)
7t [
(
Substituting this into our expression, and letting Z be random variable with density proportional to the Gaussian part, i..., (iee (1/) -111-11 -Ia), we obtain the exact relation:
EylVf(()]
Eewgle [ell 7i [r(s(Y)]
and we further have
Bvd 78
[1E 1elll2 1
By the condition 0z/0, = ((vd). we finally have
Moz (1re) (M)
of + Cop Ee -
Gaussian Posterior Mean
and we complete the proof.
A.4.3 PROOF OF LEMMA A.7
Proof. First, we rewrite the inner product bilinear form in terms of the independent vectors & and Ej. which are entrywise independent sub-Gaussian random vectors with zero mean and unit variance as stated in Assumption 4.2:
eT - (T2))))))))))))) = E2,
The expression E, Σ is bilinear form with deterministic matrix Σ and independent sub-gaussian vectors Ey. We can now directly apply the Hanson- Hanson-Wright inequality (see Vershynin (2018) Theorem 6.2.2), which states that for any fixed matrix A:
CHAL CFTAAT } )
P(ITT AE,1 2 t) s 2xp exp (-co min
for some constant Co > 0. By setting = E in the inequality and invoking our condition ||E]]v < C1 ||=||= < C.vd, | C'., we immediately arrive at the final bound:
{- cot2
P([] e 2 2xxp
d
where co > 0 is a constant depending on C, Co, C1, C2.
B REPRESENTING EMPIRICAL AND GROUND-TRUTH SCORE FUNCTION USING DEEP NEURAL NETWORKS
We follow the idea of network approximation in Fu al. (2024) to build our proof.
We express the empirical score function as
log (p((
(x)
similarly for the ground-truth score function, and approximate the numerator (x) and de- nominator P. (x) separately. To ensure uniform approximation, we restrict the domain of x to a bounded set. In addition, impose lower threshold flow on (x)) to prevent instability caused
31

---

Published as a conference paper at ICLR 2026
by extremely small density values. Finally, within the overlapping regions of these two truncated domains, we employ ReLU networks for approximation.
We organize this section follows. Appendix B.1 presents the main lemmas and propositions that form the foundation for the proof of Theorem 5. and uses these results to give a complete proof of Theorem 5.1. Appendix B.2 provides the proof of Proposition B.4, which establishes the network approximation of both the numerator (pr(x) and the denominator Pr (x)). Appendix B.3 collects the proofs of the auxiliary lemmas used throughout this section. Finally, Appendix B.4 details the network architecture and analyzes the error propagation of the score approximation network.
B. PROOF OF THEOREM 5.1
We begin by stating the main lemmas and propositions needed for the proof.
We first establish that the for -orm of the empirical score function can be bounded in terms of the b -orm of x. We denote BD maXigign [ llee.
Lemma B.1. The empirical score function satisfies
| + BD
I log (=)))l I
o7
The proof is provided in Appendix B.3.1. This lemma shows that the -orm of the score function is controlled by both the input magnitude and the magnitude of the dataset.
Next, we establish some results on complement of the bounded domain x.
Lemma B.2. Suppose B > max(2Bn, 2vd). For fixed time i E [0, , holds that
J |<||>>> Vlog (p) & of
Bd Bexxp exp (- B²
(- 8
J J (x) S of exp
B
Bd-2
The proof is given in Appendix B.3.2. Lemma B.2 follows from the light-tailed nature of the empir- ical distribution, which ensures exponential decay outside the bounded domain.
In similar fashion, we show that analogous bounds hold when the empirical density Pr truncated by threshold.
Lemma B.3. For any B > 2Bp and flow > 0, we have
J [1]e(()) < flow) Pe(x)) de S Bd flows
(B.1)
1 J
(1)((()) < (low lo dx S Cow Bd42 |B
(B.2)
of
The proof is provided in Appendix B.3.3.
By combining Lemmas B.2 and B.3, complete the truncation step. We introduce network approximation result in Proposition B.4.
Proposition B.4. Suppose that the density function of Paata satisfies the sub-Gaussian Hölder density condition in Definition 3.2. For any sufficiently small > 0. Define the early- stopping time to satisfying log tu Ollog c) and the terminal time T = Ologee")) We con-
I Then there exist ReLU neural network architectures strain x E [-2v 2og T.2 2.222
Fi (W,, L1, Ni). such that 35 E Fi(W, L1, N1) satisfying for all E [to, 7]
p. V log (z) 3(z, t) Z of
The configuration of Fi is
I. = Olog -1),
W O(n log -t) N Omllog'
32

---

Published as a conference paper at ICLR 2026
The proof is provided in Appendix B.2.
Now we start to prove the approximation bound for empirical distribution. We claim 3(x, t) is (Pi) approximator of the score fucntion. In order to prove it, we choose B 2 log - and Gow 4c. We decompose the score approxiamtion error into three parts
J [( V log (x) (x)) de
[|((x,t) v log Pe (x) (x) dr
|101b>>B
(D,)
[(Dec(x)] < flow [[((x,) ( da
|1 SB
(Dz)
[[br(()) 2 flow || t) ( dx.
(Ds)
We bound three parts separately.
By Proposition B.4, we know || t) S 2//2log--1
Bounding D
of
|5(x () - (ogg:(<))) Pr dx |=|l>>B B
(21 1) + 2 log P. (x) (x) dx
||
2 (log c -yd/22
(B.3)
We invoke Lemma B.2 in the second inequality.
Bounding D2 Similar what we did in bounding Dr. we have
(1p((x)) flow [|{(,,t) t) (z)
|1 (21 1 + lo) (x)) <low (o())) dr
Flow (log c 1d//2+1
(B.4)
We invoke Lemma B.3 in the second inequality.
Bounding D3 By Proposition B.4, we have
(1))(x)) 2 flow 11 log (.(x) de
[xll ao SB
1 (1)) 2 flow }d]lS(r,
V lo (x) da
d
1 [1)p(x)) 2 CLow pz) 2 dx
1B
11 (1p((z)) 2 flow ) delow dr
flow
(log 1loge-1)07
(B.5)
Elowol
33

---

Published as a conference paper at ICLR 2026
Combining (B.3), (B.4) and (B.5) together gives
[(x, 1)
log De (2) dx
(log e -1)// +
/ (loge (1oge
e
E + (loge (loge-1)op)
ot of
(log c 11dd/2+1
(B.6)
here we plug in flow 4c.
Set = Coe(loge 1))/2211 where C, represents the constant hidden in S in (B.6). Also, when
goes zero, will go zero. Then immediately derive
J/l 15(2,, V log (l do S
e'
it implies
E ED
E [1(,, V (x)
-
ot
The network configuration of the entire network architecture satisfies
W = Ö(n log (E)) = (=)(
N N Ö log log"(e)) (t)
For the approximation of ground-trune toout score function, apply the Theorem 3.4 in Fu et al. (2024)
with dy = 0.
Theorem B.5. (Theorem 3.4 in Fu et al. (2024)) Suppose Panta has a sub-Gaussian Hölder density with Hölder index 8. For sufficiently large Ni and constants Co Co by taking the early- stopping time to N- and the terminal time T C. log Ni, there exists
E F (W,
such that for any E [tu, T), holds that
f |1((2, 1) V llogr( (og 0
N₁ 20 (log N1) 8+1 N1)
(B.7)
The hyperparameters in the ReLU neural network class F satisfy
W O (Ni Nog L O (log^N1).
N N 0 (og)
(B.8)
29 We étrue C. N, (log Ni) 8+1, where C² denote the constant hidden by O when N is sufficiently large, Ctrue will be sufficiently small. Then we immediately have
J [1((e,0) (log dx 5
Ctrue
of
Namely
ED E ExmP, [lls(Xe,() V (log
<
étrue of
The network configuration is
W2 = δ ((erue)
1 log" cure)
L2 = E( sirde),
N₂ = Ō ((Erne)) - log 1 E
We complete our proof.
34

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF PROPOSITION B.4
We denote the first coordinate of a vector x E N2 as [x]1- Without loss of generality, we focus on the j-th coordinate of the empirical score function. The explicit form of is
Dr
(4y3s - x)
exp
20
[Vog
exp
x
207
D4
We approximate the denominator Da and numerator D3 with ReLU networks, and subsequently combine these approximations to construct score estimator.
Lemma B.6. (ReLU approximation of D4) For any sufficiently small en 0, there exists ReLU network architecture F(W, L, N), such that 3fRettU (2, E F satisfying
Σ 1=1
1 exp
1 2
x [ - fRRLU (xx t) S Ete1
(B.9)
for any x E 2 log 7, v 2 log 7i and t E [to, 77, where log to Oloo en ) and
T = Olog f. and the network configuration is
L = Olog 7 W = OOn 7 N = On 77
The proof is provided in Appendix B.3.4. We also have the following result approximate D5-
Lemma B.7. (ReLU approximation of Ds) For any sufficiently small ffz > 0, and j E [d]. there
exists a ReLU network architecture FW L,, N), such that Aff (xr, (, E F, satisfying
a xly
exp
| amx:13
-f PRoLU (x,t,5) S Ef2
(B.10)
n
O2
20
for any x E
-2 2 2 log f, 2 2 7/
and E [to, 27, where logto O E/2), and
= Olog -1
f/2 ), and the network configuration is
L = Olog* 7zt), = O' 77t,, N O( 77)
The proof is provided in Appendix B.3.5. Now we are ready to finish the proof.
Proof. Let clow = de, and set fʃi = 6/2 = Then when (x) > flow we have fReLU (z,t) > (z). Using Lemmas B.6 and B.7, we denote the clipped version of fi by fi,elip max(ffRot (RoL.U, (ow), and for j E [d], define the score approximator as
Ia(x,1,5) = min f (R, t.j) 2 2loge-T + BD o.si,clip (x,,
of
By the definition of fa(a, t.j), we know (fs(a, t,j) S 2 2llg this actually matches the upper bound of V log P. (a) when | I B. Next, we bound the difference between [V log and fz
of
RReLU (xc, 1,j)
[Vlog fa ((,, <
V log (x)))
o, fi (x, t)
35

---

Published as a conference paper at ICLR 2026
[D(())>s
[Vic(t)]s
[Vpi(>)
ال (a, orJi,clip 1)
+
(c( (x,
[1,clip((r,11)
1
(x)(
(x) (x,
0x V
0, (x)
(z,,
From [Vlog & c/2loge-1+BB BB we derive [Vi(()) S B+Bo pr, for Ps 2 Clos we have of
V (e)) - fs(x,,t,))
2 2 logg el I BDD
1
oz VD(()>)) 1. (x,
Pt
+
of
(ar
S1,tiip
Ithi.dip
1
(2 (2g2loge -1 Bp) (x) heeppp
[ie f fRoLU (x. t,5i
/1,cltp
o?
ar
< 2V22ogg-14 Fio7
Then we can obtain mapping fa t) approximate V Voog (x
[olg fs(x, S 2v2 22loeee
p:o7
Here fz(2, t) is defined as
fa [/a(1, 1, (,) (,))
We now construct a ReLU network fftcLU (z, t) to approximate fo t), namely
[f((,, P (z. 1) S.
Given ReLU realizations fi and f2. we build upon them by implementing the following basic oper- ations via ReLU networks: the inverse function, the product function, a ReLU-based approximation of os, and entrywise min max operators. Details on determining the network size and analyzing error propagation are deferred to the Appendix B.4. Once construct f ffe..U (x, t). we have
(c( V fo (x) - RRRLUU (xr, 1)loo of
where EfteLU (I,t) E F/s, the network configuration of Fjs satisfies - E
L= Olog²
W = On On -1), = O(nlog' E
We complete our proof.
B.3 PROOF OF LEMMAS
B.3.1 PROOF LEMMA B.1
Proof.
Em || -aatt loo exp
I
||
V
logg (x))l(
o?
En exp
1
||
ax2;113)
Σn (1122|100 + (10y)=il>>)))))))))))))))))))))) exp
||
0
Σ exp
1
20
||:1|00 BD
of
36

---

Published as a conference paper at ICLR 2026
B.3.2
PROOF OF LEMMA B.2
Proof. We first prove the inequality for the score function.
/ |1 BB
V| log (x( 1P (x)dx
1
/
(-11z -|1-~000 axzill3
V log (x)]l
20 dr.
n o2
>
We only need to bound this term
(- (-lle - af0il1
V log (x))leexp
dr.
(27))//12 1a
20?
By applying Lemma B.I, we have
1 of |=
| a,3i
V po( exp
dx
(2m)
20
d
||
(1/0/100 + BB) exp
da
4+4 (2T)d/2
11=1100 >B
207
d
|
03,23111
1 (11//2 + BD)3 exp
dx
0, (2T)///
20
d
115
(1los5i + actell2 + Bb))² exp
de:
(2m)//22 d
Loisitanmlla22 112B
2
|15:12
(orrsil222Bn)) exp
dec
(2m)d//
(Bella>(B-B))) BB)/
2
d
/ (B-B/////
/ (orr + 2BB)) exp
p2 2
pd
drd
(B.11)
(2m)4/12
The third inequality follows from the change of variable E; = z-a from changing variables to spherical coordinates. Next, we consider give a upper bound for (B.11), we derive by firstly substituting with m r². then (B.11) becomes
The last equality follows
ai
(n-n-B)//n / (0or + 2Bp)² exp (-2)
n2 pd drdw
(B.12)
0. (22)
(2(2////2 (B-BBB))) / (ofm + Bo,Bpm + 4BB) exp
d
(-2
m1 m *
dmdw. (B.13)
We bound this integral using Theorem 1.1 and Proposition 2.6 in (Pinelis, 2020).
Lemma B.8. Let Ga(x) be defined as
x 2e
if -1,
(x. ba) )a
Ga(z)=
c
if E (-1, \ (0).
aba
log stl
a
where
T(a+1)2/e-1)
if (-1, \ (1),
b.
af
and 7 the Euler constant.
Then, for -1 S S 1, it holds that
8 8- a S Ga(x).
37

---

Published as a conference paper at ICLR 2026
Moreover, for any real L, we have
[ l ea-le- < e
for all real x > a 1.
By applying Lemma B.8, we obtain the following estimates. When 0, one has
fe t e dt < Go s
> 0,
(B.14)
since log L+ta < L. For a E (-1,1]} {0), it holds that
ta-e s Ga(x) S 0-0
(B.15)
Furthermore, for >> and >> 1, we have
tu-Le dt xaa
(B.16)
Combining (B.14) .B.15), (B.16) and (B.13) together, we can conclude, when B
2vdd),
1
VlogP(())) exp (- a
12
(2T)0/22
|
207
4-2
d
(oêm + 40,Boym 4B3) exp
dmdu
(2(m////2
JM 1MS(A-BB))//d Bo)
2
m d-2
1
(0,t+ 40, 4BB) exp
m
dmdw
2
2
1
fo,+44,,By/nn 4B3) exp
(-
m
m
2
dmdw
m2
2
B2
Bd exp
8
Then conclude
/ JalB>>B
|
(
1
(-15)
B2
S
Bd exp
8
Similarly we have
(r())dr
|+|| B
(-
|x
12
cxp
dx
12 0z
d+1
||
20
B²
Bd- exp
8
B.3.3 PROOF OF LEMMA B.3
Proof. For the first inequality, we have
f
flow Pc(x) dr
38

---

Published as a conference paper at ICLR 2026
11 flow dr
SBu 6low-
For the second inequality, by Lemma B.I, we have
/
11 < flow H| V log Pr (2)) dæ
|1 SB
flow (1kz)/o + Bp)² de
0 |T
2 low Bd42.
ot
B.3.4 PROOF OF LEMMA B.6
Proof. For any > let U₂ be the set satisfies
(r - axt;))
Uz i [n]]
S /2lge
ot
2
immediately gives us
71
exp
207
| 0r3,
Σ n
exp
(227
20
1
||
iEU.
Σ
exp
||
0 1
2
<
-6
Se.
(B.17)
Then, we approximate exp loge-!. ! By Taylor expansions, we have
(-
207
1 |z azz [b]
for E Uz. We already have
209
1
1|
ax2;11
k
||- a
Σ k<p
(
1
1I axx:1
log" c-
exp
20
k!
20
p!
where we use the fact le- - E
x² pf
when x > 0. Let p = [3u Sogl 11,, where
satisfies 3u log u 1, and invoking the equality p!
(2), it yields
exp
1 x ax3:13] 20?
- -E
20? ||
"
log" e-
S u -3 loge E.
kd
p!
k<P
(B.18)
By (B.17) and (B.18), we have
m
exp
1
- Σ
Σ
|
112
71
20?
20
EEU.
k<p
exp
1
|
a,,r;13
Σ
exp
1
||
a,
72
207
n
20
iEU-
exp
20
7
||
Σ
k!
20
||
2,
12
KEU
k<p
39

---

Published as a conference paper at ICLR 2026
2ec.
(B.19)
We set B 2 log 6 for convenience. We denote Et<p il
(-
207 || amall1) as Sp. (z, 1),
and p,,s( (2, (2, 1) (iEU.). for any i Uz. approximate the Taylor expansion using ReLU network.
Lemma B.9 (Concatenation, Remark 13 of (Nakada & Imaizumi, 2020)). For a series of ReLU R + R with fi E F(W,, L., N,)
networks fi Rd + R R Rdd,.... ..... . fk.
(i = 1,2, 1,2,,...,,..,,... k), there exists neural network f FW, L, N) satisfying
[(x) = SN Sk-1 0..0 f1(),
Vo R
with
L Σ Li W < 22 W. N N 2 No.
Lemma B.10 (Identity function, Lemma F.2 of (Fu et al., 2024)). Given d E N and L 2 2, there exists fla F(2d, L, 2dL) that realizes an L-layer d-dimensional identity map
(a(x) xx, ER.
Lemma B.11 (Parallelization and Summation, Lemma E3 of (Oko et al., 2023)), For any neural networks fi, f2..... f& with fi Rd: + R and fi E F(W,, L4 N:) (i 1,2,.... ...., k), there exists a neural network f E F(W, L, N) satisfying
f(z) = [fi (z1) f2(22) fa(x))
tdd+ - R +
for x = (x] 2. ..... x. td+ (here x; can be shared), with
1= max Ti, 1sisk
W W W.
N 2 (N: Idy).
Moreover, for σ1 = I2
= I E Rd and d¹ = d2 = =... dr. = d', there exists
Saum (1:) E F(W, L, N) that expresses Saum (x) = = (i(x), with
L = max Lú + 13iSk
W < 4
W 1=1
N 4
(N: Id)) + 2W.
(F.3
1=1
Lemma B.12 (Entry-wise Minimum and Maximum, Lemma F.4 of Fu et al. (2024)). For any two neural networks J1, f2 with Sr Rd Rd", Ji E F(W,, N,) (i 1,2) and In 2 L2, there exists neural network f F(W, L., N) satisfying
f(x) min(fi f2), (or (xx) for all æ E
with
1- W 2(W1 + W2), N 2(N1 N2) 2(L - L2)d'
Lemma B.13 (Approximating the product, Lemma F6 of (Oko al., 2023)). Let d > 2, C 2 1. For any éproduct O, there exists fmult (æ1, D2, .2, E F(W, L.N) with
L O (log d(log< c-1 --1 product
dlog C)),
W 48d,
N O(dlog -1 -
dlog C)2
product
such that
d II
fmule (a', ..)..)),))))
-
x2 i=1
I Eproduct + dd E1.
(B.20)
< e1. Moreover, [fmmtt( S Cdd for x E Rd and mutte (2 ,., = 0 if at least one of x 0.
.... for x E c--C, Cjd and x' Rdd with I|x - x' xo
We note that if d = 2 and J; 22 x. it approximates the square of We denote the network by fsquare(z) and the corresponding Eproduct by Esquare- Moreover, for any x E Rd and n E Nd, we denote the approximation of n. [_ xi by Spoly. (x) and the corresponding emro by 6poly-
40

---

Published as a conference paper at ICLR 2026
Lemma B.14 (Lemma F.7 of (Oko et al., 2023)). For any 0 Cinv F(W,L,N) with
there exists f-1
== O(log= m
W = Ollgg" m
N = Ollog" fm
such that
f-1(z²) S Cinv
1
- x
for all [Einv midl and x' R.
(B.21)
e
Lemma B.15 (Lemma F.8 of (Fu et al., 2024)), For En E (0, 1), there exists fo E EJ(W, L, N) with W = =Olog
= O (
N = O(log (al) ea),
such that
(aa(t) arl S Ca:
for all t 1 0.
(B.22)
We can readily extend the approximation of a, to at ee= by doubling the coefficients in the first linear layer.
Lemma B.16 (Lemma F.10 of (Fu et al., 2024)). For E E (0, 1), there exists fo E F(W, with
L Olog² t W OO
N N E
such that
(t) or <o. co:
for all 12 126.. 6g.
(B.23)
Lemma B.17. For any Cot E (0, 1), there exists for E F(W, L,N) such that
(t) dl S Ea',
for all t 2 Ed',
with network parameters satisfying
= =Ollog Eg!),
W = Ollog* Ec), N = Olog Ez
Proof. We define the network by composition
for(t) = f-i(fo(t),
where f-a approximates the reciprocal funetion (Lemma B.14) and fo approximates d; VI e (Lemma B.16).
By Lemma B.14, the approximation error of f-1 satisfies
(tor
1
S
Cinv + Eo Einv
o,
Now we set
<in min
O(Err),
Ea Cinv
VI - '-.
With this choice, the total error is bounded by car for all 2 Cor. Finally, according Lemma B.9, we can verify the network parameters F(W, satisfy
= O c,!),, W W = Olog G,), = Ollog* co!))
Lemma B.18 (ReLU approximation of the interval indicator). Fix B 0 and a margin parameter T(8) (0, 1]. Let (u) max 0, u) and define the "unit "unit-ramp"
(=(()(u)) =
u
u
-1)
E [0. 1].
T(8)
T(6)
Consider
fB. r ( (6)(z + B) r, (0) B),
x E
Then fB. R + [0, 1] is realized by a two-layer ReLU network with width 4, and it satisfies
0.
|o|| 2 B + T(5),
(z)
1,
[x| S B,
linear in s,
x E [-B B T()), B B,
Moreover, fB, E F(WWW N) with
L=
W 4, =
41

---

Published as a conference paper at ICLR 2026
Proof. Since r- (u) requires two ReLUs, the entire construction uses four ReLU units in parallel in a single hidden layer, followed by a linear output combination. This corresponds to a two-layer ReLU network (one hidden nonlinear layer plus the output layer) with width W 4. Because all nonlinearities appear in one hidden layer, we have N = 1. Thus the stated bounds hold.
IT
With these lemmas established, we are ready to approximate the Taylor series using ReLU network. By Lemmas B.9, B.10, 1.1 B.15, and B.17, we define the network as
(1, 1) Smutt (SammAcsp
(111/2)* h!
J (((((x)
where
d funll for, ((([)))) - fflell))-f(((aaa (a ((,,)) fa(t)(a)l)) (k 2 1)
(i t) = Σ
1=1
Spoty.0 =,
Sindicator (2, 1) V2luge- ((()) (1))) (z,
We further define
(-1/2)*
Fna(E, 8) &su
&ooty, 1))
k!
We first compute the approximation error between fr.( t) and fna(x, t), which is
Cpoly.k
Cp. 1
CCpoly.k
2
k<P
where
Epoly,k fproduct,k;, Ck,15k,1
6k11 d(e k 40 k,25k,2)
[Vdl++ Bp)
2(k-1)
1 Vd BD)
C k
C max
Fx.2 max(BDea, Cor).
Tto
0%
We set e* foxp and take
(product,k; 2
Aproduct,A;2 =
4dC,,1
6a = 40 k.2 B
Ck.d
Then, by the definition of éproduct,1, verify cp eexp. We decompose the total emror into three parts
[xp.(2, t) hpp((x,))
< Mp.(2, 1) Pp,la 1) x findicator (x, t)
Do,1
+ (a, t) x fundicator (x, t) - (2, 1) x findicator (z, t)
Da.2
+ (p,a(a,)) t) x fundicator (x, t) fpi(z, x ((ev.)) |.
D6.3
The first part arises from multiplying two networks. The second part comes from the approxima- tion error of the Taylor expansion fp.((x,)) t). The third part due to the approximation error of the indicator function 1l (iEU.). We now bound these three contributions separately. For D6.1. by Lemma B. 3, it implies
(p - -p,s(2, x Aindicator (z. 1) S product,.-
(B.24)
For Do.2
[ra(x,)) t) x Fidicattor Spa(x1, t) x Findicator (2, t)) 1Pra(1,)) Spo(dr,)] Cp.i S Cexp.
(B.25)
42

---

Published as a conference paper at ICLR 2026
For D6,3, when --مم || E [0, V 2 log - U [v logg el + +(6),, ool, findicator (2. 1 (iEts))
then
(f, 1) Sindicator (2, 1) [p,( ((Eu,) 0. (V2loge e +(6))
When
p
Una(1, 1) Sindicator (ze, 1) - Sp,(1,1) (iEU,)
(2,
(loge
+ 2T2 Loogee + T(
p!
1+ T(6)2
2 T(6)
log )
exp
3u log
log
(loge-1)
Vloge -
T(d)
exp
3u loge
2 2og
1+
-log
Vlog c-1
3u (ō) Vlovolo-- log log )))
exp
(B.26)
Set r8) 6u /uvloge-1" then from (B.26),
conclude
(x, t) x findicator (æ, je,t t) fp,s(x, t) x 1 (ieti.) S cc.
(B.27)
Combining (B.24), (B.25), and (B.27) together gives
(1, t) ( t) Eproduct,3 + Cexp + CC.
(B.28)
We choose Exxp Epruduct,3 and define fReLUU
ReLU Smult (1/n, Ssum, (i
Consequently, from (9) and (B.28), we have
20 || a.2:113) - FReLU (ar, S (e + 4)c Cproduct,f
1
exp
n
We choose fproduct.f
=, by Lemmas B.9, B.11, B.13, B.15, B.17, we have
20 | - a,2:113) pRoLU (x, t) s ()+
exp
The network size parameters of (ReLU(r, 1) satisfy
L = Ōloga c
W = log e
N = Ö(nlog* é -
Substituting with 5/ immediately give us (B.9), and proof is complete. 645
B.3.5 PROOF OF LEMMA B.7
Proof. This lemma serves the counterpart of Lemma B.6. The proof follows similar struc- ture, and is same for every entry E [d]. with the only difference lying in the construction Un- Therefore, will focus on elaborating this part. Let U/ be the set satisfies
(or o43;)
Uh
i E [n]
< 2 22
immediately gives us
[or0;; z1,
aa - x),
|| 2x
20 aa - x113)
exp
exp
Je
207
43

---

Published as a conference paper at ICLR 2026
exp
(-27 ag
2 1
11
0x
loge-I62
Sc.
The last inequality holds because is sufficiently small, ensuring that 2 lloge -l S 1 (in fact, this condition is satisfied whenever S )
Then, we construct the network approximation in similar manner. First, for each < n, approximate the exponential function exp
2,7 | a, 12 and the term [m-aşal separately
using ReL.U networks. Next, we combine these components using Lemma B.13. We then sum the =
resulting functions and multiply by 1. applying Lemmas B. and B.13 needed. Finally, obtain the network configuration, completing the proof.
B.4 CONSTRUCTION OF fz(2,t)
We denote the entry-wise maximum function Lemma B.12 as fmax, and entry-wise minimum function in Lemma B.12 as fmin- By Lemmas B.9, B.11, B.12, B.13, B.14, and B.17.
We define
f3
PReLU
xx, t,j)
2 2yggl loge-I+ + BD
= -Suin
[fmute(for, Fa
#ReLU (x,
f (fReLU t), (ow)),
fg.
We have
f RRoLU
pR (z,t,j)
< max (Caulte,3 + 307,,6r,, Sproduct.fs + 333 + cor))
oefi,olip
where
BD, 1/2 ) CF max
1
1
2
2 log e -1 + BD
C5.1 = max
2 2 loge
flow
Tso
We choose
mmult,3 fproduct,fs
2
6C
iinv = Ex'
12C
Then we can conclude
f2
/ReLU
s ReL.U (x, 1,1)
Se.
ot
Using Lemma B.11, we can construct
PReLU
(x,
[ ReRL
(x, 1, 1), ReLU 1, ( ReLLU (x, (x,
such that
RuLL (x, 1) fa 11 Sc.
The hyperparameters (L. W, N) of the entire network satisfy
L O=o(l =)),
W = On 1) N = O -
C STATEMENT AND PROOF OF LEMMA C.1
In the following lemma, we investigate the Lipschitz continuity of score functions by computing the Hessian matrix of log density.
44

---

Published as a conference paper at ICLR 2026
Lemma C.I. The Hessian of log (z(z) admits the following explicit form:
+ of 2 log = of of
I
CoviXo|Xx =x)) xz).
(C.1)
where the covariance is taken with respect to the posterior distribution of Xo given X,. Define the Lipschitz constant of the empirical score function log (xe) as
C
sup
V (z
Assume that and the minimum pairwise distance between data points satisfies
20. az
min | - x,1l2 2 *
log
Under this assumption, the Lipschitz constant C2 satisfies the bounds
of min
at
+
|| =5y11 < C. < 2
max
||: - x5113
(C.2)
of 16ot (sAAJ([m)
40, s/i.EE|n
When is small,
conclude C, Ma min ||., æ 12).
The proof is provided Appendix C. Lemma C.I provides characterization of the Lipschitz con- stant of the score function. In particular, via (C.1), the posterior covariance Cov controls the smoothness the score function.
X, = æ.)
For the empirical score V log (Ite, the covariance term is replaced by empirical covariance computed from the sample. This empirical covariance varies significantly across I and depends on the sample configuration, especially the pairwise distances between data points. As shown in Lemma C.I, under a separation condition on the data, the Lipschitz constant the empirical score satisfies (C.2). This bound shows that C, can grow sharply when there widely separated clusters - (min,.eem) || - æyارı large), especially at small noise levels at, where the term strongly amplifies these effects.
Proof. We first write the explicit form of the Hessian of log pe
2 log pel (xx)
+ oo f(ae - (21 - )nto)
exp
||a, a.alll?
Pdata (x)) dæo
207
of
ʃ /exp
a1+ 10.-0, ll
Pdata (xo) dæo
20
(e(=)((
(J exp
|| 200001
Pdata (zo) dro
207
where we define
e(xx)) = / ((cr (. exp exp
|x, 04
Paata (2r0) dxo-
20
Notice that density function of the posterior distribution of X, given X, is
exp
|| a, 2d
Pdata (1o)
p(zoloz) = f exp
|| Ce 18 20
( dzo
Using this posterior, the Hessian simplifies to
1 of
Cov oxXo xz).
to
45

---

Published as a conference paper at ICLR 2026
where the covariance is taken with respect to p(zulzz). Since X, is constant given 2,. this further reduces to
1 of oo
V log log
CovlXolXx =x]]
(C.3)
which is the form in (C.1).
To derive the upper bound for the Lipschitz constant of the empirical score function, we first obtain the expression for 4 log Po (x:) in similar manner, using equation (C.3).
v (ogg((x))
Cov[X,]X, = xz]]
- of o
where Xz|X, X2 denotes the posterior distribution of X, given X,. For any E Rd satisfying ||u||2 = 1,
a di ot
T V logFe(
Var(u X,|x; zz).
To bound the variance term on the right-hand side, we introduce the following lemma.
Lemma C.2 (Variance bound on a bounded interval). Let X be a real random variable supported on [a, 6] (i.e., a S X S 6 almost surely), and set L = b - a. Then
L
Var(X) <
4
Proof. Fix [[[]]. Since X [a, b] a.s. and m [a, b]. we have the pointwise bound
(X - m) s maxf(o - m)² (b m) The function m ++ max{(a m)2 (b } on [a, is minimized at a+h value is b-a Hence, for the actual m = [[ E [a, 6).
and its minimum
2
b
(X (x-E(XJ <
a.s.
Taking expectations yields
(b a)²
Var(X) = (X =
By Lemma C.2, conclude that
1
of (max; u² uTx; x; min; 2 xx,))²
w 2 log Pe (x.)ul
+
40²
a max || xoll2
+
40
By definition of Ct, we have C sup the upper bound for Cr.
|u||==1 Iu' 2 log Pe (1)))u), and then immediately derive
1 of max
1 -
C. S of
40
To establish the lower bound, we begin by expressing 4
4 log
more explicit form.
2 log (1te)
1 Ee (2: (n. (nT;)) exp
Laz a.s. 1
I
σ7
20
o?
Em exp
|| |
20r
46

---

Published as a conference paper at ICLR 2026
Ei=i(a (It - 2122, exp
20²
Ei= (0t
axti)'
exp
I -aa-a 20z
E exp
| axi cx 11
207
1 Ei
Denote ((xz)
Eim exp
1=2-012,13
2
wx(1))
Eis exp
exp
1
20
we can
201
201
rewrite V 1o as
1
V log
+
Σ
(In (x) ax2i)) "w((zs) ((=)))(z1)))
of
ot
i=1
1
1
Σ
(Ia
af (xxx)))(xxx out; p(xxa)) ro (0:(02))
of
ot
li=l
For any u E Rd satisfying
||ul|2 we have
2 log (z - oo
1
+
1 of
Σ
wx) ((x, - ayz; azz; u(xx)) u))))
We choose (i,j) such that || |z;--;;1 x,yl min we have
|| 11,1, 1l2. At the midpoint x, (x, x33) 1/2,
wx(xx)) (x((1))=
_ 2n 21 112 [[(a -1,,(;(;;,/)223 22112113)
2 + Σ Engtih exp
20
We introduce two lemmas to bound the difference || - x2 112 - 11(x; - x,)/21 in terms of the minimum pairwise distance be]] ||
Tb 122-
Lemma C.3. Let a, b, e Rd. set the midpoint 2+b and 2
2 b||a - 6-12. Then
|| - ml12 = 2 (|lt a + -11) ,2.
Proof. Observe that / - m ((1 a) + (1 b)). hence
4|lt ml12 = ||(t a) + (t - b)]l3 = || a || - 1 + 2(t
t-b). b).
Also,
||(t a) (t- - b)ll = ||a - b12 ||t all2 + ||| b1l 2(t - t- - b).
so
2(t - a, t b) ||e
all + ||e
b11 ||a 6112.
Substitute into the first display:
4|Ll ml 1| 2 2 al12 ||l - b1 ||a b1/2.
Divide by 4 and note 2 ||a b to obtain
||| - m 1| = 1/2 (1le al3 + ||t - -b11)) ,
Lemma C.4. Let
A Amin min || rolle- azb
Then we have
| 2nl12 - [1(x;; - 2;1/2112 >
min
h h
2
where 2.t2, (i, satisfies || xzحl Amins
47

---

Published as a conference paper at ICLR 2026
Proof. By Lemma C.3
A min
| - sallé [1(2; - 2/)/2112 = 2 (112% 22112
=2113)
A min
min
2
2
min
2
By Lemma C.4, we obtain 1| In 12
|1
(2,
x,)/2113
min 1|2, 21112- Since
min,, (m) azth I| - zolle 2 at V log m= then we have (x1) = wy(21) 2 1. Let =
20,
2: 21-2,
m: 2.
2
2 (o
Σ W/ (24) (10) - 043; -
- -u(xa)) u)²
o
oi
(aar(0) ((ar(0;-2;)) x;)/2 2 + u(xx)) u) + ((ar(z;- - plxx)) 2)²)
of
40
(u(2::) u)² + (1)/((===))/ x;)/2) u)²) 2)
(u(
of
40
02
2
||2;
22y12
01
160
af
min
||.; - xz 12
df
16
Therefore conclude
1
of
2 loggi((a))) -
min ||,, xz 112 1,
0
160 (F..JE(M)
which immediately implies
a
2 log 2
+
min
||x;- - 22
og 16o (saJelM
and it implies the lower bound for C.
af
C o
min 16 (fj.JE(M
|| 2z
Moreover, when / is small, we can conclude C, = M(a minizy |1,0 - xz 113).
D EXPERIMENTAL DETAILS
D.1 COMPUTING THE IMPORTANCE SCORE
To formalize the computation of importance scores, we follow the masking-based framework of (Liang et al., 2021). In each Transformer layer of the diffusion model, we associate binary mask variable E {0, 1} with every attention head h. Setting & keeps the head active, while E = prunes it away. Let C(x, f; M)) denote the training loss of the model M on input æ at diffu- sion step The sensitivity of L with respect to &n quantifies how important head h is the model's predictions. We thus define the importance score of as the expected gradient magnitude of L with respect En, averaged over data and timesteps, and layerwise la normalized:
EywD, L-T
(C(mttM)
(b)
D6
E [0, 1].
Elayer(h) (EanD, EMeleyer(h)) eNT
ac(m.: 1)
DE
48

---

Published as a conference paper at ICLR 2026
Algorithm 2 IMPORTTANCESCOREEMM D)
1: Input:
2: Model M with mask variables {&A) } for all heads / E H.
3: Dataset D, Time Sampling Distribution T.
4: Initialize: Accumulated scores ( for all h H.
5: for each batch of data x ~ DD do
6: Sample timestep
T.
7: Compute loss L(x, 1; M)).
8: Backpropagate to obtain all gradients
ac
BE
hEw
9:
Accumulate scores: S(h) ((h) +
Oc 88
for all h E H.
10: for each layer in the model do
Compute layer-wise norm: Ni + V (ENes)(()))
11:
12: for each head h. in layer I do
Normalize score: [(h) 8(h) /N-
13:
Output: Importance scores } JNER-
D.2
COMPUTING PRECISION AND RECALL
We follow the definitions in Kynkäänniemi et al. (2019) to compute precision and recall. Let the real and generated feature sets (obtained from Inception V3 feature embeddings) be
+, {of Ji=
D
{
1
For feature vector and set Φb, define the indicator function
( 1, if a E t such that || - d' l2 S 1 - NN 1)
f(o, 1)
otherwise,
where (Ø' ② denotes the k-th nearest neighbor of d' within 4. Precision and recall are then defined as
Precision( o,, b,))
Σ S(gg, #7): Recall 3 0g
Σ d+ E.
S(Or, $g).
+
Intuitively, each feature set induces a manifold via local k-nearest-neighhor neighborhoods. Preci- sion measures the fraction of generated samples that lie within the manifold of real data, while recall measures the fraction of real samples that lie within the manifold of generated data.
D.3 MODEL CONFIGURATION AND TRAINING
For numerical simulations on Gaussian mixture data, we build standard MLPs and vary their width, depth and diffusion time embedding dimension to achieve different parameter settings. Detailed configurations are presented in Table 3.
For the experiments CIFAR-10 10, we adapt the implementation of DiT (Peebles & Xie, 2023) from https: //github/ com hww//rrhhiicceey.... / /Just-a-DDas chosen subset of CIFAR-10 containing 5,000 images. The model has hidden dimension 384, 12 layers, and 6 heads per layer. We use a learning rate of 2 x 10-4 with a cosine scheduler and train for 100,000 steps without weight decay obtain the original model. After pruning, the model is further trained for 5,000 steps to obtain the results. When sampling, we use a deterministic sampler with 50 steps, classifier free guidance scale 2.0, and randomly generated labels for each sample. Both memorization ratio and FID are evaluated using 50K generated samples.
Our training set is randomly
Additional results including the case with pruning ratio 17 40% are summarized Table 2.
49

---

Published as a conference paper at ICLR 2026
Model
Precision (1)
Recall (t)
Memorization Ratio (%) (J)
FID (L)
Original
0.39
0.08
73.82
15.47
Our Pruning (20%)
0.33
0.1210.01
68.58
15.07
Random Pruming (20%)
0.30
0.09 01
66.87 10.94
17.14
Our Pruning (40%)
0.25
0.08
58. 18.63
16.53
Random Pruning (40%)
0.24:10.2
0.06
55. 55.72.1.99
20.16
Table 2: Additional results including pruning ratio = 40%. We report precision, recall, memoriza- tion ratio, and FID. Each value is shown ata over 5 random seeds.
Original Model: Generated Images
Original Model: Nearest CIFAR-10 Images
CIFRR-1D Images
Figure 4: Left:
Generated images from the same random noise, with the original model (top) and Nearest neighbors of the generated images in the CIFAR-10
our pruned model (bottom). Right: training set. At a comparable level of quality, the pruned model shows greater diversity, while the original model tends to replicate training samples.
Network Configuration
Parameter Count
Width
Depth
Time Embedding Dim
24,128
32
3
32
118, 144
64
64
417.344
128
6
64
2, 2,180,224
256
8
128
5,009,920
384
10
128
12,888,328
512
12
256
44,396,288
1024
14
256
Table 3: Network configurations corresponding to parameter counts.
50

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs as assistants for writing and coding tasks such as formatting results into tables, refining phrasing, polishing standard sections, and assisting with code logging and debugging, but not for generating research ideas, designing experiments, or analyzing raw results; all substantive contributions were carried out by the authors, who take full responsibility for the final content.
51


Reviews:
[{"review_id": "sRAmKpg34w", "reviewer": "Reviewer_M2PN", "summary": "The paper proposes a theoretical approach to understand and mitigate memorization in diffusion models. The paper shows that there is (even with a polynomial number of training samples) a non-negligible loss gap for smaller t, which leads to the empirical loss being minimized, which in turn leads to memorization. As a solution, a pruning-based approach is presented, pruning the attention heads with the lowest importance score.", "questions": "Q1: How does this theoretical approach support the empirical observation that deduplication seems to help with memorization mitigation?  \nQ2: Why is the fine-tuning after pruning the attention heads necessary? And more importantly, how do you make sure that no new samples in the fine-tuning set are memorized?  \nQ3: How are recall and precision calculated in Table 1?  \nQ4: Does this only hold for unconditioned text-to-image diffusion models? Does this also hold for other noise schedulers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a strong theoretical understanding of why memorization happens in diffusion models.\n- The paper shows that even with more training data, the loss gap will be present, showing that adding more training data to prevent memorization is not sufficient.", "weaknesses": "- In the experimental section, it is not mentioned which model architecture was used.\n- The experiments are only conducted on small datasets such as the CIFAR-10 dataset and a synthetic Gaussian mixture dataset\n- The pruning-based method is not evaluated against other SOTA pruning-based methods.\n\nMisc:\n- In line 95 there is a typo in \"correspnding\"\n- Line 103 \"an\" -> \"a\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yeh6SpoBSN", "reviewer": "Reviewer_XbEb", "summary": "This paper develops a theoretical framework for explaining memorization and generalization in diffusion models. First, it shows that the denoising score matching loss with respect to an empirical distribution is not minimized by the ground-truth score function, due to a statistical lower bound.\nSecond, it proves that under the universal approximation perspective, representing the empirical score requires larger networks than the case of ground-truth score.\nThese results establish quantitative separations between the regimes where diffusion models memorize versus generalize.\nThe authors further propose a pruning scheme for mitigating memorization while preserving generation quality.", "questions": "1. What is the central hypothesis that the experiments are designed to test? How are the theoretical results meant to explain the observed effects of sample size, network size, and weight decay on memorization and generalization?\n\n2. Does the theory predict a quantitative regime (e.g., for sample size, $\\log n = O(d)$ or larger) in which generalization can emerge, and are the chosen sample sizes in experiments intended to probe that regime?\n\n3. Regarding the statement that “proper network width and weight decay prevent memorization,” can your theory specify how network size and weight decay should quantitatively relate to achieve generalization?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provides a solid theoretical analysis necessary for understanding the phenomenon of memorization and generalization in diffusion models. It rigorously establishes, in a statistical sense, the loss separation between the true and empirical score functions on empirical data, and quantitatively characterizes the network complexity required to learn each by leveraging results from universal approximation theory, which are appreciable theoretical contributions. Moreover, the authors effectively convey the high-level intuition behind their theorems, enabling readers to grasp the core ideas without following every technical detail.", "weaknesses": "- **Unclear theoretical hypothesis and motivation.** It is not entirely clear what specific hypothesis the theory aims to formalize or explain. Judging from the experiments, the paper seems to intend to relate sample size, network size, and weight decay to memorization and generalization in diffusion models, but the central claim or insight remains vague. The analysis does show that limited sample size can cause the empirical loss to favor memorization over generalization, but this is well-known and qualitatively unsurprising. It is unclear whether the authors' goal is to quantify the transition (e.g., identifying a regime such as $\\log n = O(d)$) or to argue its relevance to large-scale diffusion models. The theoretical results are not explicitly connected to the empirical observations in Section 6.1, making it difficult to understand what the experiments are meant to test or demonstrate.\n\n- **Weak empirical validation.**\nThe experiments do not concretely demonstrate the quantitative connection to the theory, e.g., the right balance between sample and network size is not discussed. Some claims, such as Lines 402–403 (“With sufficient sample size, network width increase promotes generalization”), depend heavily on narrow parameter ranges and is not properly substantiated (network width much greater than 1024 may eventually lead to memorization even for n=10K samples). The discussion of weight decay remains tautological (“proper network width and weight decay prevent memorization”) without clarifying what constitutes a proper scaling or whether such relationships can be derived from the theory.\n\n\n- **Ambiguous role and limited contribution of the pruning method.**\nThe pruning scheme introduced in Section 6 seems disconnected from the main theoretical narrative. Reducing model size to mitigate memorization seems to be a straightforward idea and is not clearly tied to the presented theoretical framework. Its empirical impact appears modest, and many existing fine-tuning or guidance-based unlearning approaches are likely more effective. Its inclusion blurs the paper’s focus between theoretical analysis and engineering heuristics.\n\n\nOverall, the paper seems uncertain about its main contribution: as a purely theoretical paper, the results are technically sound but conceptually expected; as a scientific study linking theory and empirical phenomena, it lacks a clear hypothesis-testing structure; and as an applied work, the proposed pruning method does not add substantial novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfY7ZbZ5wc", "reviewer": "Reviewer_ENVr", "summary": "This paper theoretically proves a fundamental separation between memorization and generalization in diffusion models. It shows that finite-sample Fisher divergence and network capacity jointly cause models to fit empirical rather than true score functions. Experiments demonstrate that pruning and regularization can reduce memorization while preserving generation quality.", "questions": "See Weakness. Further:\n    1. The results rely on sub-Gaussianity and Hölder smoothness assumptions. How sensitive are your main theorems to these conditions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides a rigorous theoretical analysis of diffusion models by quantifying the loss gap between the empirical and ground-truth scores under sub-Gaussian, Hölder-smooth data distributions, and further establishes an architectural separation showing that approximating the empirical score requires higher model complexity.\n2. The paper proposes a practical one-shot pruning method for Diffusion Transformers that removes low-importance heads in the small-t regime, effectively reducing memorization while maintaining or even improving generation quality, as supported by CIFAR-10 experiments.", "weaknesses": "1. The analysis mainly focuses on the small-t regime, but the paper does not empirically verify whether memorization indeed concentrates in this phase during real generation — an experiment comparing large-t and small-t generations could strengthen the claims.\n2. I think The theoretical results rely on strong assumptions such as sub-Gaussianity, which limit their applicability to real-world data.\n3. The proposed pruning method is tested on a single dataset, its generalization to other datasets or more complex diffusion models is uncertain, and its ability to substantially reduce memorization appears limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxwkWg0Q3N", "reviewer": "Reviewer_95eC", "summary": "The paper proposes a theoretical analysis of the memorization phenomenon in diffusion models. The analysis offers two perspectives: a statistical and an architectural one. The statistical perspective distinguishes between the ground-truth score function, which reflects the underlying data distribution, and the empirical score function that minimizes the loss on the (sampled) training data. Optimizing the diffusion model pushes the model to learn the empirical score function, which minimizes the training loss, instead of approximating the ground-truth score function, thereby leading to memorization. The subsequent analysis of the network complexity (number of parameters) demonstrates that over-parameterized diffusion models learn the complex empirical score function, also leading to memorization. Based on these theoretical insights, the paper explores mitigation strategies by reducing the model complexity by pruning attention heads or weight decay. These experiments are conducted on CIFAR-10 and Gaussian mixtures.", "questions": "- Do the findings also apply to large-scale diffusion models, e.g., Stable Diffusion trained on billions of data samples?\n- As a follow-up question: We know that Stable Diffusion has memorized some of its training samples. Does this mean the capacity of Stable Diffusion is too large? Or can we assume, since only a small share of duplicated training samples has been memorized, that Stable Diffusion already has a sufficient size, and the issue is the duplicated dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides fascinating insights into the training dynamics and influencing factors favoring memorization in diffusion models. Thorough mathematical foundations support the proposed framework. All necessary proofs are provided in the main paper or the appendix.\n- The theoretical analysis of the loss gap between the ground-truth and the empirical score function offers an intuitive explanation of why diffusion models can inadvertently memorize and replicate parts of their training data.\n- While, to some extent, this is to be expected, the pruning- and regularization-based mitigation strategies support model developers in their design decisions for building diffusion models and setting the training parameters.", "weaknesses": "- While the paper provides an interesting perspective on diffusion memorization, the core insight that a model tends to minimize the empirical loss instead of the ground-truth loss feels like the traditional bias-variance trade-off and overfitting phenomena in statistical machine learning. I do not want to reduce the paper’s contribution, but want to emphasize that the high-level message of the paper might have limited novelty. However, it might be the case that I overlooked some crucial novelty of the paper here.\n- The paper focuses on unconditional and low-resolution diffusion models trained on comparably small datasets (a couple of thousand training samples). Whereas I think this is appropriate for the provided analysis, it remains unclear to what extent the results transfer to more complex distributions, such as LAION data, and sample counts in the billions. Yet, I acknowledge that running such experiments is not feasible for small research labs, which is why I do not request such experiments. However, a more thorough discussion on how the authors expect their findings to scale or behave in such regimes would benefit the paper.\n- The results of the proposed pruning mitigation support an improvement in memorization mitigation. However, the reduction is relatively small (about 5 percentage points compared to the baseline, still reporting a 68% memorization ratio). Therefore, it feels less suitable for practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)