Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models
Authors: Benjamin Dupuis
Dario Shariatian
Maxime Haddouche
Alain Oliviero Durmus
Umut Simsekli
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion models
Generalization bounds
Score-based generative models
Primary area: theory
TL;DR: We prove algorithm- and data-dependent upper bounds on the generalization error of diffusion models by using tools from statistical learning theory

Abstract: Score-based generative models (SGMs) have emerged as one of the most popular classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, often demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt a largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice to train the score network. To support this observation, we first present simple experiments illustrating the concrete impact of optimization hyperparameters on the generalization ability of the generated distribution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior of SGMs. Our theoretical findings are supported by empirical results on several datasets.

Algorithm- and Data-Dependent Generalization Bounds for Diffusion Models
Benjamin Dupuis 1,2 Dario Shariatian 1,23, Maxime Haddouche
Alain Durmus Umut Simsekli
INRIA, France
CNRS, Ecole Normale Supérieure, PSL Research University, France Ecole Polytechnique, CMAP, IP Paris, France
indicate equal contrihutions.
Abstract
Score-based generative models (SGMs) have emerged as one of the most popu- lar classes of generative models. A substantial body of work now exists on the analysis of SGMs, focusing either on discretization aspects or on their statistical performance. In the latter case, bounds have been derived, under various metrics, between the true data distribution and the distribution induced by the SGM, of- ten demonstrating polynomial convergence rates with respect to the number of training samples. However, these approaches adopt largely approximation theory viewpoint, which tends to be overly pessimistic and relatively coarse. In particular, they fail to fully explain the empirical success of SGMs or capture the role of the optimization algorithm used in practice train the score network. To support this observation, we first present simple experiments illustrating the concrete impact optimization hyperparameters on the generalization ability of the generated distri- bution. Then, this paper aims to bridge this theoretical gap by providing the first algorithmic- and data-dependent generalization analysis for SGMs. In particular, we establish bounds that explicitly account for the optimization dynamics of the learning algorithm, offering new insights into the generalization behavior SGMs. Our theoretical findings are supported by empirical results on several datasets.
Introduction
Score-based Generative Models (SGMs) are among the most popular classes of generative models [HJA20a, SSDK 21, DN21, KAAL22a, EKB + 24]. with applications ranging from computer vision and medicine to natural language processing; see [YZS 24] for a recent survey.
The starting point of Score-based Generative Models (SGMs) to consider a stochastic process (X )xeej,,77 referred to as the forward process, which is the solution of an ergodic diffusion over the time interval [0, and initialized from the data distribution μ. Typically, (X.)tefo.r) is either d-dimensional Brownian motion
process, with stationary distribution given by the standard Gaussian, denoted by 7d [SSDK+21] In this work, we focus on the latter case. This construction defines path measure connecting μ to 7 and thus ideal generative model can be formed, by taking a large value of T and considering the time-reversed process associated with (Xx ()))))) defined for any t E [0, as X Xr-r. --- It turns out that the backward process is itself diffusion process, whose (non-homogeneous drift (t) s(t, x) depends the Stein scores of the forward marginals [HP86], which can be characterized as the solution to regression
problem [Vin11, Hyv05] involving simulations of the forward process (X t)eeoo.T) This drift can
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Butch
Text WarserN matee
14.00
bonch dian
10.00
Learning rate
Leaming rete
Loarinn nate
Figure I: Experiments with varying learning rates and batch sizes obtained with the ADAM optimizer. (left) test W metric on a Gaussian mixture dataset (middle) FIDJ, on MNIST (righ() FID on the butterflies dataset [WME09]. See Section 4.1 for full experimental details.
be estimated using a family of neural networks 80 (t. x:) + so(t, xt) parameterized by 0 E θ [SDME21]. Once the parameter& has been learned, an approximation of the backward process can be simulated by starting from 7 and applying numerical scheme discretize the corresponding stochastic differential equation, using the approximate score $g(w) in place of the true score function
8, where ((x) is the parameter obtained by the learning procedure and n is the number of data points. Because of their practical relevance. providing performance guarantees for SGMs has received increasing attention in recent years [DBTHD21, LDQ24]. A popular line of research (LLT22b, CCI,*23, CLL.23, LLT22a] provides theoretical guarantees on the discrepancy between the true data distribution and the generated distribution various metrics; particular we focus here on the
Kullback Leibler (KL) divergence. More precisely, denoting by 2 the distribution of the SGM, the
KL divergence the data distribution with respect to vy(h) can be bounded by three terms which are each associated with one type of approximations of the backward process:
(L(u) (m)) ) S & + + (0(m)) + &a.
where (i) &, accounts for the fact that the initialization is taken as 7d and not the distribution of X (ii) (n) (0(m) accounts for the approximation of s by sa and (iii) 6a accounts for the discretization error since in general solving the backward stochastic differential equation (SDE) is not an option even if we would have access the true score function.
Several studies provide quantitative bounds for the first and last terms & and 6s, which do not depend on the training data and the optimization algorithm that provides ((m), and they make the underlying assumption that the second term e!n)) (0(m) is small, i.e., the score network is good approximation of the true score of the forward process. This makes the bounds completely neglect the impact of the training set and the training algorithm used in practice. This question is at the core of generalization properties of SGMs since training a perfect score model on the empirical dataset would result in straight-up memorization of the dataset, entailing non-negligible score with respect to the true data distribution in the finite data regime [LCL.24, YSL23)..
A popular approach for analyzing the statistical properties of score-based generative models (SGMs) is to rely on approximation theory. The goal is to show that, within a given class of functions {so A θ). for any number of samples
1, there exists score estimator $g() such that
(!") (d!")) < C/n for a E [0, 1] which depends on intrinsic properties of μl and a constant C, both being independent of n. By combining such results with existing discretization error bounds, one can derive statistical guarantees for SGMs under various metrics. In the continuous-tine setting (i.e., & = 0). score approximation rate of order n (((/dd) was obtained in [OAS23], where a is a parameter related to the smoothness of the data distribution. As this rate deteriorates exponentially with the ambient dimension d, several studies have proposed relying on geometric assumptions on the data distribution, such as the manifold hypothesis [Bor22] (i.e., the support of lies on a bounded submanifold of dimension d, & d). In this context, also using neural networks, has been shown in [ADR24] that a rate of order n (d())d)) can be achieved, where a again depends on the smoothness of μ.. Alternatively, approximation guarantees for neural networks were also established in [CL24] by relying on a notion of complexity of the relative density of μ.. Similar approximation results have also been obtained for neural networks in the so-called neural tangent kernel regime [HRX24], as well as for kernel-bacr based score estimators [WWY24, ZYLL24, DKXZ24].
Despite recent advances, this approach suffers from two main limitations: (i) the considered class of score estimators sometimes far from those used in practice (e.g., UNet architectures (HJA)0a]), and, more importantly, (ii) it does not account for the impact of the learning algorithm (c.g., ADAM,

---

SGD) used in practice to obtain an estimator ((n)) on the generalization error. Indeed, while existing works establish existence results, the generalization error associated with the actual parameter f(m) returned by a learning algorithm remains unknown. In this paper, we argue that the learning phase has significant influence on the error of SGMs. We briefly illustrate this in Figure I, which shows the effect of ADAM optimizer hyperparameters (learning rate and batch size) on generation performance across three datasets Gaussian mixture model, MNIST [LBBH98], and the butterflies dataset [WME09] we observe that hyperparameters clearly influence performance as measured by the Wasserstein distance and Fréchet Inception Distance (FID). Such algorithm-depender behavior has also been observed in [SOE+24, 24, Figures F.7 and F.8).
Recently, several studies have proposed analyses that aim to account for both the impact of the learning algorithm and the data properties in the study of SGMs. To the best of our knowledge, the first attempt in this direction was made in [LLZB23], which analyzes idealized learning algorithm consisting of gradient flow a random feature model in the infinite-width limit. Alternatively, other works have adopted information-theorede work complements these efforts by taking an orthogonal approach and addressing some of their limitations. In particular, the bounds in [CZS25] involve only implicit dependence on the learning algorithm, making the bounds rather abstract and difficult to assess the actual effect of algorithm dependence on generalization. Moreover, their analysis does not incorporate the training set.
tools to derive generalization bounds, as in [CZS25]. Our
Contributions. Relying on an alternative approach, we propose a framework to derive data- and algorithm-depennent generalization bounds for SGMs. Our main contributions are as follows.
Generalization adapted decomposition. In Section 3.1, we provide a key decomposition of (0) for any 0 & θ, informally stated as
ef (m) = spln) (SM (0) + (m) + g((1
where 0 is any parameter of the score network. This decomposition highlights three distinet contributions. First, the explicit score matching loss L (see Equation (10)) that is optimized during the learning phase. Second, the data-dependent constant A." is concentration term capturing the interconnection between the data distribution, the dataset, and the forward process. Finally, (((")() (0) is a score generalization gap, quantifying the difference between risk that measures the quality of the score estimation and its empirical counterpart at parameter 0.
(0). We provide quantitative upper bounds on (m) in Section 3.2, and by making connections to smooth Wasserstein distance [NGK21] we show that is of order (l) + Eg (with a potentially large constant). We then show that ( (0) is directly amenable existing learning theoretic tools and we use two existing algorithm- and data-dependent bounds [MWZZ18, ADS DDS+ that cover a broad range of algorithms. Combined with our theory, these bounds suggest that the gradient norms and the topological properties of optimization trajectories can provide useful information about the generalization performance of SGMs. Furthermore, since (((") (0) is also of order O vT). this ultimately gives us a high-probabiliin bound of the form (x) ESM (0) + O(1//) + & + & on the KL divergence between the true data distribution and the generated distribution.
Characterizing !n) and (i(T)
Experimental validation. We design low and high dimensional experiments to validate our theory on different algorithms, varying optimizers (SGLD, ADAM), learning rates and batch sizes. Our make our implementation publicly available https: //github/tttttt/// com/ dari /ahrrrh Generalization-Diite
Notation. For two probability measures μ and v, the absolute continuity of μ. with respect to is denoted by < u. The Kullback-Leibler divergence of with respect to is KL(A)v) == I log(dp//dz)d if μ & v, and (L(plv) == +oo otherwise. We also define the Fisher information of μ with respect to v as I (plvv) == I|V log(du/dd dv) ||²du. We denote by 7d the standard d- dimensional Gaussian distribution. For any random variable Y, we denote by Law(Y) its distribution. We denote B(0, D) the ball of radius D, centered at We write A S B whenever & CB for universal constant C' that neither depends on the assumption's 's constants nor the parameters at hands.
3

---

2 Background on Score Generative Models
Forward and backward process. We consider as the forward process standard d-dimensional Orastein-Uhlenber process, solution of the SDE starting from jr:
= V2dB,
NO~P,
(2)
where (Be) is standard d-dimensional Brownian motion. Denote by p', the density of X, with respect to the Lebesgue measure and by Pt- F h//n its density with respect to y for t > 0. We assume that j has a density with respect to the Lebesgue measure, and Pu u denotes this density.
Under mild regularity conditions [And82, HP86], the time-reversal (Xe)OEtET of (X eloGIET over a time interval [0, T) for some time horizon T > 0, defined by X, = 1=xT-1 T it,,i, is solution of the SDE! - s(' - X Jdt + v2dB, X Law((a)))
where we define the score function s(t, x)) = 2V log pr for any t E [0, T) and x E K. Note that (Be)czo denotes a standard d' d-dimensional Brownian motion, which is distinct from the one used in (2). However, for notational simplicity and by convention, we use the same symbol. The function (1, + V log Pe (ar) is known as the score function. In practice, simulating the backward process is infeasible, and approximations are required. The first challenge arises from the fact that the score function is unknown. However, it can be estimated from data sampled from fa described below.
Score Estimation. Using Fisher's identity [Efr11], it is well-known that the score function (1,z) + s(1, x) satisfies for > 0, s(1, X.) = 2 V log Pejo( X, xx))) where Prjo denotes the conditional density of Xx given Xo. with respect to 7 Therefore, it the solution of a regression problem. Based
zt) I }). typically neural networks with denoting their weights, we can then learn the parameter by minimizing the population risk 0 + El (0, 2).. associated with denoising loss function, where Z is a sample from p., 0 E O
parametric family <(t,z)
E IRd and w probability distribution over R4, as:
(0, == / E so Xi) - 2V log (111))
(4)
where N indicates the forward process (2) with initial value Xo z. In practice need rely on the empirical risk associated to a dataset Z (ZI...., ....... Zn) nm of i.idd. samples from μ.. Therefore, a learning algorithm (e.g., SGD or ADAM) is used for obtaining a parameter g(n) by minimizing the following empirical denoising score matching loss:
gln,. DSM
(0)
1 Σ la (0, = n
(5)
Backward simulation. Once an estimate (0) has been obtained, the corresponding score network Sgin) is used for approximately simulating Equation (3). However, even when replacing the true score in (3) with this estimator, the resulting SDE cannot be solved explicitly. In practice, we must address two main challenges: (i) the initial distribution is intractable, and (ii) it cannot be exactly simulated. To overcome the first issue, we exploit the fact that the OU process converges geometrically fast the standard Gaussian distribution 7 and use this as the initialization of our model. For the second issue, we rely on discretization scheme. In this work, we focus on the exponential integrator (EI) scheme [DM15], which has also been adopted in several recent studies [CDS25, BBDD24, ADR24, ZC23],
Let N E N and h > 0 be the step size, with T Nh. Define the time steps by ts := kh for k E 1, l,....,.. N. The Fuler El scheme then defined as follows: starting from X(n) ~ 7d, for each
l E 1, N, and given {(n) the trajectory (X((m)) ) is the solution to linear SDE.
dxp (n) (-X(")) + Sgcw) (T L2, 8(()) ()tt) + V2dB.
(6)
We denote by vt the distribution of {(11) and refer to it as the generated distribution.
Convergence bounds. To avoid technicalities and in particular the use of early stopping procedure, we rely for our anlaysis on the following assumption following [CDS25]:
Note that consider the density of X with respect 7dh leading to negative linear drift in (3) [CDS25].

---

Assumption 2.1. The Fisher information between μ and 7 is finite, i.e., IuU))) < oc.
major challenge emerging from the above procedure is control the discrepancy between μ and (m) In particular, under Assumption 2.1, [CDS25, Theorem 1] states the following bound.
Theorem 2.1. Under Assumption for any and N N such that T hN, it holds
KL(p
(
2 c 2 (1 + Te! (f(x)) + h. 1- (ulgt).
where (0) T E 1
hE
||se(T - tk x12) 2V log - (X (XT-w)1?)
The two terms accompanying e! are specific to the approximations involved in modeling the backward process underlying the diffusion model we consider. The first term accounts for the fact that the diffusion model is initialized from 7d rather than from Law (X ) The second term corresponds to the discretization error introduced by using the exponential integrator (EI) scheme. Finally, the quantity e!") reflects the quality of the score approximation achieved by the score network. We note that several studies have established other guarantees for SGMs under various assumptions, which Es naturally appears [CDS25, BBDD24, CLL.23, 23,+ LLT22b].
3 Towards a better understanding of generative performance
This section proposes more in-depth study of e²eh Informally, show in Section 3.1 the following decomposition (t) (0) L (0) + !n) + "0(")(( (0), where LESM(() is defined in (10), and A (n) (n) (0) highlight respectively the influence of: (i) statistical behavior of the training set, (ii) the problem of learning
We then upper-bound A." in Section 3.2.
3.1 A key decomposition
wirst,we define the following probability measure over I4::
N
A
(8)
T
k=
where d denotes the Dirac measure. For case of notation, we denote DDS by _o(n) see (5).
With these notations, it is clear that L (0) R (0) n n D == (0, Z:), which we refer
as the empirical risk, following standard terminology learning theory. This observation naturally leads the definition of the corresponding population risk, R (0) [0, 2)] with Z μt, and the associated score generalization gap:
(2) Iz (1) ()) = R (0) - z (0) = / (x(0, 1 Σ (0 Z.) 0)
(Z
8=1
This definition of the generalization gap consistent with practice as en involved during training. Hence, upper bounding (Z))), g(m)) is meaningful. Before exploring this route in Scction 4, we first show that Sx (Z( ((x)) naturally stems from (n)
To state our next result, we define for any n E N, (X (X!")) )es[o,T) as the solution of Equation (2)
(1)
initialized randomly from the empirical distribution X pn - -n 82, instead of μ.
Theorem 3.1. For all 0 E θ, we have:
a (0) = (0) + Am)
where (rr)
=C(1)
-Cr,
n
Tim) :
E E||V log Pe (x - V logg (x?
n1
i=1
5

---

CT = 4
B|IV log (X;) - -V Pog (Xi1=) ())))
N
(BM =
h
E ||so(T (T - tR. X x 2V (()) (x (17111))))
(10)
T
k=0
where we denote by ml the density of X(n) with respect to yd.
The proof is postponed to Appendix A. We refer to the quantity A as the 'data-dependent diffusion gap' as measures the discrepancy between the forward diffusions that are initialized at either the empirical data and the true data distribution. In addition, the term L (0) is called the explicit score matching loss [Vin]l] 1]. It corresponds to the quality of the approximation the empirical score v log p. (!()) by the score network, and is optimized by the learning algorithm.
Combining Theorem 2.1 with Theorem 3.1 implies that the control of the generative performance of (()) boils down bounding (m) and S (Z(Zz) (Z()),) 0). The former is handled in Section 3.2, quantifying the impact of n via concentration arguments, and the latter in Section 4.
3.2 Quantifying the influence of the dataset size in generalization
We aim to upper-bound the term A of Theorem 3.1. To do so, make the following assumption. Assumption 3.1. The data distribution p has bounded support included in B(0, D). for some D > 0.
This assumption is relatively common in the literature, and typically expect it be satisfied for image datasets. We expect that most of our proofs would still hold with assumptions on the moments of Ju. We leave these considerations for future works.
An important remark that ((r) the difference between an empirical average and its theoretical counterpart. Based on this observation, we aim at quantifying the influence of n in the generalization phenomenon. A first step is provided the following result.
Lemma 3.1. Under Assumptions 2.1 and 3.1, with probability at least 5 over Z(n) p 0m
1og(1/0)
(m)) < 4D2
21d((1) +4 +4 A dx(1), aX()),
2n
where A. == ( I (7)(())
(1) ((a) and X is defined in (8).
Note first that (n)) has a small contribution to the error if +> A stays negative over a subset of [0, 7| with large Lebesgue measure. A precise understanding of this phenomenon is promising research direction, which we leave for future works.
(n)
To contextualize the previous remark, we rely on an observation that was also made by [FRDD25], in submission that is parallel our work. The next lemma shows that the data-dependent gap is negative in expectation over the dataset.
Lemma 3.2. We have E ((r)) <
where the expectation over (n) @
This shows that, in expectation, the data-dependent gap (x) can be discarded from our decomposition. However, our work, we are mainly interested in proving high- probuublitt bounds, which we believe is essential in the following proposition.
expect to have important variance in certain settings, make more precise
We provide in the next proposition a quantitative bound on A
Proposition 3.1. Under Assumption 3. with probability at least - 25 over, we have: (00) ~ on:
A (n) (D² K?) V
1og(1/8)
+ h
T (p)n") + K? log(1/8)
W² + KgVhW
2n
Th
with K3 = D log (T) (T/h) + hd, K? = d(1 - e 2h + D2
W: We (FN/2) {1122

---

Proposition 3.1 does not provide an explicit convergence rate in n. The Fisher information term is the same as in Theorem 2.1 and controlled when N T/h is large. On the other hand, when h is fixed, the quantity W satisfies W < e w (u * N(0, L') fhn * N(0, d²La)). where ā ;= V 1, and the right-hand side corresponds to the smoothed Wasserstein distance between μ and f... Bounding such quantities has received increasing attention in the literature INGK21, BJPR25, GGNWP20J.
In particular, [GGNWP20] proved that E [W O (n cxp /(c* " - 1))). High-probability bounds with similar constants and n-dependence have also been established [NGK21]. Therefore, Proposition 3.1 implies that for a fixed h, a convergence rate of O (n-1/2) can be achieved, albeit with constants that can grow rapidly the generalization ability of SGMs. It can be seen that in worst case scenarios, plugging this bound in Theorem 2.1 and optimizing over h leads to rate of n -O114), similar to existing works [ADR24, Øks03).
h + 0. This quantifies the influence of dataset size on
In summary, combining Theorem 2.1, Theorem 3.1, and Proposition 3.1 yields full characterization of the generalization error KL(μ f)"),, ). up to the score generalization gap (Z 0), which we
analyze next.
4 Unveiling the influence of the learning algorithm on generalization
We amalyze next the score generalization gap 9 (Z(Z()) ((x)) when ((() is the output of two algorithms: (i) the stochastic gradient Langevin dynamics [WT1] 1] and (ii) the ADAM algorithm [KB17] In both cases, the proposed results build upon existing generalization bounds. To our knowledge, this is the first time such a transfer of theoretical tools from supervised learning to generative modeling is possible, highlighting the benefits of the optimization algorithm, measure of the generalization error of diffusion models.
4.1 Experimental Setup
We set the Omstein-UUllnbeck process as our forward diffusion and use the cosine noise schedule [DN21]. We opt for the denoising parameterization of the model and its associated 'c-loss' (see Appendix C, (22)), as introduced in [HJA20b] and widely used afterwards [DN21, HS21, SH22, KAAL22b, EKB + 24]. This improves numerical stability and yields faster convergence as the high variance of the DSM loss (5) incurs noisier gradients and losses. As a result, we plot the generalization error as the difference between the train and test c-loss, which serves as proxy for the score generalization error, as they only differ by dime-dependent multiplicative factor (SSDK*211 We employ the Euler EI (6) to sample from our trained models. Full experimental details are available in Appendix C.
The first set of experiments is based on a 4-dimensional dataset consisting of a mixture 9 Gaussian distributions, with random means, and with some class imbalance to make the learning task harder?. Supplementary details are given in Appendix C.I. We then shift focus to higher- higher-dimensional datasets, the flowers dataset [NZ06], and the butterflies dataset [WME09]. We also include experiments on the MNIST digits in Appendix C.3. For images, we use the DDPM+ U-Net architecture as implemented in [DN21]. We also study topological generalization bounds [ADS* $ 24] associated to the training trajectories of the ADAM optimizer [KB17], which is the optimizer used in practice for state of the art models [RBL* EKB 24]. We vary learning rates and batch sizes to obtain multiple measure points and validate our bounds. Supplementary details are given in Appendix C.2.
4.2 Stochastic gradient Langevin dynamics and the inffuence of gradient norms
We first study the stochastic gradient Langevin dynamics (SGLD) [WT11], which we see as a noisy variant of SGD. In learning theory, the generalization ability of SGLD has been widely studied [RRT17, PJL18, FR21, NHD 19, DS24J. We define SGLD by the following recursion:
0 (1 anz) ())r
(0) / /2m.B-IG.
where G2 ~ (O, Oh is an unbiased estimate of the gradient of the empirical risk, and a >> 0 regularization coefficient. The term 3 is called the inverse temperature parameter.
"The exact mixture weights are (0.01, 0.1, 0.3, 0.2, 0.02, 0.15, 0.02, 0.15, 0.05)
7

---

0.420
tempereture 0.0003
temper ature 06
0.925
tomperature + 1m-10
-
Caneraltaation gap
0.010
Goseraltantton
0.010
0.015
0.000
0,000
0.000
0.005
0.000
-8.000
-0.010
0.2
0.5
944
0.5
0.6
0.7
-0.015
-0,810
200
400
Va
Figure 2: SGLD optimizer (17) on a low dimensional Gaussian mixture dataset, for different value of the temprature (1/ We use full batch size, constant learning rate 7) grid of values of (n, 7) and 10 random seeds. x-axis: Value of v ( /n. y-axis: Score generalization gap.
In the following, we fix number of iterations K E N' and denote by 0x the output of SGLD after K steps. To analyze the term 9 (Z("), 0x) in the case of SGLD, a wide variety of generalization bounds are available [DS22, Table 4], often involving expected gradient norms of the training process [MWZZ18, NHD* 19, NDHR21, HNK*20) 20]. Here, we exploit the seminal result of [MWZZ18] in the context of SGMs. First, recall that a random variable X is -ussuaaausia if for any a E R. E [exp(a(X - E [xi))] < exp(a? (2/2) [Ver18].
Theorem 4.1 ([MWZZ18)> We assume that for any w E Rd,, the loss (x(w, 2) is -ssuuguussia with respect to Z ~ Js and that sup (M)a) < 1. We also assume the algorithm is initialized with
to ~ To = N (o, LaLa) with ooy Ba & V2. Then, with probability at least 1- ô over Z(x) ~ n we have:
2r 8 K
3 31 1/2
k-
E
3 (7
S
Σ nhe
+ log
Sk = Th
Σ j=0
vn 2
8
Theorem 4.1 shows that, up to a multiplicative constant involving n. the averaged gradient norms form an upper bound on Gx (z)), 0x) and thus impacts the generalization error of the model. To verify this claim, we consider the case of constant learning rates, i.e., fu2 = 1/, and take n = 8192, 0 and batch size equal to n. Let (I[@= 11?) be the average gradient norm all the iterations. In Figure 2, obtained with SGLD on a low-dimensional gaussian mixture model, we compare the score generalization gap to the value of ((n, 17) == / (1122)1)) for different inverse temperatures B, grid of values of n and 7), and 10 random seeds. The order of magnitude of Bnn, 11) in Figure 2 bigger than the observed score generalization gap. This behavior commonly seen for gradient- based bounds [DDS23] and may come from the unknown subgaussian constant 7 in Theorem 4. or additional implicit regularization. Yet, the results support Theorem 4.1, reporting a good correlation between B(n, 1/) and the generalization error, especially for high values of the inverse temperature implicitly
9. While our theory does not take explicitly into account the model architecture, impacting the bound through the gradient norms in Theorem 4.1.
While our theory does not rigorously apply to more practical optimizers like SGD or ADAM because of the absence of Gaussian noise, we use the following heuristics based on [MHB16] 16] to extend our experiments beyond the class of noisy algorithms. By considering that the variance of the stochastic gradient is of order 1/b, where b the batch size, we replace 3 by b/п and propose to compare the generalization error to [|@+ ?| and apply to the ADAM optimizer. In this last case, we only average the last 200 gradients of training, to avoid noisy gradients in the first observation and characterize the geometry of the local minimum the model converged to. We observe in Figure 3 that this quantity correlates very well with the generalization error on the butterflies and flowers datasets. We provide in the appendix an additional experiment on a dataset (MNIST) with more data points, where the correlation strong for most hyperparameters, A refined analysis shows that the observed correlation is related to the train loss of the model, suggesting that the relevance of the experiment increases near convergence, see Figures 5 and 6. Thus, results suggest that gradient norms are a pertinent indicator of generalization for SGMs.
The question arises, as to whether the subgaussian assumption made in Theorem 4.1 can be met in practice. Verifying this assumption depends simultaneously on the datasel and the architecture, as refers to the denoising score matching loss. While it cannot be seen direct consequence of our other assumptions, we expect that it can be satisfied in simple settings, such as image datasets or finitely supported distributions. We leave these considerations for future works.
8

---

385
209
430
18.
9.00 6.62 6.84 9.86 0.03 010 032
0.00
9.05
0.30
18-1
6.06
3.09
6.10
0.22
6.34
9.00
0.19
0.12
Baba
4300
Poutto weng
2000
5
Datch Bba
4400
3630
4400
123
4300
0.82 0.08 0.00 0.22
Deseraication
5.00
0.10
GesecaBbation
0.10
0.36
0.02
Figure 3: ADAM optimizer on the butterflies dataset (left) and the flowers dataset (right). Generalization gap vs. several complexity metrics: b(Il&(l2) (top left), E (W(n)) (top right), PMag(10-2. W(() (bottom left) and PMag(v' W(r) (bottom right).
4.3 The influence of training trajectories and application to ADAM
While providing fruitful insights and generalization measures, SGLD might far from the learning algorithms used in practice (e.g., ADAM). Here, exploit the recent topological generalization bounds of [DVDS24, ADS 24], which can be applied for a large class of algorithms including ADAM. Topological bounds are based the intuition that the training trajectory (i.e., the parameter sequence generated by the optimization procedure) might encode topological properties of local minima, related to their generalization ability.
We build our analysis on the results of [ADS+24]. Let us fix ko, k E N+ and introduce the training trajectory Win) == (0("). ko < k < k1), where o(m) denotes the learned parameter of the score network at the k-th iteration, and ho is chosen such that o(m) close to a local minimum (near convergence). Topological bounds relate the generalization eπтоr quantities quantifying the topological complexity of W(n),, stemming from topological data analysis (TDA) [BCY18]. We focus here on the particular case where this complexity is the weighted lifetime sum [Sch20], which informally tracks down the number of clusters of W)() at different scales. We denote E' (WV()) and formally introduce it in Appendix B.2. More precisely, E' (W(") denotes the cost of minimum spanning tree over the finite Wi(), for a given distance (e.g., the Euclidean distance). We refer to [ADS* 24] for an introduction of this notion in the context of generalization bounds.
The next theorem shows how the weighted lifetime
upper-bounds the generation performance. Theorem 4.2 (LADS 24]). Assume that the loss (0, z) + (0, z) is uniformly bounded by B > 0. Suppose Assumption 2.1. Then, with probability at least 1 - 5, we have for all E Wi)) that:
log(1+ (B
+ + Lo0 (W(n), Z(m) + log (1/8)
(Z( (1) 0) & B
where Toc (W(n), (()) is mutual information term defined in Appendix B.2.
Note that similar bound involving another notion of complexity (the positive magnitude) presented in Theorem B. due to space limitations. The positive magnitude, introduced by [ADS' 24], is quantity of similar flavor to E (W)()) and additionally depends on a scale parameter r > 0, is denoted PMag(r- Wo(i).. In our experiments, we consider the choice r = /n (which is theoretically justified by [ADS+24]) and r 10-2 as these authors argued that positive magnitude for smaller values of r empirically correlates with the generalization error (the scale 10-2 is used by these authors).
The information inforrmation-thcorttii term in Theorems 4.2 and B. is hard to estimate in practice, even though was successfully bounded in particular cases [DVDS24]. For this reason, proceed as in [ADS 24] and empirically illustrate the correlation between the three topological complexities (I)))), PMag(10-2 W((o) and PMag( V)(m)) and the score generalization gap in Figure 3. Up to our knowledge, it is the first time that these topological generalization bounds are evaluated for diffusion models. For the butterflies and flowers datasets, we observe in Figure 3 that the three

---

proposed topological complexities correlate very well with the score generalization gap. Slightly worse correlations are observed for PMag((10 (w(m)),)) which is in line with the theory of [ADS 244. We provide some generated image grids for the butterfly dataset in Figure 4 for visual comparison. Table 2 reports Train/Test DSM together with bx (I[@) E' (WW())) and PMag(-) across learning rates and batch sizes for ADAM on flowers, complementing Fig. 3.
We also include additional experiments for the MNIST dataset in the appendix, see Figures 5 and 6, In that case, the positive magnitude also has satisfying correlation with the generalization error while for E' the situation is slightly more contrasted. Similar to the above, it seems the lack of convergence of the model can negatively impact the relevance of E", which is coherent with the observations of [DDS23, ADS + 24]. We also make the new observation that E and the gradient norms-based bounds have very similar behavior. Thus, our experiments show that the topology of the training trajectories has an impact the generalization error of SGMs.
Finally, let us note that our generalization bounds do not explicitly take into account the noise schedule, which is known to empirically affect the performance of the score network, but, as the noise schedule affects the properties of the optimization dynamics, we expect its effect to be captured by the generalization error term in our decomposition.
5 Conclusion
In this paper, we proposed an algorithm- and data-dependent analysis of the generalization abilities of practically used diffusion models. Our theoretical analysis is based a decomposition of the score approximation error. After providing two upper bounds of a statistical ersatz arising from this approach, we focus our diseussion on what call the score generalization gap, which represents the generalization error associated to the denoising score matching loss used during training. We apply our framework to several classical stochastic optimization algorithms and obtain generalization bounds with explicit dependence on the training dynamics. These results altogether yielded KL bound with (n-2/22 rate. Based on these observation, we numerically evaluated the correlation between the score generalization gap and the two topological complexity measures and gradient norms, hence, providing new empirical insights for diffusion models.
Limitations and future works. Our theoretical bound in Section 3.2 may be coarse in certain scenarios, and overall suggests potential refinements incorporating such as the conditional entropy of the data given its noisy observation, as inspired by recent work on entropy-basee based noise schedules. This could lead to a deeper understanding of generalization, particularly regarding more involved data-dependent quantities. On the experimental side, thorough evaluation of our theoretical predictions requires well-trained diffusion models across a wide range of settings. However, the high computational cost of full training runs, while varying many hyperpa- rameters, currently limits the scale of our empirical analysis. In particular, we leave for future work a broader exploration involving higher-dimensional datasets and larger training sets.
coormationnthhoreee quantities
Acknowledgments
U.S. is partially supported by the French government partly funded this work under the management of Agence Nationale de la Recherche as part of the "France 2030" program, reference -3--23- IACL-0008 (PR[AIJRIEPSAAI) B.D., M.H., D.S., and U.S. are partially supported by the European Research Council Starting Grant DYNASTY 101039676. A.D. acknowledges funding by the European Union (ERC-2022-SyG, 101071600 601). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. The authors thank Tyler Farghly and Eliot Beyler for helpful discussions. The authors grateful to the CLEPS infrastructure from the Inria of Paris for providing resources and support. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD01 1015323R1 made by GENCI.
Broader impact statement. Our research is theoretical and raises no direct societal or ethical concerns.
10

---

References
[ADR24] Iskander Azangulov, George Deligiannidis, and Judith Rousscau. Convergence of diffusion models under the manifold hypothesis in high-dimensions, 2024.
[4DS*24] Rayna Andreeva, Benjamin Dupuis, Rik Sarkar, Tolga Birdal, and Umut eSimesekli. Topological Generalization Bounds for Discrete-Tine Stochastic Optimization Algo- rithms. In Advances in Neural Information Processing Systems (NeurlPS), 2024.
[AJH+23] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces, 2023.
[And82] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[Bau25] Adrian Baule. Generative modelling with
2025.
[BBDD24] Joe Benton, Valentin De Bortoli, Amaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations, 2024.
[BCY18] Jean-Daniel Boissonnat, Frédéric Chazal, and Mariette Yvinec. Geometric and Topo- logical Inference. Cambridge University Press, 2018.
[BGL14] Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffussin Operators. Springer, 2014.
[BJPR25] Adam Block, Zeyu Jia, Yury Polyanskiy, and Alexander Rakhlin, Rate of convergence of the smoothed empirical wasserstein distance, 2025.
[BLGcS21] Tolga Birdal, Aaron Lou, Leonidas Guibas, and Umut eSimesekli. Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[Bor22] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research, 2022. Expert Certification.
[BS25] Francis Bach and Saeed Saremi. Sampling binary data by denoising through score functions, 2025.
[BSDB+24] Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(2):286331, 01 2024.
[BSS 24]
Andrea Bertazzi, Dario Shariatian, Umut Simsekli, Eric Moulines, and Alain Durmus. Piecewise deterministic generative models, 2024.
[CBB + 22] Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deli-
giannidis, and Arnad Doucet. continuous time framework for discrete denoising models, 2022.
[CCL+23) Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R. Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions, 2023.
[CCSW22] Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. Improved analysis for a proximal algorithm for sampling. In Po-Ling Loh and Maxim Raginsky, editors, Pro- ceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2984-3012 PMLR, 02-05 Jul 2022.
[CDS25] Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. KI convergence guarantees for score diffusion models under minimal data assumptions. SIAM Journal Mathematics of Data Science, 7(1):86-109, 2025.
[CL17] Djalil Chafai and Joseph Lchec. Logarithmic sobolev inequalities essentials, 2017.

---

[CL24] Frank Cole and Yulong Lu. score-based generative models break the curse of dimen- sionality in learning family of sub- sub-gaussian distributions. The Twelfth International Conference on Learning Representations, 2024.
[CLL23] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score- based generative modeling: User-friendly bounds under minimal smoothness assumptions, 2023.
[CZS25] Qi CHEN, Jierui Zhu, and Florian Shkurti. Generalization in VAE and diffusion models: A unified infoociiionthheortiicc analysis. In The Thirteenth International Conference on Learning Reprexentations, 2025.
[DBTHD21] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-based generative modeling. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 17695-17709. Curran Associates, Inc., 2021.
[DC25] Leello Tadesse Dadi and Volkan Cevher. Generalization of noisy SGD under isoperime- try, 2025.
[DDS23]
Benjamin Dupuis, George Deligiannidis, and Umut Simsekli. Generalization Bounds with Data-dependent Fractal Dimensions. In International Conference Machine Learning (ICML), 2023.
[DKXZ24] Zehao Dou, Subhodh Kotekal, Zhehao Xu, and Harrison H. Zhou, From optimal score matching to optimal sampling, 2024.
[DM15] Alain Durmus and Éric Moulines. Quantitative bounds of convergence for geo- metrically ergodic markov chain in the wasserstein distance with application to the metropolis adjusted langevin algorithm. Statistics and Computing, 25:5-19, 2015.
[DN21] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780-8794, Curran Associates, Inc., 2021.
[DS24] Benjamin Dupuis and Umut Simsekli. Generalization Bounds for Heavy-Tailed SDEs through the Fractional Fokker-Planch Equation. In International Conference Machine Learning (ICML), 2024.
[DSL22] Jacob Deasy, Nikola Simidjievski, and Pietro Liò. Heavy-tailed denoising score matching, 2022.
[DV23] Benjamin Dupuis and Paul Viallard. From Mutual Information to Expected Dynamies: New Generalization Bounds for Heavy-Tailed SGD. In NeurIPS 2023 Workshop Heavy Tails in Machine Learning. 2023.
[DVDS24] Benjamin Dupuis, Paul Viallard, George Deligiannidis, and Umut Simsekli. Uniform generalization bounds data-dependent hypothesis sets via pac-bayesian theory random sets. Journal of Machine Learning Research, 25(409):1-55, 2024.
[Efr11] Bradley Efron. Tweedie's 's formula and selection bias. Journal of the American Statistical Association, 106(496):
2011.
[EKB + 24] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference Machine Learning, 2024.
[FR21] Tyler Farghly and Patrick Rebeschini. Time-independent Generalization Bounds for SGLD Non-convex Settings. Advances Neural Information Processing Systems (NeurIPS), 2021.
12

---

[FRDD25] Tyler Farghly, Patrick Rebeschini, George Deligiannidis, and Amaud Doucet. Implicit regularisation in diffusion models: An 2025.
generalisation analysis,
[GGNWP20] Ziv Goldfeld, Kristjan Greenewald, Jonathan Niles-Weed, and Yury Polyanskiy. Con- vergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, (4367) 4391, 2020.
[HJA20a] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[HJA20b] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[HNK+ 20) Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel Roy, and Gintare Karolina Dz- iugaite. Sharpened Generalization Bounds Based on Conditional Mutual Information and an Application Noisy, Iterative Algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[HP86] U. G. Haussmann and E. Pardoux. Time reversal of diffusions. The Annals of Probability, 14(4): 188-1205 1986.
[HRU+1] 17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NIPS, pages 6629-6640, 2017.
(HRX24) Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. In The Twelfth International Conference on Learning Representations, 2024.
[HS21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[HcSKM22] Liam Hodgkinson, Umut eSimesekli, Rajiv Khanna, and Michael Mahoney, General- ization Bounds Using Lower Tail Exponents in Stochastic Optimizers. In International Conference on Machine Learning (ICML), 2022.
[Hyv05] Aapo Hyvärinen. Estimation non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695 709, 2005.
[KAL222]] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[KAAL22b] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[KB17] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[Las17] Dawid Laszuk. Python implementation of empirical mode decomposition algorithm. https://githut: com/ Laszukdawid/PPEEM) 2017.
[LBBH98] Y. Lecun, I. Bottou, Bengio, and P. Haffner. Gradient-based learning applied to
document recognition. Proceedings of the IEEE, (6(11):2278-2324 1998.
[LCBH+23] Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023.
13

---

[LCL24] Sixu Li, Shi Chen, and Qin Li. A good score does not lead to good generative model, 2024.
[LDQ24] Xiang Li, Yixiang Dai, and Qing Qu Understanding generalizability of diffusion models requires rethinking the hidden gaussian structure. In Globerson, L. Mackey. D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 57499-57555 Curran Associates, Inc., 2024.
[Leil3] Tom Leinster. The magnitude of metric spaces. Documenta mathematica, 18:857-905, 2013.
[L.GL222] Xingehao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: generate and transfer data with rectified flow, 2022.
Learning
[LLT22a] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[LLT222b Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions, 2022.
[LLZB23] Puheng Li, Zhong Li, Huishuai Zhang, and Jiang Bian. On the generalization properties diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 2097-2127. Curran Associates, Inc., 2023.
[LME24] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios the data distribution, 2024.
[Mec15] Mark W Meckes. Magnitude, diversity, capacities, and dimensions of metric spaces. Potential Analysis, 42(2))549-572, 2015.
[MHB16] Stephan Mandt, Matthew Hoffman, and David Blei. A variational analysis of stochastic gradient algorithms. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference Machine Learning, volume 48 Proceedings of Machine Learning Research, pages 354-363, New York, New York, USA, 20
Jun 2016. PMLR.
[MWZZ18] Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints. In Conference On Learning Theory (COLT), 2018.
INDHR21] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel Roy. CnpemattionThhoreenn Generalization Bounds for Stochastic Gradient Descent. In Conference Learning Theory (COLT), 2021.
[NGK21] Sloan Nietert, Ziv Goldfeld, and Kengo Kato. Smooth p- p-wssersseein distance: Struc- ture, empirical approximation, and statistical applications. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learn- ing, volume 139 of Proceedings of Machine Leurning Research, pages 8172-8183. PMLR, 18 28-24.1 Jul 2021.
19] Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel Roy. Information-Theoree Generalization Bounds for SGLD via Data- Dependent Estimates. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[NZ06] Maria-Elena Nilsback and Andrew Zisserman. A visual vocabulary for flower clas- sification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006.
14

---

[OAS23] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26517-26582. PMLR, 23-29 Jul 2023.
[Øks03] Bernt Øksendal. Stochastic Differential Equations. Springer, 2003.
[Pel23] Stefano Peluchetti. Non-denoising forward-time diffusions, 2023.
[PJL18] Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization Error Bounds for Noisy, Iterative Algorithms. IEEE International Symposium on Information Theory (ISIT), 2018.
[PS0+25] Ne-Tuyet-NNi Pham, Dario Shariatian, Antonio Ocello, Giovanni Conforti, and Alain Durmus. Discrete markov probabilistic models, 2025.
[2BL*22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[RRT17] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-Convex Learning via Stochastic Gradient Langevin Dynamics: Nonasymptotic Analysis. Conference Learning Theory (COLT), 2017.
[Sch20] Benjamin Schweinhart. Fractal dimension and the persistent homology of random geometric complexes. Advances in Mathematics, 372:107291, 2020.
[SDME21] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[SH22] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[SOE*24] 24] Fabian Schaipp, Ruben Ohana, Michael Eickenberg, Aaron Defazio, and Robert M. Gower. MoMo: Momentum models for adaptive learning rates. In Ruslan Salakhut- dinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pages 13542-43570. PMLR, 21-27 Jul 2024.
[SSD25] Dario Shariatian, Umut Simsekli, and Alain Oliviero Durmus. Denoising levy proba- bilistic models. In The Thirteenth International Conference on Learning Representa- tions, 2025.
[SSDK+21] Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representationss 2021.
[Ver18] Roman Vershynin. High--Diensional Probability: An Introduction with Applications Data Science. Cambridge University Press, 2018.
[vH14] Tim van Erven and Peter Harremoës. Rényi Divergence and Kullback-Leibler Diver- gence. IEEE Transactions on Information Theory, 2014.
[Vi109] Cédric Villani. Optimal Transport Old and New. Springer, 2009.
[Vin11] Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7)11611-1644 July 2011.
15

---

[VSP+ 17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, É ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[WME09] Josiah Wang, Katja Markert, and Mark Everingham. Learning models for object recognition from natural language descriptions. Proceedings of the British Machine Vision Conference, 2009.
[WT1]] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Lise Getoor and Tobias Scheffer, editors, Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28- - July 2, 2011, 2011.
[WWY24] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Research, pages 4958 4991. PMLR, 30 Jun-03 Jul 2024.
[YPKL23] EUN BI YOON, Keehun Park, Sungwoong Kim, and Sungbin Lim. Score-based gen- erative models with lévy processes. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informaion Processing Systems, volume 36, pages 40694-40701 Curran Associates, Inc., 2023.
[YSL23] Mingyang Yi, Jiacheng Sun, and Zhenguo Li. On the generalization of diffusion model, 2023.
[YZS+24] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications, 2024.
[ZC23] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponen- tial integrator. In The Eleventh International Conference on Learning Representations, 2023.
[ZYLLL24] Kaihong Zhang, Heqi Yin, Feng Liang, and Jingho Liu. Minimax optimality of score-based diffusion models: Beyond the density lower bound assumptions. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 60134-6017 18.. PMLR, 21-27 Jul 2024.
16

---

The appendix is organized as follows.
In Appendix A, we provide the proofs of the theoretical results presented in Section 3. In Appendix B, give some additional technical background, as well some omitted proofs, related to the generalization bounds discussed in Section 4.
Finally, in Appendix C, we provide the full details of our experimental setup, discuss some additional empirical results, and finally offer some final remarks regarding extensions to other transport-bact basee generative models.
A Omitted proofs of Section 3
Given fixed dataset (()) (Z Zn) ~ n00 we will frequently use the notation:
1 pnn n2
5z.
(11)
We also recall that denote Pr the density of X, with respect to the Lebesgue measure (where X, is initialized from (4) and fr its density with respect to the Gaussian measure yd. Similarly, we denote X (n)) the process following Equation (2) initialized from the empirical distribution P., For t >>0 0, we denote by 7( its density with respect to the Lebesgue measure and by z(() its density with respect to n Finally, F'ajo ajo is the density of X given X. with respect to the Lebesgue measure and Pijo its density with respect to 7d.
We start by technical lemma which is taken from the proof of Equation (11) in [Vinl IJ, which we reprove with our notations for the sake of completeness.
Lemma A.1. Consider a probability measure v on IRd, Only for this lemma, we denote by Ps the density with respect to 7tth of the process X, initialized from X, u. For any Borel- function up Rd -> Ihd and fixed time >> we have the following identity:
[(W(X.)) Vlog Pe x.))] IE [WW(X)), logDro (x)
E
In particular, this lemma can be written as:
f (xi)), V log pa (Xi))) du(2) = f EE (x(xi) V log (Xixa)) dr(z).
Proof. Let 7 ~ n. by Fisher's identity and the tower property for conditional expectation, we have:
E (v(x)) V Vogg. X {))} E (p E
X X X }
V log
E
(el (X
loo X xl ( o)) X
(()(X)),
X x {oo)]]
E
log
A.1 Proof of Theorem 3.1.
In this subsection, we present the proof of Theorem 3.1 The proof relies on classical computations on score functions I OAS23J. We start with the following lemma, which provides decomposition of the score approximation in terms of the denoising score matching loss.
Lemma A.2. For all 0 O, have ( (0) _g(n) (0) + S1 (Z()),) Cr, where Cr 2 0 is non-negative constant (independent of 0) defined by:
Cr Cr: :0:44 f E (o
V Voog P ( x110)||? d(u @ ()),
(12)
N-1 with X == T-1 EN-1 hdT-4
17

---

Proof. Let's recall that X₁ denotes solution of Equation (2) initialized with X ps, where is the data distribution. Let us recall the definition of the probability measure
h
Note that the support of À is bounded away from O, which justifies the derivations below. We expand the square and use Lemma A. to obtain:
(+)
E
st Xx) IV da(t)
/
19 41 v Vlog 11(] da(t)
E
se (t, Xx,
E K (t, Xx), V log (x1)) da(t)
4
/
so (1, X.) 19] + 4E V Voo pa 1) 11]) di(()
E
E ( (0 Xi); log 12) du(=)da(1)
= / R
|s0(1,, Xii 2V log
dp(2)dx(())
-4 4 / (J
(1)) de(z) E ( x1)"")))) da(e)
E
V Pog
= (R /
log pojo |1 du(=) E
v log Pe X (1²) ax(1),
The derivation above is identical to Lemma C.3 in [OAS23] and is a direct consequence of the celebrated result of [Vinl]].
Now, we note that (0) R R the denoising score matching loss. Therefore, we conclude the proof of Lemma A.2 by using the following lemma.
i (0) + (Z(m), 0) DSM (0) + (Z(")) 0) by definition of
Lemma A.3. We have the following identity:
CT
- - (f [
logg Pejo {i1=) ( E (1))))))())))
(()) dx(1)
We just need to show that Or > O. we see it by the following calculations based on Lemma A.I. We have:
Proof.
OT
/
2) |1] + /B v log Pojo X F1 du(=)) da(()
E
V log Pa
/
2
E log pe (x.),
v Vogiroo
f1=)) dn(=)dx(()
=
/
V
log Pr x
/
V P. X: |=)=)| du(z) dx(1)
2
E
V log Se X ,), v log Vl (xx) da(t)
log p x f12) | du(2) E [| dx(t).
E
log
V
This completes the proof of Lemma A.2.
An immediate consequence of Lemma A.2 that (0)) (0) & (D) + I (Z((), 0). Such a result consistent with the minimisation of LDSM made in practice.
18

---

Proof of Theorem 3.1.
Proof. By definition, we have for fixed (()) (Zi...., Zn) E (R.")" th that:
n
so(t, X22:) - 2V log Pogo X x (2.) / d2(1).
DSM
Therefore, we can apply Lemma A.1 to obtain:
a Σ n
-
(1, X 2, 11]
4E (x
DSM (0)
E
8p
i=1
( (oo(t. x²2 ) log Prjo
Xx 2. 2.)) da(t))
4E
1
sp (1, X k1) || 4E (
E
(so(t, XZ)), v log (m) (x{;1)1] dx(()
4E
(a(n) ESM
(0)
+
4
/
E v log Pa o X p. 12:) 1|0
- E Voog (1)
(x
11) Janco.
i=1
Thus, we have L (0)
Lia (0) + Öia) by definition of u.. We conclude by copying the
proof of Lemma A.3 to obtain that:
1
V log Pojo X2 12/2))1 - B. || (m))
(x (22211())
T
E
4
A.2
Omitted proofs of Section 3.2
In this subsection, we present the omitted proofs of Section 3.2.
Proof of Lemma 3.1.
Proof. Using the previous computations, we have (m) = I + II, with:
/
- f
v V Pe o (11z) 27 du(z)
) dd(()).
::
4E
V
4I
n
|2]
27
V log Ps (n) (X [17] dd(t).
4E
V log pe (x)
By Mchler's formula applied on the forward Ornstein- -hheenn-Uh process, we know that (=10(-12) is the density of N(o="2z -l-e-a c 212 Ia). Therefore, we have that for any probability measure vs:
4 Af E V Pr o0 { du())a((() 4 / E
+
x;
du(z)dA(t)
1
1-e-2
2
4
{ 1 - 0¹2 1|² du(z)dA(t)
E
(e21
2
(4sinh (t)) ||21|2 + a1- -e e -222) 21))dr(())))))
(621 - 1)²
/
[1ul12+ + e2t - - )ax()
d
4
21
19

---

with ||/|2 = Bowx ||= Therefore, we have:
I 4 / -0 (100111 2 |1a112) dd(t).
We have that 112,,11 - ||a112 [112] - - Zill. with
~ 1000. By
Assumption 3.1 we have that || & D2 almost surely. Therefore, by Hoeffdiingss inequality, we have:
o (112)POu2))
n Σ
2nc
1
< exp
n i=1
D1
from which we deduce that with probability at least 8 have:
log(1/5)
f
I 4D²
(axt)
2n
Finally, we remark that by definition of the relative densities and Fisher information, we have:
11 = / (1 V log
7.(0) (5))
2
/
p. (v) (1
log
pe (u)dd dA(t)
-4 / ( dA(().
This concludes the proof.
Proof of Lemma 3.2.
Proof. By definition, we have,
ciin =4 ch/==a/ F11IVlog log [12) Vogg logg!" (X:)))) & (),
where we recall that phn == n E Sz. By Lemma A.1, we have
ciu) =4 + ʃ Vol (fi2) log X )) M @ A)(2,t)
ElIl log p. (m)
(x+) v log 5e M @ t))
By discarding the negative term and taking the expectation, we obtain
4Zz( / /ELIVIOg
- logg (x;))) @ A)(2,0)
<
Cr,
where the last equality follows from the definition of p.. This concludes the proof.
Before proceeding to the proof of Proposition 3.1, we prove three intermediary lemmas.
First, we need a discretization lemma to control the measure À and the uniform measure on [h, 77.
Lemma A.4. Let us introduce A.g(m) (n) = [( (7 hnd). Then, we have:
N-1
( 1f" A & h (
k=
Proof. Let (Pi)ezo denote the Omstein-Uhlenbeck semigroup associated with Equation (2). It is known that the Fisher information I (uP(ha)) is continuous function of time for > 0 and is
20

---

decreasing along the Ornstein- OrnsteinUUllnbber Uhlenbeck semigroup. It can be seen for instance by noting that for t O.
dt I (uPil) = - drT2(log
where o, is the density of uP = P. w and I22 is the iterated "carré du champ" operator (BGL14, Section 5.7]. Therefore, we have the following comparison between the sum and the integral.
N-1 Σ (uPr h ChS(uPr-u 1h4 + km
N-
{(PPI-ua?""ds T-
k=0
N-2
T-le
< h (uPP
+
I (uPalp"
T-14+1
R=O
I'
-ho 1h4) +
I(uPol7))ds
T-IN-1
On the other hand, we have that:
N-1
N-1 (Pn Pr-ul()))
(Br Pr-uhh))
(
R=
T ((
T
N-1
T- ((
T-h
{(
T-
Combining these inequalities, we immediately obtain the desired result by noting that for all f >
we have pP,- = 7) t F., P. = >>(") (n) and by noting that T LN-1 h.
The next lemma provides uniform bound on various expected score norms appearing in our proofs.
Lemma A.S. Assume that p. has a support bounded by D. Consider positive numbers < < b. Then, almost surely for μ (or an and (n) s, we have:
1 E V (o
dl, b-a f
pb E Vloggt (m) (x
xm)) 2(() di < K2
with:
K2 := 1 22 D2 (²=
1----2 e-26 dt.
(13)
Note that our proof actually yields a stronger resuli where D is replaced by the order moment of μ (or Day respectively).
Proof. By Fisher's ' identity [Efri1] I] and Jensen's 's inequality, we have that:
E
v (log
||]
E
E
V log Prio
X a) /| ,
<
E
E
V log pupo
E
v
log
Pejo
21

---

By the proof of Lemma 3.1, we have that:
d
E v log
< e 21
E
We conclude by integrating over (a, b). We obtain similarly the formula for 7 (x)
Lemma A.6. Let 0 < a < b and assume that μ has compact support bounded by D. Let denote the random variable.
(x2) || 1 dt,
Y
E
V log
Pa
(14)
with Z ~ μ. Then, is sub-exponential with constant | 2 K1, with:
d
Ki
<
D>?
2a
Proof. Let j E N be an even integer. By Fisher's identity and the conditional Jensen's inequality, we have (as the proof the above lemma):
X X X +
E
V Vlogii (e [|]]]
=
E
V log Prjo
X X 0)
<
X,
c 21
Let us denote N(O, I), independent Xx. 0.
We have
E || [ogg
2t
2
< E
e
By Jensen's inequality, we have
V log Po (x)) < 320-1o-3'
c e- 2jt
E
E
|
(1 22-0-2
By applying again Jensen's 's inequality, we see that:
E
[[xa|]]]
E
E
E
<
where the last inequality follows from the moments-based characterization of subgaussian distribu- tions [Ver18, Proposition 2.5.2] and > 0 an absolute constant. We conclude that
V log (x.)) < 022Cip012 +2 (K3)(1?)
with
K2 S d
+ D22
c-22
Hence, by Jensen's 's inequality have:
pm) Vlog ( (2) (|)p dt < (K7)*MM m m"
Vm E No E
E V
By the moments-based characterization of sub-exponential random variables [Ver18, Proposition 2.7.1], we deduce that Y is sub-exponential with sub-exponential norm (see [Ver18]) ||Y||,, x K7.
22

---

The next lemma is a upper bound on the KL divergence between two distributions generated by Ornstein- Uhlenbeck processes a time > In this form, this lemma is taken from Proposition 23 in [ADR24], which the authors notice can be traced back to [Vi109]. For the sake of completeness and for a minor correction of the constants, we provide a short proof of this known result.
even show that the random variable Y above is sub- gaussian, but this would give a worse final iterate than using the above lemma. Moreover, the above proof technique still works under weaker assumptions on the data distributions, such sub- gaussian or sub-en expoeexonlt distribution.
Remark A.1. Under our assumption μ4,
Lemma A.7. Let p and v be two probability distributions on Teet and 0 to < T. Let Pt be the density of the OU process (2) initialized at Xo ~ and let qi be the density of the OU process (2) initialized Xg V. Then we have:
[ (pilg))dt & (L(ptol < (eto - 1)
1
W W2(Pto/2 (to/2/2:9to/22
Proof. For the purpose of this proof, let Lø := مه (x, Vo) the generator of the semigroup (Pi), of Equation (2). As P. and L. are self-adjoint with respect to 7h, have that Op. Lpt. where Pr pt/7d is the density of ps with respect to 7d. Then we easily see that 0 log pt T. Llog ft |/V log Fell (and similarly for q:).
By exploiting the chain rule and the integration by parts for L [CL17, Lemma 1.13), we have:
d KL(P)(0))= / log gt
1tt
iid
(llgg / (ologg V/ log di)l)) p.d7d + log Pt 9:
Lp
0+ / (Vlog pe, V log @ePpdry / Vo rell p.d
V log lel p. / (V Voggr V log ".
V log - V log qill p.dra
f (prla:).
By integrating this relation and using the non-negativity of the KL divergence we obtain:
[- I (pelg)dtt = KL(Ptal - KL lar) < KL (ta)
Let X, ~ a and Yo ~ We now use the fact that is the probability density of X, := x=:=c=Xx +
V1- - -1-e-22N(0, Ia) and qe is the probability density of Y; == e eYtY, + v1 - 2NO,, Ia). By the semigroup property and by joint convexity of the KL divergence [vH14] we have the known inequality (see also INDHR21]):
S" (pela))dt <
2(1 t Wz(Law(c-ta)) 10/2 (tLLLaawww////x,////// Xto/2)
(oo/2)
2 (c - 1) W2(Ho//2.Po/22)
Proof of Propusition 3.1.
Proof. By Lemma 3.1, we have that with probability at least 1 - 8 over (()) = (Zi,..., Bn) ~ vo::
1og(1/0)
f =d(() + + Σ
N-1
T < 4D2
(5(7)-1)7))
- (m)b))
2n
k=0
23

---

where we used the fact that hy = h. We now apply Lemma A.4 to obtain that with probability at least 1 over n we have
A < 4D2
(1/6)
21 dx(t) +
4h
17
4(T 4(T-h)
(A1 + A2),
2n
T
T
with, by adding and substracting:
log Pe (x,) 11
n
V log Pe (X,:) (2/mN) 1) dt,
A = T - h
E
Σ
E
i=1
A2
Σ
1 1
T
E
V pi 2 )/ (7)
V p( (x) /|² (m) dt.
1 T h
By Lemma A.6, we know that the random variable,
/' E Vlogm
(X 1) || 12 dl,
T- J
is sub-exponential with constant K} with respect to Z μ. with:
Ki e 24 D2.
Recall that (()) nn. Hence, by Bernstein's inequality [Ver18, Theorem 2.8.1], we have that:
2
on (A1 3 () < exp
-cn min
K
K?
where 0 is absolute constant. We deduce that with probability least ( over (()) urn, we have:
A S K? (v /108(1/0 + log(1/()
n
In the following, we fix (m)), so that all expectations have to be understood as conditioned on (z). We now turn our attention to Aq. We use the identity ||a]| - |]b]|² lla -albl bl| + 2(b, a - b), the Cauchy-Schwar inequality, and Lemma A.5, to get,
V log lo (X2:) - V Vlogg!" (M) (X dl
A2
< n
E
T
h
1=1
h
1
T
(1 - v log ( (. 11@ dt
+ 2K"
E
Vog
T - h hb
1=1
1
V log P p.(X((m)
V log 7. (1) (x (m))) 17 di
E
T h Jh
T- h ['D E Vlog P. (x(()) - Vlog P: ( |1² 11] dt
+ 2K"
1
(T (x)) ( +2K'
+
T-h h ['
1
T h
with:
T
-
R2
22 + - c 21
dt
(15)
J
1)2
hd
log (Tg()/h))
(16)
T
-
T h
24

---

By Lemma A.7 we obtain:
T
T A2 S T (Ka + +2K2v Krn)
with (:::=D++oo((T/n>---)) and Kn == KL (T (n)
h 7n))
From the second part of Lemma A.7,
we also deduce that:
T h
W
W2
W
K2
W2
A2 2T (ch - 1)
2K +
V
2T (ch - 1)
<
2Th
2h
with W = W2 : $
We conclude by union bound that with probability at least 1 26 over (n)) ~ nm we have:
1og(1/8)
h
log(1/8)
W + K2VhW
(n) T (D² +9))
K?
2n
T
n
Th
with:
K2 D2 + +2dlog(T/h) + hd,
K] =
d
D2, W: W: (
1/2 h/22
c- 2h
B
Omitted proofs and additional details on the generalization bounds
B.1 Noisy SGD
Consider the SGLD recursion, with a >> 0 a regularization parameter.
2Tk
0x+1 = (1 - amp 0x - mkğr +
V
Ek.
0o ~ Vo-
(17)
B
with Is unbiased estimate of the gradient of the empirical risk, and & ~ = N(0, Ta) indepen- dent of Ok- We denote by Ps the distribution of Gk-
The proof presented here is an adaptation of [MWZZ18] and is also inspired by the so-called half step technique used in [DC25]. The main difference with [MWZZ18] is to remove the need for a Lipschitz continuity assumption of the loss by using recently proposed PAC-Bayesian bound adapted to sub-Gaue GabsGassiinn losses [DS24]. The proof is based on classical arguments the noisy SGD literature, but present it for the sake of completeness.
Proof of Theorem 4.1
Proof. We consider a "prior" stochastic process defined as X6+1 = (1 xx + 2 EA with Xo
N(O, ).). Then it is clear that X₁ == N(0, la) with Ak E N, o (1 (1-an)) 2of + 3-1, Then we can see by recursion that Vk E N, σ Ba v2.
Thanks to the subgaussian assumption, we can apply Theorem 2.1 of [DS24] to obtain that with probability least 1 over ((a) p f0n we have:
KL N + log(3/8)
Eanpy
B x)) 0)
< 2
(18)
Now us fix some E and introduce (1) and (l) the distribution of 0c(() = (1 a07))0 7x9k + 2t8x and (t( := (1 ax + V2tkR. respectively, for E [0, m7.51111 Let ah (t)² be the variance (t). Then we have the following identity, which is generalization of
De- B-BBrrujj'' 's identity (see for instance [CCSW22] or our proof of Lemma A.7), for > 0:
dt (L(x)()) - (vn
25

---

By the logarithmic Sobolev inequality of (t),,), we have:
2 (l(px(0)=A(()) & (Fa)
(L((x(0)=A(()) < (t)2)²
where the last line follows from (x)))² & 2/((Ba) By integrating find that: KL(pe+1 [T++1) < " (T (m
We know apply the data processing inequality and the chain rule for KL divergence to get:
(L( (2)) (T /2)) < KI(Law(O,,, 1/2) (Xw(X (x+1/2))
< (L(Pu|TTR + Eu [KL(Law(O = x)) (X =x)]]
By joint convexity of KL divergence (see also [NDHR21), I), we now have:
| = - = m |1 12
Ezwpa [L([La((0+1//200 = Law Xx =x)]] < 21R
E
Therefore:
[L(p++1/T441) <e 312
3 (KI(PuTTT)) +
[15t611))
2
This implies that:
KL(pN|TN) 8 N-1 e (8N (
|| [[]]
with Sy = Ei=0 k-1 Ty. This implies the desired result by using Equation (18).
Case where a = O. In that case, we apply the data processing inequality and the chain rule for KL divergence to obtain that:
KL(pN(TN) < (I.
Ex-pe [KL(Law(0s+110) = (X = x))
B
Ezwpn
[|m=
4ml
k=
where the last inequality follows again from the joint convexity of the KL divergence. This leads to the desired result.
B.2 Background and additional results on topological complexities
In this section, we present some technical background on the topological complexities considered in Section 4.3 and also provide some additional topological generalization bounds for the score generalization gap.
B.2.1
Information- terms
In this subsection, we quickly define the information- thhorreti terms appearing in the topological bounds presented in Section 4.3. These terms come from the PAC-Bayesian theory on random sets introduced [DS24]. While several choices are possible, we focus in our work on the total mutual information I,o, which has the advantage of yielding high-probability bounds. Note however that it could be replaced with the usual mutual information in the case of expected bounds fro isntance.
Definition (Total mutual information). Consider two random variables X and Y on an arbitrary probability space and with values in measurable spaces (0xxFx)) The total mutual information between X and Y is defined as:
PB) Px 60 Py
(X,))
sup BEFx
In the context of learning theory, this quantity has been used in several studies [HcSKM22, DDS23J.
26

---

B.2.2 Definition of weighted lifetime sums
In this section, we quickly provide additional technical background the topological complexities mentioned in Section 4.
Consider finite set W (which is Section 4 we take to be the trajectory W(n) and pseudometric p on W. There exist two equivalent definitions of the weighted lifetime sums used in Section one using the notion of persistent homology (BCY18) and definition based on minimum spanning trees [Sch20], which we present here for the sake of simplicity. See [ADS* 24, BLGeS21, DV23] for additional details and connections to learning theory. We first introduce the following definition.
Definition 2 (Spanning tree). A tree T over W is connected acyclic (undirected) graph over W. We represent by a set of edges, where each edge is denoted {a, b) E 'T. The cost of an edge {a, b) is set to be p(a, b) and the cost of T is defined
8(T)=
Σ {a,j)TT
p(a, b).
We can now define the weighted-lifetinn lifettime sums (of order 1). Definition 3 (Weighted lifetime sums). The (I-)weighted lifetime sum of is the cost ((T) of spanning tree of W with minimal cost.
In the context of generalization bounds, several choices are possible for the choice of the pseu- dometric p [ADS 24, Section 3.1]. In our paper, we focus on the particular choice of the so-called data- data-dependent pseudometric [DDS23], which gives the most promising empirical re- sults in existing works. Given a dataset (()) (Zi,...: 2n) nr²n, we define the vectors Ez (w) (u, Z.)1<ten) E R". The data-dependent pseudometric then defined as:
P2mo (w, w') = Σ (()))
Egin) (w'))l,
(19)
where ||-|| is norm on R", which can be the el or le norm. The él is mostly used in [ADS+24] and it is also our choice in our work.
The quantity E, (Wirl) appearing in Theorem 4.2 defined as the weighted lifetime sum of W(n) for the pseudometric Pz(m).
B.3 Positive magnitude bounds
We start by defining the notion of positive magnitude. Magnitude was initially introduced in [Leil3] and positive magnitude a variant introduced by [ADS+24], which more suited to learning theory. While more general definition is possible, time stochastic optimizers considered in our study generate finite trajectories. In the following. let (W, p) be finite metric space in the above subsection. Given positive scale parameter >> 0, say that (XW) has magnitude [Mec15] is there exists vector B W - R (called weighting) such that:
focus here the case of finite sets, the discrete-
Va E W, Σ E r- 1.
BEW
This has been shown to be satisfied for the metric spaces considered in our study [Mec15, ADS 24). The positive magnitude then defined as:
PMag(r: W) = Σ B(a)+s
NEW
where 34 denotes the positive part of B. In all our work, we take to be the data-dependent pseudometric defined in Equation (19). Applying [ADS+24, Theorem 3.5], obtain the following bound on the score generalization gap.
27

---

Theorem B.1. Assume that the loss Ex 2) is uniformly bounded by B > 0 and that we have Fisher information I (1ln2)) < +00 and use constant step size hig = h. Then, with probability at least 1 - 8, we have for all 0 E Win) and all r > 0 that:
(,(1()), (())
<
2
B log PMag(Lr W(n) + +++ 3B1
I log (3)
where 1 = Lo Z() is total mutual information term and Kn A/TlBB
C Experimental Details
In this section we provide full details regarding the experimental setup. All experiments are imple- mented using Py PyTorch.
Forward process. We use the Oustein-Uhlenbeck (OU) process for the forward diffusion, also known as the variance- preservingg process [SSDK To characterize the conditional law Prjo of
X, given X o, define a +++ exp( 22)). The process admits the following reparameterization:
X Va(t)X o + WI-a(t)G; where G ~ La).
(20)
We adopt the cosine schedule introduced in [DN21] for both training and sampling. Specifically, we construct a discretization {4.) of [0, 1] with N equally spaced points, and define:
f(,) with f(t) = cos ār, = f(0)*
( s π m) 1 ++s
s = 0.008.
To ensure numerical stability, we truncate the schedule to [0, 1 c]. where is chosen such that 1- -a(Tx)/(((T))
0.999. This prevents divergence of the final time T and step size hv, and ensures that bounds remain finite, as recommended in [DN21]. The final time grid (1:) is then obtained by solving att;) atf,) for each
Training and e-parameterization. We adopt the e-parameterization introduced in [HJA20b], widely used in subsequent works [DN21, HS21, SH22, KAAL22b, EKB + 24]. This choice is motivated by practical considerations: improved numerical stability and faster convergence. The standard DSM loss exhibits high variance, resulting in noisier gradients and losses. We do not explore alternative approaches such as v-prediction [SH22].
The score model is parameterized as:
-26
st(, V1 al))(t)"
(21)
which is motivated by the identity 2V log 7' e (r)) -2(x - va(l)ao)/(1 - a()). together with the expression the noise term in (20).
= + vI- a(tG where (Gr)ogreT are i.i.d. samples from N(0, La). The training objective is the e-loss:
We define a simulation-free process (Xt)nstsT via X7 =
(O-DSM(O):- :- 1
(t. - -coll |z, 2, v(dt), where v = (fte)
E
(22)
For reference, the original DSM loss integrates with respect to À = T-Σ hN as in Section 2. Notably, at fixed timestep f, the e-loss equals the DSM loss up to a multiplicative factor w(t) = 1/(11 [HJA20b], which diverges near f. = O, causing instability and increased variance. Thus, minimizing the c-loss effectively corresponds to minimizing the DSM loss.
During training, for each datapoint in a batch, we sample a single timestep Gaussian variable G, to evaluate the stochastic objective in (22). For evaluation on the train and test datasets, to reduce the variance of the estimated losses, we sample 10 independent timesteps and 10 corresponding noise terms per datapoint, and average the resulting losses.
v and a single
28

---

Generative process. To sample from the trained model, we use the Suler-exppoentiic integrator described in (6).
Compute resources. Experiments are conducted using 8 NVIDIA A100 GPUs. training run of 100,000 steps MNIST takes approximately hours single GPU. Sampling 2,000 images with 500 reverse steps takes approximately 10 minutes on a single GPU. The VRAM consumption typically varies between 4-20GB depending on the chosen hyper-parameter configuration. The total computational budget for this work amounts to approximately 1,250 GPU hours.
C.1 Low-dimensional dataset
We provide additional details our low-dimensional experiments used to validate SGLD bounds in Section 4.2.
Mixture of Gaussian dataset. The dataset consists of mixture of nine Gaussian distributions in R.::
Σ uy NN,N TT,,,
(23)
where the weights are (w:) (0.01, 0.1, 0.3, 0.2, 0.02, 0.15, 0.02, 0.15, 0.05), and we fix = 0.05. The component means (14)==1 are sampled once uniformly at random from [-1, 1 We observed no signíficant difference in generative performance across seeds or across handcrafted choices, like arranging means in grid-like pattern.
Model architecture. We use neural network consisting of three time-conditioned MLP blocks with skip connections. Each block consists of two hidden layers of width 32. The input timestep t first processed through two fully connected layers of size 32 32, and then passed to each MLP block via an additional 32 x 32 transformation before being added (element-wise) to the intermediate representation in the block.
SGLD generalization bounds. The optimization is carried out using Stochastic Gradient Langevin Dynamics (SGLD) with momentum weight decay, using the torch sgld package. Models are trained for 100,000 steps using N 1000 forward timesteps. We vary the inverse temperature parameter (10 10 1010) (i.e., T = 1/8). We also sweep the step size, batch size, and dataset size, with batch size equal to the number of samples. Specifically:
Step size (2e-4, 5e-4, 1e-3, 2e 3,, 5e-3),
* Total number of samples(= batch size) E (512, 1024, 2048, 4096, 8192).
For each hyper-parameter configuration, we run experiments with different random seeds.
Evaluation. The number of steps at inference fixed to = 100, where all models are observed to perform optimally. To evaluate generative quality, we compute the Wasserstein-2 (WW) distance be- tween the generated and target data distributions. The squared W/2 distance between two distributions i and over R2 is given by:
/ 118-v1177dm,
(H, inf -EM(u..
1l2 dy).
where M(,, v) denotes the set of all couplings between j and u. In our case, we compute the empirical W distance between 25,000 generated samples and 25,000 real samples using the pyemd package [Las 17], with default settings.
C.2 Image data
We consider three image datasets to validate bounds in Section 4.3: MNIST, the butterflies dataset [WME09], and the flowers17 dataset [NZ06], simply referred to as flowers.
Model architecture. Our implementation relies on the U-Net architecture from [DN21], available httpt::// h/tttthb.bttttttt/ com/ wwt.//co..ooiiieeeeeoeee http///annnnmmrdddaaaaaaa http/mimpoovddeeee difssiffffoooooo and with configurations described in Table 1. The activation function is SiLU, and self-attention is applied at the specified resolutions. The diffusion time rescaled to lie in (0, 1) and encoded via the Transformer sinusoidal position embedding [VSP+]] 17].
29

---

Configuration
MNIST
butterflies
flowers
Input dimension
x 32 x 32
3 64 x 64 [4. 8, 16]
3 x 64 x 64
actn_resolutions
[2, 4]
[4, 8, 16]
channel_mun
[1, 2, 2, 2]
[1, 2, 2, 2, 4]
[1, 2, 4, 4]
model_ channels
32
128
64
num_res_blocks
2
2
num num_heads
4
4
4
dropout
0.0
0.0
0.0
Table I: U-Net architecture contigurations for each image dataset.
MNIST. Images are resized from 28 x 28 to 32 x 32. Each model is trained for 500,000 steps.
Butterflies and Flowers. All images are resized to 3 x 64 x 64. The butterflies dataset consists of 702 training images and 130 test images; the flowers dataset consists of 1, 1,020 training images (combining train and validation sets), and 340 test images. Each model is trained for 200,000 steps.
Evaluation. We use N 500 steps during sampling. We evaluate generative performance using the Fréchet Inception Distance (FID) [HRU+ 17J. For MNIST, we compute the FID using 2,048 generated images compared against 2,048 real images, once using the training set (train FID) and once using the test set (test FID). We refer to the latter simply as "FID" throughout our experiments. For the butterflies and flowers datasets, we match the number of generated images to the number of real images in the train/test splits.
ADAM generalization bounds. We use the Adam optimizer [KB17 17] for training. For each model, the time discretization during training corresponds to N 4000. We vary the learning rate and batch size across:
Learning rates: (5e-6, 1e-5, 2e-5, 1e-4, 2e-4),
Batch sizes: {4, 16, 64, 128).
To study the generalization properties of the trained score models, we compute topological bounds by monitoring their behavior during optimization over fixed subset of the training data of size min (dnatttset size, 3000). Moreover, we also sample and fix single noise term and single timestep per datapoint using the same random seed, computing loss terms on the same datapoint, timestep, noise term triplets across all configurations, obtaining comparable trajectories. For each iteration, we evaluate the per-subset c-loss and score loss, allowing us to track the evolution of local training dynamics. This procedure is repeated for 5,000 optimization steps over the train set starting from each fully trained model, where each train loss computation and optimization step is done as described in the introductory paragraphs of Appendix C.
The resulting trajectories are then analyzed and topological complexities (weighted lifetime sums and positive magnitude) are then computed using the procedures described in Appendix B.2. As is the case in [ADS* 24], we consider the a-weighted lifetime sums with a = 1. Regarding positive magnitude (denoted PMag(A W(n))), we make two choices, as briefly discussed in Section 4.3:
The first choice is to take À = ni, which is the thcoretical value suggested in [ADS +24].
It was also argued by these authors that small values of the scale parameter can yield good correlation with the generalization error. As they their study, we also report experiments using the value 10
For the butterflies and flowers dataset, we used the whole training trajectory to evaluate the data- dependent pseudometric (19). For the MNIST dataset, in order to reduce the storage and computational costs of this eperiment, we preselected subset of size 3000 of the training set and used it to estimate (19). This procedure standard in the literature and experiments have shown that it accurately estimates the topological complexity [DDS23, ADS 22J.
30

---

ADAM, 7 10 batch size 16
(b) ADAM, 10 batch size 128
Figure 4: butterflies samples after 200k training steps, 500 sampling steps.
Table 2: ADAM on flowers. Test/Train Denoising Score-Matching loss (()) R(A) complexity indicators and hyperparameters.
2(n) (0) vs.
(0)
R (0)
bx (II?I))
E² ('(w())
PMag(10 (w())
PMag( / W())
b
0.0773
0.02966
27.29
0.000100
0.0811
0.0128
0.087
564.7
42.88
4605
0.000100
16.00
0.0842
0.0262
0.082
453.8
28.
4290
0.000050
4,00
0.0851
558.5
41.98
0.000050
16.00
0,0859
0.0073
0.526
615,8
52.18
4698
0.000050
64.00
0,0865
0.0074
0.242
628.1
52.91
4712
0.000100
64.00
0.0894
0.0129
0,959
591.4
41.97
4625
0.000020
16.00
0.0948
0.0084
2,470
746.6
51.74
4835
0,000020
64.00
0.0959
0.0066
0.397
637.3
58.81
4731
0.000100
128.0
0.0972
0.0264
0.337
460.4
27.67
4302
0.000020
4.00
0.0266
1,270
27.42
12
0,1023
0.0065
0,932
667.2
57.14
4757
0.000050
128.0
13
0,1071
0.0146
3.690
683.2
40.22
4735
0.000010
16.00
840.7
0,000020
15
0.1134
0.0094
9,890
831.0
50.90
4879
0.000010
64.00
16
0.1362
0.0085
19.380
1005.0
55.01
4942
0.000010
128.0
C.3
Additional results and remarks
In this section, we provide additional experiments to complement the discussion Section 4. We also make several remarks possible extensions to other transport-based generative models.
Experiments on the flowers dataset. Table reports Train/Test DSM together with bx ?I E' (W("), and PMag(-) across learning rates and batch sizes for ADAM on flowers, complement- ing Fig. 3.
Experiments on the butterfly dataset. We provide some generated image grids for the butterfly dataset in Figure for visual comparison.
Experiments on the MNIST dataset. We also computed the complexity presented Sections 4.2 and 4.3 for the MNIST dataset. The results are reported in Figures 5 and 6.
Regarding the gradient norms-based bound studied Section 4.2, we observe a quite satisfying comelatiin with the generalization error, apart from some points in the figure. We see in Figure 6 that is related the value of the train loss, suggesting that the models need to reach certain threshold of convergence for gradient-basse based complexity to be more relevant to understand generalization.
31

---

The weighted lifetime sums E' yield a slightly more contrasted result. As we observed for the gradient bound, this behavior seems to be connected the train loss and, hence. the convergence of the model (see Figure 6). These observations are coherent with existing works on topological generalization bounds suggesting that they characterize geometric properties of local minima and, thus, the experiments require the models to reach such a local minimum [BLGGe211, DDS23, ADS 24]. As we are the first to evaluate these quantities for diffusion models, we observe that these conclusion seem to hold also in this case.
Regarding the positive magnitude, the observed correlation for PMag PM0 (W(()) is satisfying, even though the lack of convergence of certain experiments might be affecting the result. An interesting behavior can be observed for PMag( VT W(m)), where most points attain the maximum value of 5 103. This is known phenomenon that can happen with positive magnitude when the scale is not adapted and it is the reason that prompted the authors of [ADS * 24] to introduce PMag (00gg11022 R.W())). Overall, these experiments suggest that PMag(10 W(() might be the best complexity metric for more complex datasets, which is in line with the results of [ADS* 24, Table 1].
10
4000
bx
3000
10
Learning
100
2000
10-1
0.0000
0.0005
0.0010
0.0015
0.0000
0.0005
0.0010
0.0015
22
10
Learning
pelag(te
20
PMau
5e3
16
5a3
Sa3
0.0000
0.0005 Caneralization gap
0.0010
0.0015
0.0000 0.0005 0,0010 0.0015
Generalization gap
Figure 5: ADAM optimizer MNIST dataset. Score generalization gap vs. several complexity metrics: (( (top left), E (WW()) (top right), PMag(10 -.000....0 2.W(n)) (bottom left) and PMag( ( Win) (bottom right).
101
2.75 H
6x119933
3000
Train loza
2.65 10
10-2
2.55x10
0.0000
0.0005
0.0010 0.0015
5e3
2.75x10
5c3
PMagtto
20
Batch size
PMaglei
5e3
2.65 10
Train
5e3
106
64
5e3
1.-
128
0.0005
0.0010
0.0005 0.0010 deneralization gap
gap
Figure 6: Same experiment as Figure 5, except that the color bar represents the value of the train loss instrad of the learning rate.
32

---

Possible extensions beyond diffusion. We expect that similar generalization phenomena may be experimentally observed in other classes of generative models. particular, extensions more general continuous state spaces could be considered, such as the setting of bridge matching models [Pel23, LGL.22, LCBH+23), which admit more flexible structure between base/target distributions. Discrete state spaces could be an interesting avenue of research too [AJH+ CBB+22, LME24), especially in structured settings like the hypercube [PSO*25, BS25], where sharper convergence bounds have been obtained. Another direction could be generative models built upon alternative noise distributions, such as heavy-tailed distributions, which exhibit behaviors like robustness to mode collapse which could be tied to generalization abilities, [YPKL23] for the continuous time regime, or [SSD25, DSL22] for a discrete time regime. This would require advanced concentration tools, going beyond classical assumptions (sub-Gre Gaub-Ga etc.) distributions are unbounded or lack finite variance, complicating the analysis both A and generalization gap. Finally, another avenue could be generative processes based continuous-time Markov chains, beyond standard diffusion and score-matching, including. e.g., jump processes and piecewise deterministic dynamics [BSDB + 24, Bau25, BSS + 24]. We hope that such extensions will motivate the development of new theoretical tools tailored to broader generative modeling paradigms.
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, claim that we derive algorithm- and data-depedent gener- alization bounds for diffusion models, and that we support our findings with experiments. These claims are supported by main theoretical results in Sections to 4 and our experiments in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our theoretical results when we analyze the bounds that we obtain in Section 3.2, e.g., discussing the numerical behavior of the constants. We also discuss the limitations of our experimental results, particular results exhibiting noisier and more complex trends such as those relative to the MNIST dataset. We also discuss computational resources constraints, as the exploration of the hyper-parameter space prevents much larger-scale experiments. In addition to these discussions, we included limitation paragraph in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
34

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The proofs of all main results are presented in the appendix. The proofs are presented with full details and with the associated technical background and references make them clear the reader.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
We provide all necessary details to reproduce our experimental results, primar- ily in the appendix. This includes full descriptions of our experimental design choices, model architectures, datasets used, training and sampling procedures, optimization algorithms with corresponding hyperparameters, and the necessary details required implement the relevant terms in our theoretical bounds. Furthermore, we provide our exact code implementation.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
35

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide in the supplementary material the code necessary to run our exper- iments, that is training, evaluating, computing the quantities of interest, and reproducing the figures presented in the text. We have also included descriptive readme files, comments and instructions to reproduce our results. The datasets and neural network architectures used in our experiments are all publicly available. Finally, we will make our code publicly available upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps:/nnips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSebmiii ssi iesionPooicy for more details.
/nips. wttpcccccpcpcppeeececccccccccctt /
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide in the appendix all the details regarding our training and evaluation procedures, and precise deseriptions of the model architecture, optimization algorithms and chosen hyperparameters. This includes, for instance, the hyperparameters used for the SGLD algorithm and the ADAM algorithm, as explicitly listed in Figures and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
36

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We made a strong effort to ensure statistical significance within the limits of available computational resources. For the lower-dimensional experiments presented in Section 4.2, we report error bars computed across ten multiple random seeds, capturing variability due to initialization and stochastic optimization. These clearly indicated in the corresponding figures and properly discussed in the text. The higher-dimensional image experiments
challenging, as they require long training runs, since the models need to have converged relatively well. This is combined with extensive hyperparameter sweeps, which all the more so computationally demanding. We believe that our hyper-pan pper-parameter sweep strategy, resulting in clear signal trends, coupled with consistent results across datasets offers strong support for the results highlighted in our theory.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The details of the computational resources, mainly terms of GPUs, as well corresponding run time for training and evaluation, have been clearly indicated in the Appendix C.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
37

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http:///eerisppp.ee wwtppccwpppcccccccccccccccccccca hetpoiic/Ehhissuidellliiieee
Answer: [Yes]
Justification: All authors and the research conducted in this paper conform to the NeurIPS code of conduct.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We included a broader impact statement after the conclusion. As our work is mostly theoretical, we do not have any societal ethical concern associated to it.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: As research is mostly theoretical, we don't believe that there is risk of misuse of any part of our work.
Guidelines:
The answer NA means that the paper poses such risks.
38

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We make precise references the authors of the models and datasets that use, which are all publicly available.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
39

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https http:///eeuppp//ee/e wwwpe//oooeenne/////0ee/ for what should or should not be described.
40


Reviews:
[{"review_id": "aB6qd9UhgB", "reviewer": "Reviewer_6v73", "summary": "This paper analyzes in more detail the generalization error of learning a score function. They propose a new decomposition of the score error term that is commonly assumed to be small (under large hypothesis classes) and analyze each term of the decomposition separately, providing concentration bounds and ending with progress towards end-to-end analysis of the error of learning a score. In particular, the score error is broken up into 3 terms, one involving the inherent loss of the score function trained on the data, and the other two representing errors that occur from using an empirical dataset rather than the true distribution. These terms represent the generalization gap. These latter terms are then analyzed and bounded. They also show numerical experiments that support their theoretical claims.", "questions": "1. Can assumption 3.1 be weakened to be bounded with high probability? being strictly bounded is a little strong, considering Gaussians have exponentially small (but nonnegative) infinite tails. Perhaps some subgaussian assumption instead? \n2. Is Figure 3 just evidence that supports the theorem by [MWZZ]? Is there anything represented by the graphs that is meant to represent theory found in this paper? \n3. In general, is there anything in section 4 that is particularly novel to this paper?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe breakdown of the score error into three (somewhat) interpretable terms is great, especially since these terms can be bounded. The more careful analysis of the score matching error term allows one to more clearly understand what is going on with the error term. The writing is also clear and seems well-focused on conveying the intuitive meaning of the work.\n\nWeaknesses:\nAlthough the paper does bring up a new way in which to analyze the score error, the analysis of one of the terms is from previous results, and merely borne out in experiments in this work, and another term is (understandably) left as inherent. The third term is bounded using a (somewhat coarse, as admitted by the authors) concentration inequality, which seems to be the main theoretical contribution of the work. Still, the decomposition is novel in itself and serves as a useful tool for further analysis.", "was_revised": true, "final_justification": "My main initial concerns were that of novelty. Coming back to this submission a week later, and thinking more about the merits of the results, I am convinced that the framework presented and justified in the paper is worthy of an accept.", "rebuttal": "We thank the reviewer for their insightful feedback. We address below the raised concerns and questions.\n\n**Question 1**\n\nWe thank the reviewer for raising this important point. Indeed, in our work, we assume that the data distribution has compact support since it simplifies several technical derivations. \nAs the reviewer suggests, we expect that it is possible to weaken this assumption.\nIndeed, the compact support assumption is mainly used for two reasons in our proof: (i) to apply Hoeffding inequality in the proof of Lemma 3.1 and (ii) to obtain estimates of the moments of the norm of the score function. In both cases, it should be enough, at the cost of additional derivations, to only make assumptions on the growth of the moments of the data distribution, e.g., that it is  sub-Gaussian.\nIn the final version of the paper, we will add a discussion about this aspect.\n\n\n**Question 2**\n\nThe reviewer is right to notice that Figure 3 support the findings of [MWZZ] and others, **in the context of generative modeling** (notice that these bounds were developed for supervised learning), hence it serves as a sanity check to illustrate that the bounds are still meaningful in the generative modeling context. \n\nWe would like to underline that our work is the first, to our knowledge, to provide a sound theoretical framework to exploit these previous results (crafting the generalization error of Theorem 3.1 is not obvious) and exploit them in concrete experiments for diffusion models.\nBeyond this original connection, we believe that Figure 3 is necessary to assess that these generalization bounds have empirical relevance when applied to score-based generative models.\nIndeed, such experiments has not been done in the existing literature. \n\n\n**Question 3** and **Weaknesses**\n\n\nWe thank the reviewer for assessing the novelty of our proposed decomposition.\nThe key novelty of Section 4 is not in deriving entirely new bounds from scratch, but in showing that existing generalization results—originally developed for supervised learning—can be repurposed to analyze generative models. This repurposing is made possible by the new connection we establish in Section 3, which reduces the analysis of generation algorithms to the score generalization term $G_\\lambda$. To our knowledge, this is the first time such a transfer of theoretical tools from supervised learning to generative modeling has been made explicit. We believe this is a substantial contribution, as it opens up a rich and previously untapped body of results for use in the generative setting.\n\n\n\nThus, while Theorems 4.1 and 4.2 rely on existing results, Section 4 brings up original results in two distinct ways. \n\n*Theorems 4.1 and 4.2 draw links between DMs and statistical learning.* \nOur theorems should be thought as a continuation of the theoretical results of Section 3. Indeed, the conclusion of Section 3 is that we boiled down the problem of generalisation in DMs to a well-defined statistical learning problem (the generalization error of Theorem 3.1), where a loss function $\\ell$ is well-defined. Then, the results of Section 4 exploits this setting to involve the benefits of statistical learning (i.e, the dynamics of the considered optimization algorithm) for score-based diffusion models.\n\n*Theorems 4.1 and 4.2 allow novel, optimization-based, correlation measures.* \nTheorems 4.1 and 4.2 should also be considered as ways to obtain novel and practical correlation measures of the generalization ability of diffusion models. To our knowledge, it is the first time that the benefits of the optimization algorithm are involved as a measure of generalization of diffusion models.\nTo highlight this strength, we computed practical experiments showing the relevance of these correlation measures on several datasets. \n\nTo sum up, while Theorems 4.1 and 4.2 exploit existing generalization bounds, they should be thought simultaneously as the final development of our theoretical contributions (expanded in Section 3) and the first step towards practical experiments, by unveiling original diffusion models-tailored correlation measures."}, {"review_id": "qcH65u2Ty1", "reviewer": "Reviewer_x95P", "summary": "This paper provides a theoretical bound on the Kullback-Leibler (KL) divergence between the data distribution and the generated distribution of score-based generative models based on Ornstein-Uhlenbeck process.\nThe paper presents a key decomposition of an approximation error term of the score function, enabling further analysis of generalization performance based on empirical datasets and optimizers.\nUsing this decomposition, the paper derives a gradient-based generalization bound for stochastic gradient Langevin dynamics (SGLD) and a topological generalization bound for ADAM.\nExperiments on image datasets demonstrate a correlation between the generalization error and the derived upper bounds.", "questions": "1. Could you please clarify the connection between the presented theoretical results and the limitation (i) mentioned in L73 regarding practical classes of score estimators?\n2. Could you please address the errors in Weaknesses 2 and 3?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nThe paper is well-structured. The introduction clarifies the target issues of previous work and provides a clear overview of the theoretical results. The logical flow is easy to follow, and the purpose of each section is clearly defined. The limitations of the work are well-discussed.\nThe paper demonstrates the usefulness of Theorem 3.1 and Proposition 3.1 by extending algorithm and data-dependent generalization bound analyses based on these derived theorems.\nThe experimental results on image datasets support the presented theoretical claims.\n\n### Weaknesses\n1. In the introduction (around L73), the lack of consideration for practical model architectures such as UNet is raised as an issue of an existing approach. However, the algorithm-dependent generalization analysis in the paper focuses primarily on optimizers, and the effects of model architectures do not appear to be discussed.\n2. In Figures 4 and 5 in the Appendix, the vertical axes of the bottom right figures appear unusual. They repeat the same value as the tick labels.\n3. There are some minor typos and missing information. I have listed some of them below:\n    * There appears an unnecessary parenthesis ( within s(T-t, \\overleftarrow{X_t}) in (3).\n    * The order of arguments of s_{\\hat{\\theta}^{(n)}} in (6) seems to be swapped; it is supposed to be (s, t) instead of (t, s).\n    * Just above (6), t\\in[t_k, t_{k+1}] is supposed to be the subscript of (\\hat{X}_t^{(n)}).\n    * The definition of B(0,D) is missing in Assumption 3.1. It likely refers to the ball centered at 0 with radius D.\n    * \\sigma^2 around L669 does not appear to be introduced, and it should consistently be 2 throughout the paper.\n    * There is a word duplication: \"have have\" in L696.\n    * In the equations below L680, d\\nu(x) is supposed to be d\\nu(z).\n    * In L715 and L716, the paper states that Hölder's inequality was used to derive inequalities. However, it seems more likely that Jensen's inequality was used.\n    * \\zeta around L754 is supposed to be \\delta.", "was_revised": true, "final_justification": "During the rebuttal, the authors clarified that the presented theoretical results implicitly take into account model architectures, in addition to optimizers and data structures. The authors also stated that the formatting issues of Figures 4 and 5 would be addressed accordingly. Therefore, all issues I pointed out have been adequately addressed.", "rebuttal": "We thank the reviewer for their detailed review, which will help improve our work.\n\n**Weakness 1 and question 1**, **the algorithm-dependent generalization analysis in the paper focuses primarily on optimizers, and the effects of model architectures do not appear to be discussed.** \n\nThank you for this insightful remark. Indeed, our bounds (Theorems 4.1 and 4.2) have no explicit dependence on the model architecture, but an implicit one. \nMore precisely, our theory does not require explicit assumption on the structure of the model class, which makes it applicable to a broad range of architectures, even the ones that are used in practical applications, such as the UNet.  \nThe generality of this approach stems from the generalization bounds we used, derived from PAC-Bayes theory (Alquier 2024), which allow considering any architecture by relocating the intrinsic properties of the model within a KL divergence term, translated here by the gradient norms (Thm 4.1) and the topological complexity measures (Thm 4.2). These terms take implicitly into account the model architecture in our generalization bounds, as it is clear that the model architecture will have an impact on the gradient norms, for instance. \nIn the next version of the paper, we will add a precise discussion about this matter.\n\n**Weakness 2**\n\nThank you for catching this issue regarding the axis of some figures in the appendix, we will correct it in the final version of the paper. This formatting issue is due to the fact that in these experiments, the positive magnitude with scale $\\sqrt{n}$ saturates to its maximum value, so all points on the y axis have approximately the same value. The fact that the positive magnitude can saturate when the scale parameter is too high has been documented in previous works, see for instance [ADS+24] and references therein. We reported these bottom-right figures for the sake of completeness; they do not yield significant experimental insights.\n\n\n**Weakness 3** and **Question 2**\n\nWe thank the reviewer for pointing out several typos, we will correct all of them in the final version of the paper. Here are a few additional comments.\n\n- The notation $B(0,D)$ indeed refers to the ball of radius $D$, we will add this definition to the assumption.\n- L669 $\\sigma$ is indeed a notation for $\\sqrt{2}$ it is a typo and will be replaced by $\\sqrt{2}$.\n- L715 and L716, by Hölder's inequality we mean that we use the inequality $(a + b)^j \\leq 2^{j-1}(a^j + b^j)$, which is also a consequence of Jensen's inequality. We will clarify it.\n\nReference: Alquier 2024 User-friendly Introduction to PAC-Bayes Bounds"}, {"review_id": "aRRNp8sYOW", "reviewer": "Reviewer_Bifm", "summary": "This paper delves into the statistical analysis of diffusion models and is the first to take the Dirac sampling from the dataset and optimization generalization error into account. They further conduct experiments to show the effect of optimization dynamics on the generalization ability of the DMs.", "questions": "See weakness", "limitations": "no", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n1. The mathematical derivation is rigorous.\n2. The experiments and figures are illustrative to show the effect of optimization dynamics on the generalization ability of DMs.\n\n**Weaknesses**\n1. The generalization analysis of the optimization algorithms like SGLD and ADAM directly follows the previous works, but only adopts them into the diffusion models’ score matching analysis.\n2. I am not sure if the ‘Dirac sampling from data distribution’ is both the source of the data-dependent constant and the score generalization gap in your analysis. I think a more heuristic explanation of the source of the different error parts presented in your analysis is needed.", "was_revised": true, "final_justification": "As mentioned in the discuss phase, I believe this work benefits both academic and industry in learning the generalization bound of diffusion model.", "rebuttal": "We thank the reviewer for their feedback. We believe that we have addressed all the raised issues. We hope that the reviewer could reconsider their score if they believe their concerns are resolved.  \n\n**Weakness 1**\nWe emphasize that the contributions of Section 4 are more diverse than direct adaptations of existing results, let us develop this below.\n\nThe key contribution of our work is showing that these results, which were originally developed in the context of supervised learning, can be meaningfully applied to the analysis of diffusion models through the lens of score matching. This adaptation is enabled by our theoretical reduction (Theorem 3.1), which bridges the gap between generative modeling and the generalization of score functions. While the bounds themselves are not new, their application in the generative setting is, and we believe this is an important and nontrivial contribution. Hence, our results are original within this context.\n\nTo be more precise, in Theorem 3.1 (our main decomposition), we unveil the impact of three terms. The main theoretical contribution of this work regards the decomposition itself and our bound on the third term (the data-dependent constant), most of our theoretical efforts are focused on this part. Thus, while Theorems 4.1 and 4.2 rely on existing results, Section 4 brings up original results in two distinct ways. \n\n*Theorems 4.1 and 4.2 draw links between diffusion models and statistical learning.* \nOur theorems should be thought as a continuation of the theoretical results of Section 3. Indeed, the conclusion of Section 3 is that we boiled down the problem of score approximation in diffusion models to a well-defined statistical learning problem (the generalization error of Theorem 3.1), where a loss function $\\ell$ is well-defined. Then, the results of Section 4 exploits this setting to involve the benefits of statistical learning (i.e. the dynamics of the considered optimization algorithm) for score-based diffusion models.\n\n*Theorems 4.1 and 4.2 allow novel optimization-based correlation measures.*\nTheorems 4.1 and 4.2 should also be considered as ways to obtain novel and practical correlation measures of the generalization ability of diffusion models. To our knowledge, it is the first time that the benefits of the optimization algorithm are involved as a measure of generalization of diffusion models. We insist that to highlight this strength, we computed practical experiments showing the relevance of these correlation measures on several datasets. \n\nTo sum up, while Theorems 4.1 and 4.2 exploit existing generalization bounds, they should be thought simultaneously as the final development of our theoretical contributions (expanded in Section 3) and the first step towards practical experiments, by unveiling original correlation measures that are tailored for diffusion models.\n\n\n**Weakness 2**\n\nAs the reviewer mentions, our theory makes use of a finite dataset sampled i.i.d. from the data distribution (hence, a Dirac sampling from the data distribution). \nThis is a classical setting in learning theory, as in practice, models are trained on a finite dataset. \nAs it can be seen in Theorem 3.1, this empirical distribution of the dataset plays an important role in every term of the proposed decomposition of the score approximation error, as the reviewer suggested.\nLet us make it more clear by providing a more intuitive explanation of these terms.\n\nThe first term (empirical explicit score matching loss) corresponds to the quantity that is being minimized by the learning algorithm on the dataset, and it is the lower bound of the empirical denoising score-matching loss.\n\nThe third term represents (the data-dependent constant) a statistical bias that only depends on the data distribution and the dataset, i.e., not on the score network.\nTherefore, the existence of this term is a consequence of the Dirac sampling of the data distribution.\nIt is a characteristic of the learning problem and providing an upper bound on this term is another main contribution of this work.\n\nFinally, the remaining term (second term) is the generalization error associated with the denoising loss.\nAgain, this term is non-trivial because of the presence of the empirical distribution of the dataset in the optimization process. \nThis allows us to exploit the rich literature on generalization bounds and to apply, for the first time, classical generalization bounds to diffusion models. Note that we used as our examples existing bounds for SGLD and ADAM, but other generalization bounds would be applicable to our framework too, hence, highlighting its generality.\n\nIn the final version of the paper, we will add more details on the interpretation of these terms, as asked  by the reviewer."}, {"review_id": "iEWsCEqudM", "reviewer": "Reviewer_TFd7", "summary": "This paper analyzes both the sample complexity (data-dependent) and the algorithmic generalization ability (algorithm-dependent) of the score-matching error that arises when training score-based generative models using practical optimization algorithms. The theoretical analysis investigates the generalization behavior of using SGLD and ADAM to solve the empirical score-matching objective, offering insights into how hyperparameters and topological complexity measures influence the generalization error in the context of training score-based diffusion models.", "questions": "**Q1:**  \nIn Theorem 4.1, the loss function $\\ell_{\\lambda}(w, Z)$ is assumed to be $\\tau^2$-subgaussian. Is this assumption directly implied by Assumptions 2.1 and 3.1? Similarly, Theorem 4.2 assumes that $\\ell_{\\lambda}(\\theta, z)$ is uniformly bounded with respect to both $\\theta$ and $z$. Does this automatically follow from Assumptions 2.1 and 3.1?\n\n---\n\n**Q2:**  \nHow should we interpret the three topological complexity measures introduced in Section 4.3 in the context of score-based diffusion models? Do these quantities provide practical guidance for model training?\n\n---\n\n**Q3:**  \nFigure 1 suggests that using a large batch size makes training less sensitive to the choice of learning rate and also improves generation quality. Can the theoretical results offer any insight into this empirical observation?\n\n---\n\n**Q4:**  \nBeyond the choice of training hyperparameters, the noise scheduling in the forward process [1] is known to significantly affect the performance of score-based generative models. Is it possible to incorporate the scheduling of the forward process into the current theoretical framework to offer guidance on how to choose or optimize the noise schedule?\n\n---\n\n**Q5:**  \nThe results in Theorems 4.1 and 4.2 appear to be independent of model complexity. Could the authors comment on why this is the case? Is it correct to interpret that model complexity only influences the empirical explicit score-matching loss (i.e., the first term in the error decomposition in Theorem 3.1)?\n\n---\n\n**Q6:**  \nCould the authors provide additional numerical results illustrating how the empirical explicit score-matching loss depends on the hyperparameters and the other complexity measures introduced in the paper? For instance, similar to Figures 2 and 3, but tracking the score-matching loss instead of the generalization gap.\n\n[1] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is clearly written and provides a helpful overview of prior work, effectively situating its contributions within the existing literature.\n\n2. The theoretical results are well presented and, to the best of my knowledge, offer novel results in the context of score-based diffusion models.\n\n---\n\n**Weaknesses:**\n\n1. Based on my understanding, Theorems 4.1 and 4.2 are adapted from existing results. However, the required assumptions are not explicitly verified or discussed in sufficient detail.\n\n2. The topological complexity quantities introduced in Section 4.3 are not intuitive, and the paper does not provide adequate interpretation of these terms within the context of diffusion models. It would also be helpful to have some discussions on how the result differs from standard settings where ADAM is used to minimize generic training objectives.\n\n3. The theoretical findings offer limited insight into the empirical success of score-based diffusion models or practical guidance for their training. \n\nPlease see the questions related to above three points in the Question section.", "was_revised": true, "final_justification": "I have read all the reviewers’ comments and the corresponding replies from the authors (including my own). In my view, most of the points I raised have been satisfactorily addressed. I find this paper to be theoretically well‑grounded, and I believe the proposed decomposition of the score‑matching error will be of interest to the community. Accordingly, I recommend acceptance.", "rebuttal": "We thank the reviewer for their insightful comments. In the following, we address all the remarks and questions.\n\n**Weakness 1** and **Q1** \n\nWe emphasize that Theorems 4.1 and 4.2 are not only consequences of existing results, but also from our theory, developed in Section 3. Indeed, the conclusion of Section 3 is that we boiled down the problem of score approximation to a well-defined statistical learning problem (the generalization error of Theorem 3.1), where a loss function $\\ell$ is well-defined. Then, our theorems should be thought as a continuation of Section 3 by exploiting this setting to involve the benefits of statistical learning (i.e. the dynamics of the considered optimization algorithm) for score-based diffusion models. \n\nWe thank the reviewer for the question of satisfying the assumptions of Theorems 4.1 and 4.2. Let us first note that the sub-Gaussian or bounded loss assumptions are very classical in the learning theory literature [HNK+20,MWZZ18,NHD+19].\nVerifying these assumptions will depend simultaneously on the *dataset* and the *architecture* of the score network, as they refer to the denoising score-matching loss. \nWhile it might not be a direct consequence of the assumptions required by our theory in Section 3, we expect that these conditions on the denoising score matching loss can be fulfilled in simple settings, like image datasets. \nIn the final version of the paper, we will add a discussion on satisfying these assumptions for particular data structures and leave a more precise understanding of this matter to future work.\n\n**Weakness 2** and **Q2**\n\nThank you for this insightful remark. These topological complexities have been used in several previous works (see [ADS+24] and references therein). They intuitively represent different notions of ``complexity'' of the trajectory of the optimizer (eg, SGD, ADAM, ...) near a local minimum, as is briefly explained in the appendix. They essentially compute the \"intrinsic dimension\" of the local minimum found by the optimization algorithm (by looking at the topological properties of the trajectory), hence, they are expected to perform better than bounds that are based on the parameter count, which is always larger than the intrinsic dimension. \n\nIn the final version of the paper, we will use the additional space to provide a more intuitive explanation of the meaning of this quantity.\n\nConcerning how these quantities should be interpreted within the context of diffusion models, let us detail this.\nIntuitively, the trajectory of the optimizer should be more complex when the learning task is more difficult. In the context of diffusion models, it means that several parameters should affect it like, for instance, the discretization step of the forward process simulation, as the loss is more sensitive for small times. \nIn the final version, we will add a discussion about these aspects. Thank you for raising this and help us improving the quality of our work.\n\n**Weakness 3**\n\nOne of our goal is to relate the performance of diffusion models to the optimization dynamics of the score network. For instance, this allows us to connect (part of) the score estimation error to the norms of the gradients encountered during training. To the best of our knowledge, such theoretical observations are new in the context of diffusion models.\nTherefore, we believe that this provides insights towards better training of diffusion models. For instance, Theorem 4.1 is supporting the idea that the generalization error of diffusion models should improve when the model converges to a flat minimum in the parameter space, as it is commonly found in the supervised learning literature [1,2].\n\n**Q3**\n\nThe batch size (as well as some other hyperparameters) does not appear explicitly in our theoretical contributions. \nHowever, note that these hyperparameters appear implicitly through the complexity measures that we use in Theorems 4.1 and 4.2 (through the gradient norms and the topological complexities). Figure 1 reports the Wasserstein metric and the FID calculated on the test set. \nWe see in Figure 1 that both metrics decrease, stabilize and then increase for higher learning rates. \nThis illustrates the existence of a trade-off, in the sense that the test error depends on both the optimization error and the generalization error, whose analysis is not the aim of this paper.\nFinally, Figure 1 was obtained using very simple models and datasets and its purpose is to illustrate that the learning algorithm affects the test performance of diffusion models, hence, justifying our approach and our main decomposition.\n\n**Q4**\n\nIncorporating the noise schedule is definitely an interesting suggestion by the reviewer.\n\nIn our setting and under our assumptions (especially the finite Fisher information), we believe that it is non trivial to explicitly factor the noise schedule in our contributions and we leave this fr future work.\nHowever, the reviewer is right to mention that the noise schedule is known to affect the performance.\nIn particular, as the noise schedule affects the properties of the loss function, should have an impact on the optimization dynamics and, hence, on the generalization error term (i.e., the second term $G_\\lambda$ in our decomposition). Therefore, the effect of the noise schedule is implicitly taken into account by the complexity measures studied in Section 4.\nIn the final version, we will add a discussion about this remark from the reviewer.\n\n**Q5**\n\nIt has been a long standing topic in learning theory that what is a good notion of \"model complexity\" for neural networks. It has been shown in numerous works that the good notion is definitely not the number of parameters of the model. \n\nWithin this context, the results in Section 4 suggest two notions of model complexity, that are implicitly measured by the quantities appearing in Theorems 4.1 (gradient norm) and 4.2 (topological complexity). \nMoreover, our framework is compatible with more classical measures of the complexity of the model class, such as the Rademacher complexity, so that they could be incorporated in our analysis of the second term of the decomposition (the generalization error). However, we believe that such complexity metric would have little empirical interest but we are open to any suggestion of the reviewer regarding other pertinent complexity metrics.\nTo sum up, one of the goals of our paper is to introduce a notion of model complexity through the analysis of the second term on our decomposition. Our theoretical contributions allow us to do that by upper bounding the other term in the decomposition.\n\n**Q6**\n\nWe thank the reviewer for the opportunity to further clarify the precise role of the complexity measures introduced in our work and their relationship with the optimization process.\n\nBecause of a recent update to Neurips guidelines, we are not allowed to include figures in our rebuttal and have enough characters for only one table. However, we have produced additional tables to satisfy the reviewer's request, tracking the denoising score-matching (DSM) loss against our complexity measures. Indeed, the DSM loss is equal to the empirical explicit score-matching (ESM) loss up to a constant independent of model parameters. We will also include these experiments, along with more figures, in the final version of the paper.\n\nAcross these experiments, contrary to the generalization gap, we observe no simple relationship between the train ESM loss and our complexity measures, as clearly exemplified in Table 1, where a low complexity value is observed concurrently with high and low train loss values. In Table 1, where all hyperparameters achieve a similarly low ESM loss, we see non-trivial generalization patterns showing a clear positive correlation with our topological or gradient norms-based complexity measures. This confirms that we have not collected spurious observations from trivial regimes and that our complexity measures are relevant quantities\n\n| Test DSM | Train DSM | Eval Gradient Norms Bs | Eval E Alpha | Eval Positive Magnitude Small | Eval Positive Magnitude Sqrt N | Optim Lr | Training Batch Size |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 0.0773 | 0.0296 | 0.042 | 480.8 | 27.29 | 4396 | 0.000100 | 4.00 |\n| 0.0811 | 0.0128 | 0.087 | 564.7 | 42.88 | 4605 | 0.000100 | 16.00 |\n| 0.0842 | 0.0262 | 0.082 | 453.8 | 28.18 | 4290 | 0.000050 | 4.00 |\n| 0.0851 | 0.0126 | 0.207 | 558.5 | 41.98 | 4573 | 0.000050 | 16.00 |\n| 0.0859 | 0.0073 | 0.526 | 615.8 | 52.18 | 4698 | 0.000050 | 64.00 |\n| 0.0865 | 0.0074 | 0.242 | 628.1 | 52.91 | 4712 | 0.000100 | 64.00 |\n| 0.0894 | 0.0129 | 0.959 | 591.4 | 41.97 | 4625 | 0.000020 | 16.00 |\n| 0.0948 | 0.0084 | 2.47 | 746.6 | 51.74 | 4835 | 0.000020 | 64.00 |\n| 0.0959 | 0.0066 | 0.397 | 637.3 | 58.81 | 4731 | 0.000100 | 128.0 |\n| 0.0972 | 0.0264 | 0.337 | 460.4 | 27.67 | 4302 | 0.000020 | 4.00 |\n| 0.0998 | 0.0266 | 1.27 | 497.6 | 27.42 | 4391 | 0.000010 | 4.00 |\n| 0.1023 | 0.0065 | 0.932 | 667.2 | 57.14 | 4757 | 0.000050 | 128.0 |\n| 0.1071 | 0.0146 | 3.69 | 683.2 | 40.22 | 4735 | 0.000010 | 16.00 |\n| 0.1116 | 0.0075 | 4.22 | 840.7 | 57.32 | 4894 | 0.000020 | 128.0 |\n| 0.1134 | 0.0094 | 9.89 | 831.0 | 50.90 | 4879 | 0.000010 | 64.00 |\n| 0.1362 | 0.0085 | 19.38 | 1005 | 55.01 | 4942 | 0.000010 | 128.0 |\n\nTable 1: ADAM optimizer on the flowers dataset. Train/test Denoising Score-Matching loss (DSM) vs several complexity metrics and corresponding learning rate and batch size. \n\n[1] Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization. In Conference on Neural Information Processing Systems, 2023.\n\n[2] Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos J. Storkey. Three Factors Influencing Minima in SGD. arXiv, abs/1711.04623, 2017."}]

Author rebuttal:

Meta-review: This paper considers an quantitative bound of the generalization error of score learning in diffusion model. The key contribution is a refined analysis that is algorithmic- and data-dependent, which is tighter than traditional results that are largely approximation theory based and therefore naturally generic and loose. While training aspect is touched upon, which is a highlight, this paper however overlooked existing works on quantitative bounds of score training dynamics, such as [Evaluating the Design Space of Diffusion-Based Generative Models] and [A precise asymptotic analysis of learning diffusion models: theory and insights]. Along the same line, most reviewers would like to see a clearer clarification on what parts of the paper are based on existing results and what are truly innovative. Nevertheless, all of us agree that the merits of this work outweigh its imperfection, and I'm happy to recommend acceptance. Meanwhile, I strongly recommend the authors to take the reviewers and AC's comments into consideration while preparing for a revised/final version.
Decision: Accept (poster)