Evaluating the design space of diffusion-based generative models
Authors: Yuqing Wang
Ye He
Molei Tao
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion-based models
denoising score matching
convergence of neural network training
generation error analysis
noising distribution
loss weighting
time schedule
variance schedule
Primary area: diffusion_based_models

Abstract: Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. In addition, a refined sampling error analysis for variance exploding models is also provided. The combination of these two results yields a full error analysis, which elucidates (again, but this time theoretically) how to design the training and sampling processes for effective generation. For instance, our theory implies a preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in [Karras et al., 2022]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design in [Song et al., 2021] is more preferable, but when it is less trained, the design in [Karras et al., 2022] becomes more preferable.

Evaluating the design space of diffusion-based generative models
Yuqing Wang Simons Institute
Ye He School of Mathematics Georgia Institute Technology yhe3670gatech, edu
University of California, Berkeley yq wang@berkeley. edu
Molei Tao
School of Mathematics Georgia Institute of Technology mtao@gatech. edu
Abstract
Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to certain accuracy, and then this priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts non-asymptotic vergence analysis of denoising score matching under gradient descent. In addition, refined sampling eтrоr analysis for variance exploding models is also provided. The combination of these two results yields full error analysis, which elucidates (again, but this time theoretically) how design the training and sampling pro- cesses for effective generation. For instance, our theory implies preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in Karras et al. [30]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design Song et al. [46 is more preferable, but when it is less trained, the design in Karras
al. 30
becomes more preferable.
1 Introduction
Diffusion models became a very popular generative modeling approach in various domains, including
computer vision 20
27 28 38 51] natural language processing 6 34
377, various modeling
tasks 15, 41 55
and medical, biological, chemical and physical applications 3
49 23
[66
(see more surveys in 14). Karras al. (30) provided a unified empirical understanding of
the derivations of model parameters, leading to new state-of-the-ar performance. Karras et al. [3I)
further upgraded the model design by revamping the network architectures and replacing the weights of the network with exponential moving average. As diffusion models gain wider usage, efforts to understand and enhance their generation capability become increasingly meaningful.
In fact, rapidly increasing number of theoretical works already analyzed various aspects of diffusion 9
models 32 52 16
44 25). Among them, a majority [3 19 52
12 I8
focus on sampling/nfferene; more precisely, they assume the score error is within certain accuracy threshold (i.e. the score function well trained some sense), and analyze the discrepancy between the distribution of the generated samples and the true one. Meanwhile, there are handful of results 4 (55 that aim at understanding different facets of the training process. See more detailed discussions of existing theoretical works in Section I.1
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, as indicated in Karras et al. [30,, the performance of diffusion models also relies on the interaction between design components in both training and sampling, such as the noise distribution, weighting, time and variance schedules, etc. While focusing individually on either the training or generation process provides valuable insights, a holistic quantification of the actual generation capa- bility can only be obtained when both processes are considered altogether. Therefore, motivated by obtaining deeper theoretical understanding of how to maximize the performance of diffission models, this paper aims at establishing full generation error analysis, combining both the optimization and sampling processes, to partially investigate the design space of diffusion models.
More precisely, we focus on the variance exploding setting [46]. which is also the foundation of continuous forward dynamics in Karras et al. [30]. Our main contributions are summarized follows:
For denoising score matching objective, we establish the exponential convergence of its gradient descent training dynamics (Theorem D. We develop a new method for proving key lower bound
of gradient under the semi-smoothness framework 1I 35 57 (8).
We extend the sampling error analysis in [ to the variance exploding case (Theorem 2 under only the finite second moment assumption (Assumption B) of the data distribution. Our result applies to various variance and time schedules, and implies a sharp almost linear complexity in terms of data dimension under optimal time schedule.
We conduct a full eπror analysis of diffusion models, combining training and sampling (Theorem 3}
We qualitatively derive the theory for choosing the noise distribution and weighting in the training objective, which coincides with Karras et al. (30] (Section[4.) More precisely, our theory implies that the optimal rate is obtained when the total weighting exhibits similar "bell-shaped" pattern
used in Karras et al. [30]
We develop theory of choosing time and variance schedules based on both training and sampling (Section 4.2). Indeed, when the score error dominates, i.e., the neural network less trained and not very close to the true score, polynomial schedule [30] ensures smaller error; when sampling error dominates, i... the score function is well approximated, exponential schedule [46] is preferred.
Conclusions and limitations are in Appendix A
Fastst conveganoo
"Bell-shaped" weighning
EDM
1.1 Related works
Training
Less Inszed
Training
Sampling. There has been significant progress in quantifying the sampling error of the gener- ation process of diffusion models, assuming the
Full ereer analysis
Polyromial schodule
Practice
Sampling
Sampling
Exponential
score function is already approximated within
schedule
ct.
Figure 1: Structure of this paper.
certain accuracy. Most existing works [e.g.,
16 12, g focused on the variance preserving (VP) SDEs, whose discretizations correspond to DDPM. For example, Benton et al. g is one of the latest results for the VPSDE-based diffusion models, and it only needs a very mild assumption: the data distribution has finite second moment. The iteration complexity is shown to be almost linear in the data dimension and polynomial in the inverse accuracy, under exponential time schedule. However, a limited amount of works [32 24 [54] analyzed the variance exploding (VE) SDEs, whose discretizations correspond to Score Matching with Langevin dynamics (SMLD) 45 46). To our best knowledge, Yang al. (54 obtained the best result so far for VE assuming the data distribution has bounded support: the iteration complexity is polynomial in the data dimension and the inverse accuracy, under the uniform time schedule. In contrast, our work only assumed that the data dis- tribution has finite second moment, and by extending the stochastic localization approach in [8] to VESDE, we obtain an iteration complexity that is polynomial in the data dimension and the inverse accuracy, under more general time schedules as well. Note the improved complexity in terms of the inverse accuracy and the data dimension dependencies; in fact, under the exponential time schedule, our complexity is almost linear in the data dimension, which recovers the state-of-the ofthe----he-rttt result for VaSDE-based diffusion models.
Training, To our best knowledge, the only works that quantify the training process of the diffusion models are Shah et al. 44 and Han al. (25). Shah et al. (4 employed the DDPM formulation and considered data distributions as mixtures of two spherical Gaussians with various scales of separation, together with K spherical Gaussians with a warm start. Then the score function can be analytically solved, and they modeled it in teacher-student framework solved by gradient descent. They also provided the sample complexity bound under these specific settings. contrast, our results work for general data distributions for which the true score is unknown, and training analysis is combined with

---

sampling analysis. Han et al. [25] considered the GD training of a two-layer ReLU neural network with the last layer fixed, and used the neural tangent kernel (NTK) approach to establish a first result on generalization error. They uniformly sampled the time points in the training objective, assumed that the Gram matrix of the kernel is away from (implying a lower bound on the gradient), and lacked a detailed non-asymptotic characterization of the training process. In contrast, we use the deep ReLU network with L layers trained by GD and prove instead of assuming that the gradient is lower bounded by the objective function. Moreover, we obtain non-asymptotic bound for the optimization error, and our bound is valid for general time and variance schedules, which allows us to obtain a full error analysis.
Convergence of neural networks training. The convergence analysis of neural networks under gradient descent has been a longstanding challenge and has been developed into an extensive field. Here we will only focus on results mostly related to the techniques used in this paper. One line of them is approaches directly based on neural tangent kernel (NTK) [22 21 5
36). However, existing works in this direction focus more on either scalar output, or vector output but with only one layer trained under two-layer networks, which is insufficient for diffusion models. Another line of research also considers overparameterized models in regime analogous NTK, though not necessarily explicitly resorting to kernlls. Instead, it directly quantifies the lower bound of the
gradient [T 35 2 57 58 and uses a semi-smoothnes, property to prove exponential convergence. Our results align with the latter line, but we develop a new method for proving the lower bound of the gradient and adopt assumptions that are closer to the setting of diffusion models. See more discussions in Section 3.1
1.2 Notations
We denote || to be the f norm for both vectors and matrices, and || ||r to be the Frobenius norm. For the discrete time points, we use f denote the time point for forward dynamics and f. for backward dynamics. For the order of terms, we follow the theoretical computer science convention to use (-), (()) We also denote f $ g if f s Cg for some universal constant C.
2 Basics of diffusion-based generative models
In this section, we will introduce the basic forward and backward dynamics of diffusion models, as well as the denoising score matching setting under which a model is trained.
2.1 Forward and backward processes
Consider a forward diffusion process that pushes an initial distribution Po to Gaussian
d.X, = -f. Se. X, dl + /2 dW.,
where dW, is the Brownian motion, X, d-dim. random variable, and X, Pt. Under mild assumptions, the process can be reversed and the backward process is defined as follows
dY. = (JT, + 207 (Y:) di 20
where Yo ~ Pr, and Pt is the density of P.. Then Yr- and X, have the same distribution with density Pt [4], which means the dynamics (2) will push (near) Gaussian distribution back to (nearly) the initial distribution Po- To apply the backward dynamics for generative modeling, the main challenge lies in approximating the term V log (Y,) which is called score function. It common to use neural network to approximate this score function and learn it via the forward dynamics ([] then, samples can be generated by simulating the backward dynamics
The training of score function via denoising score matching In order to learn the score function, a natural starting point is to consider the following score matching objective [e.g., 29
2.2
1 S. 2 =
Lcont
v (X)) dt
(3)
where S(O;t,X,) is a 0-parametrized neural network, w(t) is some weighting function, and the subscript means this is the continuous setup. Ideally one would like to minimize this objective function to obtain 0; however, pt in general unknown, and so is the true score function V,, log Pt- One of the solutions is denoising score matching proposed by Vincent (48) where one, instead of directly matching the true score, leverages conditional score for which initial condition is fixed so that Pijo is analytically known.
3

---

More precisely, given the linearity of forward dynamics I its exact solution is explicitly known: o² ds. Then the solution is X, e Xo σ where (2mo²))-// exp(-1a
Let pe Jú fa ds, and of 2 Su eu,
Ç N' (0, We also have X,lXo, -N Xo, 1 and go (xly)
(2a)) which is the density of X1 Then the objective can be rewritten as
1
['
V Vlog 90 (x,lXo))l 1 S.d w(t)C,dtt
(f)
Ex (15
2 Jto
[
L
+ E dt S. w(t)
w(t)
Ex Ee Ex
0t
where C, = Ex,1I logpell Ex, Ex x.
(X, 12 For completeness. we will provide detailed derivation of these results in Appendix C and emphasize that it is just review of existing results in our notation.
Throughout this paper, we adopt the variance exploding (VESDE) setting [46] where ft = 0 and hence 14 0, which also aligns with the setup in the classic of EDM 30
3 Error analysis for diffusion-based generative models
In this section, will quantify both the training and sampling processes, and then integrate them into a more comprehensive generation error analysis.
3.1 Training
In this section, we consider a practical implementation of denoising score matching objective, represent the score by a deep ReLU network, and establish the exponential convergence of GD training dynamics.
Training objective function. Consider a quadrature discretization of the time integral in on deterministic ceceiiiiiooolooooooooo collocation points 0 to < t1 to <... < ty = T Then
4 based
- (((()((s-(-)))C; Loonti (0) ((0) + +C2 and
(5)
where C= = 1
2 E
N [(0)=
1 Ex. -x-) 1o,, S(0;t,,,,) +
Σ (t, 2 f=1
(6)
Define 3, = w(tt 1) -tz 1 to be the total weighting. Consider the empirical version of L (6 dz,
Denote the initial data to be { with x; ~ Po. and the noise to be { )N with Egg (0, Ia). nn Then the input data of the neural network is {43, Xij) i=1.j=1 (1,, + ãt, Eaj 111 and the output N.NN data is (Ess/ōr, 1=1 if oe, # 0. Consequently, E(f) (6) can be approximated by the following
1 N 2n
Lem( (0)
M
B, S(0:t), + Esg) E
We will use 7 as the training objective function in our analysis. For simplicity, also denote f(0;i,)) 3, S(Oity, + ō,, + E.g and then Lem (0) 1 Σ N ((;ú,,).. Note the time dependence can be absorbed into the X dependence. More precisely, because o. is monotonically increasing function of t,, we can replace ł, in the inputs by ãt, to indicate the time dependence. This is then equivalent to augmenting Xıy be d + dimensional with (2:)) 0 and (Gi) (Gij)a+1 := 1. For simplified presentation, we will slightly abuse notation and still use d as the input dimension rather than d + 1.
Architecture. The analysis of diffusion model training is in general very challenging. One obvious factor is the complex score parameterizations used in practice such U-Net 42) and transformers 40) 34]. In this paper, we simplify the architecture and consider deep feedforward networks. Although it is still far from practical usage, note this simple structure can already provide insights about the design space, as shown in later sections, and is more complicated than existing works [25] 44) related
to the training of diffusion models (see Section L.I. More precisely, consider the standard deep
ReLU network with bias absorbed:
S(0; tú, X1²) = WL. Ww
(8)
'Otherwise it is no longer GD training but stochastic GD.

---

where 8= (Wo,",,W4Hi) Wo é R WL+1 é Rdxm, We e Rmxm for 8= 1, L, and o()) is the ReLU activation.
Algorithm. Let ((*) = (Wo, ( (1) w WLt1). We consider the gradient descent (GD) algo- rithm as follows
g(h+1) = ((h) Av Levmmm((())
where h > is the learning rate. We fix Wo and W L-1 throughout the training process and only update W1 Wz. which is a commonly used setting in the convergence analysis of neural networks I1 10
25
M. which is to set (W/ N(0, ()) for 6 0, -., I,, i,j = 1, .", m. and (w2))ss ~ NN 4) for 1.., d, j 1..,m. m.
Initialization. We employ the same initialization as in Allen et al.
For this setup, the main challenge in our convergence analysis for denoising score matching lies in the nature of the data. 1) The output data that neural network tries to match is an unbounded Gaussian random vector, and cannot be rescaled as assumed in many theoretical works (for example, Allen-Zhu et al. [] assumed the output data to be of order o( (1))). 2) The input data X;y is the sum of two parts: 2, which follows the initial distribution Po. and Gaussian noise ot, Eey- Therefore, any assumption on the input data needs to agree with this noisy and unbounded nature, and commonly used assumptions like data separability [ 35) can no longer be used.
To deal with the above issues, we instead make the following assumptions.
Assumption 1 (On network hyperparameters and initial data of the forward dynamics), We assume the following holds:
1. Data scaling: || = (2) for all i.
2. Input dimension: d = (((ooy((og(nnN))
We remark that the first assumption focuses only on the initial data z; instead of the whole solution of the forward dynamics XLy which incorporates the Gaussian noise. Also, this assumption is indeed not far away from reality; for example, it holds with high (at least 1 - (exp(((2(( (()))) probability for standard Gaussian random vectors. The requirement for input dimension d is to ensure that d is not too small, or equivalently the number of data points is not exponential in d.
We also make the following assumptions the hyperparameters of the denoising score matching. Assumption (On the design of diffusion models). We assume the following holds:
1. Weighting: Σi"1 (t, - -t--1) = O(N..
2. Variance: oz 0 and oty O(1)
The first assumption is to guarantee that the weighting function w(t) is properly scaled. This expression (t, -b5-1
obtained from proving the upper and lower bounds of the gradient of 7.) and is different from the total weighting 3, defined above. In the second assumption, 2t > 0 ensures the output Et, is well-defined. The FIN ( guarantees that the scales of the noise Eij and the initial data x; are of the same order at the end of the forward process, namely, the initial data T; is eventually forwaferw to near Gaussian with the proper variance. Therefore, Assumption aligns with what has been used in practice (see Section 4 and Karras et al. (30] Song et al. [46) for examples).
The following theorem summarizes our convergence result for the training of the score function. Theorem 1 (Convergence of GD). Define a set of indices to be g(a) ((..))(((());..))) 2 f(0()) ('))) for all "'))). Then given Assumption ists some M (Etrain) = 2 (ppol((n, N, L,T/to, log( Paraia )) s.t., when m 2 M (Etrain). h
1 and 2 for any Etrain > O, there ex-
O miny (((5)((,-((-1)0 ) and k = O(d' (nN) (xp(-(((c2ao))) we have
nN
d d)), with probability at least 20 n log( (o
A-1
md mn 1
Cem(o(h) S 1 - Csh w ((tt) - -
II ==
N Lem (o(0))
where the universal constant C3 > 0, ag E (22 1), and (i'(s), (s)) aag w(t,)(t) ug
Gr,,) Moreover, when K Odd""" n2 Nlog( Creain -a))
Lem ((())) S Etrain.
5

---

The above theorem implies that for denoising score matching objective Le GD has exponential convergence. For example, if we simply take j min, ((tz)(tz -
1) then Liem (e(k))))) is k= Lum (o(() The rate of
further upper bounded by (1 Cah w(( (ts+ - -5- md n'N=
convergence can be interpreted in the following way: 1) at the kth iteration, we collect all the indices of the time points into (()) where I ((0; has the maximum value; 2) we then choose the maximum of w (())((1) ((-1))----) among all such indices and denote the index to be (k),.. and obtain the decay
ratio bound for the next iteration
Cgh (t)))) (s))(( - (N)-
Remark 1 (Can Crrain be arbitrarily small? Some ramifications of the denoising setting). Let us first see some facts about Lem and L. Under minimal assumption of the existence of score function and in the zero-time-discnt discretization-ernee
limit, the score matching objective can be made zero and therefore the denoising score matching objective is bounded below by -C, which is nonnegative and zero only when the data distribution is extremely special (we thus write -C 0 from hereon unless confusion arises). That is, ming [(0)= 2 min function S L -C O according to A Since fem -. as the sample size of the training data set n + 00, we have Lem 2 - - Cn > foror some constant Cr. > O
and C. as n2 00.
However, Theorem 1 seems to imply Cem(((k)) - O as k oo since Lem Lem(O(K)) S Exrain and étrain arbitrary, and it seems to contradict the -C> O lower bound. However, there is no contradiction due to the combination of two facts. First, the theorem states that for arbitrary Etaain > O, there exists a critical size, such that for overparameterized network beyond this size, GD can render the loss Lem (0) eventually no greater than Etrain- If we fix the network size, i.e., with m. L, d given, then K given, and Theorem Isays nothing about GD's behavior after K iterations. That is, we do not know whether lim supk--se Lem Lem(((()) (0(8) = O. Second, our optimization setting requires the sample size n to be smaller than the network width m (Assumption Tv Thus, when m is fixed, the sample size n upper bounded.
The above discussion implies, within the validity of our theory, for any fixed network width m, if Strain is small, the sample size cannot be too large, meaning Lem(0) L(0) may not be small. Therefore, we can simultaneously have Lem(f) close to O and L(0) close to C >0. 0.
Main technical steps for proving Theorem 1 The proof of Theorem I is in Appendix D where the analysis framework is adapted from Allen-Zhu et al. M. Roughly speaking, the key proof in this framework is establish the lower bound of the gradient. Then by integrating into the semi-smoothness property of the neural network, we can obtain the exponential rate of convergence of gradient descent. For the lower bound of gradient, we develop a new method to deal with the difficulties in the denoising score matching setting (see the discussions earlier in this section).
Our new proof technique adopts a different decoupling of the gradient and leverages a high probability bound based on a high-dimensional geometric idea. See Appendix D.I for a proof sketch and more details
3.2 Sampling
In this section, establish nonasymptotic error bound of the backward process in the variance exploding setting, which is an extension to Benton et al. [8]. For simplified notations, denote the backward time schedule as (tj JosjsN such that 0= to < ेτ <... <<..<<>> t = T
Generation algorithm. We consider the exponential integrator scheme for simulating the backward SDE (2) with fe = The generation algorithm can be piecewisely expressed as a continuous-time SDE: for any & (
dY, 1Y:220 20=200--500 S(0 17 Yy,Y )dl doW
(10)
Initialization. Denote a := LLw(Y.) for all t [0,T - 8J. We choose the Gaussian initialization, Uu = O.).
Our convergence result relies the following assumption.
Assumption 3. The distribution Po has finite second moment: Eg- [| [ m² oo.
Next state the main convergence result, whose proof is provided in Appendix [
'The exponential integrator scheme is degenerate since ft = 0. Time discretization is applied when we evaluate the score approximations (((;,t,Y)).

---

Theorem 2. Under Assumption 3 for any 8e (0, and T > 1, we have
m
N-1
Kl(psl0r-8)
Σ
2
By 1f pr [[S(0:T - -1,, v Vlogp--; (Ye )
T
j-0
E,
Ey
N-1
hii T dt
So 5 of
dl
N-1
0 o T TT-
+ Σ
3-0
13
T-
m2
o7
(mg + d)
j-1
Σ
(1-
07 T--
T-
En
(11)
where 7/ == J+1 -t for all j 0, 1.,...,N is the stepsize of the generation algorithm in (0)
Theorem 2 VESDE-based diffusion model's analogy of what's proved in Benton et al. for VPSDE- based diffusion model, only requiring the data distributions to have finite second moments, and it achieves the sharp almost linear data dimension dependence under the exponential time schedule. The major differences from [ are (1) the initialization error in the VESDE case is
handled differently (see Lemma 10: (2) Theorem 2 applies to varies choices of time schedules, which enables to investigate the design space of the diffusion model, as we will díscuss in Section 4 Worth mentioning is, Yang et al. [54] also obtained polynomial complexity results for VESDE based diffusion models with uniform stepsize, but under stronger data assumption (assuming compact support). Compared to their result, complexity implied by Theorem has better accuracy and data dimension dependencies. A detailed discussion complexities is given in Appendix I.T
Terms Ej, Ep, Eg in II represent the three types of errors: initialization error, discretization error, and score estimation error, respectively. Term E, quantifies the error between the initial density of the sampling algorithm 4o and the ideal initialization pT. which is the density when the forward process stops at time T. Term En is the error stemming from the discretization of the backward dynamics. Term Es characterizes the error of the estimated score function and the true score, and related to the optimization error of Lem- Important to note is, in Theorem 2 population loss is needed instead of the empirical version Zem
Besides this, the weighting 7, T-17 is not necessarily the same as depending on choices of w((1)) and time and variance schedules (see more discussion in Section 4 We will later on integrate the optimization error (Theorem I into this score error Es to obtain a full error analysis in Section 3.3
the total weighting in Lem
Remark 2 (sharpness of dependence in d and m2). In one the the simplest cases, when the data distribution is Gaussian, the score function is explicitly known. Hence KL(pslqr-s) 5) can be explicitly computed as well, which verifies that the dependence of parameters d and m2 is sharp in Es and Ep-
3.3 Full error analysis In this section, we combine the analyses from the previous two sections to obtain an end-to-end generation error bound.
Before providing the main result of this section, let us first clarify some terminologies.
Time schedule, variance schedule, and total weighting. The terms time schedule and variance schedule respectively refer to the choice of 17 and oty sampling. Meanwhile, note both the training and sampling processes require the proper choices of time and variance, and these choices are not necessarily the same for both processes. For training, the effect of these two is integrated into the total weighting By. which is also influenced by an additional weighting parameter w(t)). ). In this theoretical paper, when studying the generation error, we aim to apply the optimization result better understand the effect of optimization on sampling. Therefore, to simplify the analysis and discussions we choose the same time and variance schedules for both training and sampling. The main result stated in the following.
in Section 4
Theorem Under the same conditions as Theorem 12 and that K is such that GD reaches Crrinn in at most Kth iterations, we have
o (Ctrain + En + Eest + Capppoox)
(L(pslrr=s) $ E, + ED max IsiSN w(IN;;)
where Fr, ED are defined in Theorem étrain is defined Theorem I en = (((O(K)) ((o(K))-Lem Lem (O(K))
Lem (0n - -(-0)), Eest -E(0*) -((0F) Capprox = |[(0s) + Cl. In these terms, C is defined in (5)
7

---

0r arg ming s.st., (0)-0 (()) and 0, (rs) [E()) C with F ReLU network function defined in 8 with d (o(polyllog(nN)))) ( N. d, L, }-
In this theorem, the discretization error ED and initialization error E, are the same as Theorem 2 For the score error Es, our optimization result is valid for general time schedules and therefore can directly fit into the sampling error analysis, which is in contrast to existing works [25] 44] (see more discussions in Section 1.1 The coefficient max, oi //w(NN))) results from different weightings
in Es and Lem, i.e., T-0 and 8j. We will discuss the effect of max, , //w(tN-s) under
different time and variance schedules in Section 4
The way we bound -P [15(0) 15), (Vr;)19) in Es (see Theorem 2
to decompose into the optimization error étrains statistical error 6ns estimation error fest, and approximation error fapprox- This gives clear intuition to results, but we also note it may not give
tight bound. In fact, we have
En Etrain = (E(((K)) ( Lem (0*) [(0")|+ (1m))))))) - -em 2 L(O(K)) [(0-) 2 (()) 20 0.
'n can still be small if we take n oo, but that means Crran has to be large, and our generation error bound cannot be made 0. is unclear yet whether this is due to limitation of our analysis or intrinsic, and will be left for future investigation.
Another related note is, in this paper, we focus on étrain and the effect of optimization, but the analyses of Ex. Cest, and Capprox are also important and possible [3 39 25 50]. On the other hand, again, whether is optimal to decompose the full error into these four is unclear.
To better see the parameter dependence of the error bound in Theorem
3 the following is an example
with simplified results, where we employ the schedules in EDM 30
Corollary 1 (Full error analysis under EDM 30 designs). Under the same conditions as Theorem 3 we have
m T N
d
T
aT
1
md 20-1
KL(polur-s)
S
(m3 d)
NEN
N2
N
C
1 -shh
n
where Cs, Cg 0 and a= 7 in 30
4 Thaory-based understanding of the design space and its relation to existing empirical counterparts
This section theoretically explores preferable choices of parameters in both training and sampling, and shows that they agree with the ones used in EDM (30 and Song et al. [46] in different circumstances.
4.1 Choice of total weighting for training
This section develops the optimal total weighting 3, for training objective (7). We qualitatively show in two steps that "bell-shaped" weighting, which is the one used in EDM 30), will lead to the optimal rate of convergence: Step lo,, S(Btt) Xey) Etyl as function of j is inversely "bell-shaped"; Step 2) ((0;i,)) = 3, S(B:t, X1y) Eayl should be close to each other for any i,j-
4.1.1 Inversely "bell-shaped" loss | x.,) + Esy as a function of time index j Proposition 1. Under the same assumptions Theorem for any 0 and 1,"",, n, we have
Vez > 0, 51 > 0, s.t., when 0 S ot,- 61, S(0:t, + E.y) + En > | -1. &all M (IS(::t)
Vez > 0, M O, s.t., when ot, M. |o,,,
The above proposition can be interpreted in the following way. Given any network S, when ot, is very small, [ which is of order Vd with high probability, i.e., it cannot small. When ãty is large, shows that 24
implies that |i,,, s( 5t Eyyl is away from 0 by approximately I&:ill
it becomes larger and larger, i.e., as M increases, | S(0; L4 + o,, {) + Eajil will also increase. Therefore, the function lō,, S(0; ((,tt,,X,y) + Exyl has most likely inversely "bell-shaped" curve in terms of j dependence.
8

---

4.1.2 Ensuring comparable values of f(0;i.j) for optimal rate of convergence
Corollary 2. Under the same conditions as Theorem for some large K' O, if if i. f(g(kt)); :1,88]= ;1, then have, for some constant C7 O,
& e holds for all l > 0 and all (i,j), (1,8), with some small universal constant > 0,
(111111 (1 $1--111 1 C7h
md N The above corollary shows that if [(0()); j'' are almost the same for any i,j. then the decay ratio of the next iteration minimized. More precisely, the index set ((k) defined in Theorem 1 is roughly the whole set (1, and therefore w(tz- (tz- - - can be taken as the maximum value over all j. which consequently leads to the optimal rate.
Lem (olk+k)))))
w w(()(() (1)1))) s=1.-.,N
Lrom (OCK'))
4.1.3 "Bell-shaped" weighting: our theory and EDM
Combining the above two aspects, the optimal rate of convergence leads to the choice of total weighting 3j such that ((0;i,)) S, xx,) + Eiyl is close each other; as result, the total weighting should be chosen as a "bell-shaped" curve as a function of Exyl--
Figure 2:
Weighting choice 3rDM in EDM.
according to the shape of the curve for |ōr, S(0:t,,
Before comparing the preferable weighting predicted by our theory and the intuition-ant -nd-emiiriis- based in EDM (30,, let us first recall that the EDM training objective can be written as Eg-persin By ( Do(y+ + n; - y/
[ Z
1
(log 9 =
2P2
atd
σ² data
E Exo.e (0:,,loss0;;,,,,)
ElI do,
(12)
data
data
(lox Ponean)?
- - oata
where Z is a normalization constant, and we denote BEDM (7) =
27 aad
to be the
total weighting of EDM. Note the dependence and time be freely switched due their 1-to-1 correspondence.
F the total weighting of EDM θEDM a function of ō. As is shown in the picture, this is a "bell-Shaped" curve which coincides with our choice of total weighting in the above theory. When is very small very large, according Proposition the lower bound of Iō,, ST,:S(0 (j cannot vanish and therefore needs the smallest weighting over all o. When ō takes the middle value, the scale of the output data Êıyōō is roughly the same as the input data Xiy and therefore makes it easier for the neural network to fit the data, which admits larger weighting.
Eúlll
4.2 Choice of time and variance schedules
This section will discuss the choice of time and variance schedules based on the three errors Es, ED, Ey in the error analysis of Section 3.3 Two situations will be considered based on how well the score function is approximated in training: when the network is less trained, Es dominates and polynomial schedule [30) is preferable; when the score function is well approximated, Ep + Ej dominates and exponential schedule [46] is better.
4.2.1
When score error Ess dominates
As is shown in Theorem 3] the main impact of different time and variance schedules on score error Es appears in the term max; of //w(LN-j), when the score function approximated a certain accuracy. remains compute ut)t under various choices of schedules.
General rule of constructing w(t) To ensure fair comparisons between different time and variance schedules, maintain fixed total weighting in the training objective. Additionally, to facilitate comparisons with practical usage, we adopt the total weighting in EDM, i.e., 35 C3.EnD ), for some universal constant C3 > 0. The reason for using the EDM total weighting is that according to Section 4.1 our total weighting B, should be "bell-shaped" as a function of j. which agrees qualitatively with the one used in EDM.
'In EDM 30 , they use Pinaa 1.2, Petd 1.2, Gdata 0.5, 7min 0.002, Fmax 80. 'This horizontal axis is in lcale and the plot in regular scale is a little bit skewed, not precisely "bell" shape. However, we remark that the trend of the curve still matches our theory.

---

Polynomial schedule [30] vs exponential schedule [46]. We fix en, Ctrain
and apply the two Then, compute The Exp.'s result
schedules (Table D separately the above total weighting & (hence w)
maxj o fux_//((lon which is factor in score error Es (Thm in Table 2
ta for large N, meaning the poly. time schedule in EDM is better than the exp. schedule in (46). Note these
mmaxx Fmax
Canin
1/N) is larger than the Poly.'s result (omax-
(olia
max
Omai
Two terms are both of order 1/N as N - oo and therefore the difference lies in their prefactors.
Table Polynomial and exponential (time) schedules.
Variance schedule o, Time schedule łj
Poly.
30
1
o @maax
((mmax
1/p N-1 min
)
N
N-i N
Exp.
46
Vt
T maX
Table Comparisons between different schedules.
Es (score error) dominates
ED EE (sampling error) dominates
max, oī, /w((t)
Choice
N
Choice
Poly. 30
C4 @mmax
221p (Fmaxx
2/10 Npp
Ω
m&vd o (muas d. p²
1/p
g mux
amain
Exp.
46
C2 2 (Jmax- mmax
1/20
mdvd
In
Cunn 22 o
Omia
4.2.2 When discretization error ED and initialization error E, dominate In this section, we compare the two different schedules in Table studying the iteration complexity of the sampling algorithm, i.e., number of time points N, when ED + Ej dominates.
General rules of comparison. We consider the case when the discretization and initialization errors are bounded by the same quantity c, i.e., Fl ED $ E. Then according to Theorem 2 and Theorem 3 compute the iteration complexity for achieving this error using the schedules in Table To make the comparison more straightforward, we adopt
iy (ppol((=-)) and therefore
Tmax
More details are provided in Appendix [.1
Polynomial schedule (30) exponential schedule 446.. As is shown in the last column of Table 2 the iteration complexity under exponential schedule [46] has the poly-logarithmic dependence on the ratio between maximal and minimal variance (o maax//mii min which is better than the complexity under polynomial schedule 30), which is polynomially dependent on ō /max/ min- Both complexities derived from Theorem by choosing different parameters.
Remark (The existence of optimal p in the polynomial schedule 300). For fixed Tmax and min the optimal that minimizes the iteration complexity is p= In Emax
In 130,, it was empirically observed that with fixed iteration complexity, there is an optimal value of that minimizes the FID. Our result indicates that, for fixed maax and min, hence the desired accuracy in KL divergence being fixed, there is an optimal value of that minimizes the iteration complexity to reach the fixed accuracy. Even though we consider a different metricldivergence instead of FID, our result still provides a quantitative support to the existence of optimal p observed in (30].
Gnju
Acknowledgments and Disclosure of Funding
The authors are grateful for the partially support by NSF DMS- 1847802, Cullen- Scholarship. and GT-Emory Humanity.Al Al Award. We thank the anonymous reviewers for their helpful comments.
'This holds under parameters used in either Song al.
46
Karras et al. 30
'The exponential time schedule under the variance schedule in 30 also has the poly- lol--logari dependence on dmax//Fmiaa Under both variance schedules in (30 and 46 it can be shown that exponential time schedule is optimal. Details provided in Appendix []
10

---

References
[1] Zeyuan Aluen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterizzaion. In International conference machine learning, pages 242-20 2222.. PMLR, 2019.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. Advances in neural information processing systems, 32, 2019.
[3] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205. 15019, 2022.
[4] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313332,, 1982.
[5] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
[6] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:17981-11 17993, 2021.
[7] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffuusion models. In International Conference Learning Representations, 2022. URL htttps://openreviess http://pppaaeeeeeeeeeeeeerrdrrds wttt:/orumm:dd-SSSS/20
[8] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffuusion models via stochastic localization. The Twelfth International Conference Learning Representations, 2024.
[9] Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107, 2020.
[10] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural semporal-difference learning converges to global optima. Advances Neural Information Processing Systems, 32, 2019.
[11] Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering. 2024.
[12] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: Urie-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pages 4735-4763 PMLR, 2023.
[13] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. International Conference Machine Learning, pages 4672-4712. PMLR, 2023.
[14] Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Applications, guided generation, statistical rates and optimization. arXiv preprint arXiv:2404.07771, 2024.
[15) Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learn- ing Representations, 2021. URL
[16] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. ICLR, 2023.
[17] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 10:492479 2022.

---

[18] Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffusion models without early stopping: finite fisher information is all you need. arXiv preprint (ttd:2200.11240000 2023.
[19] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. TMLR, 2022.
[20] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 1888880-8994, 2021.
[21] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685, PMLR, 2019.
[22] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 2018.
[23] Chenru Duan, Yuanqi Du, Haojun Jia, and Heather J Kulik. Accurate transition state generation with object-aware equivariant elementary reaction diffusion model. Nature Computational Science, 3(12)::045-1055, 2023.
[24] Xuefeng Gao and Lingjiong Zhu. Convergence analysis for general probability flow odes of diffusion models in wasserstein distances. arXiv preprint arXiv:2401.17958. 2024.
[25] Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. In The Twelfth International Conference on Learning Representations, 2024. URL httpps:// net/forum?? 1d-hhhee00xtta)
[26] Ye He, Kevin Rojas, and Molei Tao. Zeroth-order sampling methods for non-log-concave distributions: Alleviating metastability by denoising diffusion. arXiv preprint arXiv:2402.17886, 2024.
[27] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[28] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022.
[29] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[30] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35: 26565-26577, 2022.
[31] Tero Karras, Miik Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696, 2023.
[32] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882. 2022.
[33] Yongjae Lee and Woo Chang Kim. Concise formulas for the surface area of the intersection of two hyperspherical caps. KAIST Technical Report, 2014.
[34] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances Neural Information Processing Systems, 35:4328-4343, 2022.
[35] Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems, 31, 2018.
12

---

[36] Xin Liu, Zhisong Pan, and Wei Tao. Provable convergence of nesterov's 's accelerated gradient method for wee-parameterized neural networks. Knowledge-Bacd Based Systems, 251:109277, 2022.
[37] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution, arXiv preprint arXiv:2310. 10..::30,,,.344,,, 2023.
[38] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022. URL httpp:://pppereiiiieeee
htlffforummid-aBBC> cPu_tE
[39] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning, pages 26517-26582. PMLR, 2023.
[40] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint 2rXfv:2204.06125,, 1(2):3, 2022.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages
Springer, 2015.
[43] Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure- based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13665, 2022.
[44] Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. Advances in Neural Information Processing Systems, 36: 16666996-9,,,, 2023.
[45] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
[47] Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound. arXiv preprint arXiv:1906.03590 03599, 2019.
[48] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
[49] Joseph L Watson, David Juergens, Nathaniel Bennett, Brian Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 188
2023.
[50] Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv preprint 24Xx:22002.0777,, 2024.
[51] Junde Wu, RAO FU, Huihui Fang, Yu Zhang, Yehui Yang, Haoyi Xiong, Huiying Liu, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. In Medical Imaging with Deep Learning, 2023. URL httpstts::/ htttps::/opeereiieeee net/forumiid
Jdw- cm2G9
[52] Kaylee Yingxi Yang and Andre Wibisono. Convergence in KL and Rényi divergence of the unadjusted langevin algorithm using estimated score. NeurIPS Workshop on Score-Baser Methods, 2022.
13

---

[53] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang. Bin Cui, and Ming-Hsuan Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
[54] Ruofeng Yang, Zhijie Wang, Bo Jiang, and Shuai Li. The convergence of variance exploding diffusion models under the manifold hypothesis. OpenReview, 2024.
[55] Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-bases based generative models. In International Conference on Machine Learning, pages 12062-12072. PMLR, 2021.
[56] Yuchen Zhu, Tianrong Chen, Evangelos A Theodorou, Xie Chen, and Molei Tao. Quantum state generation with structure-preserving diffusion model. arXiv preprint arXiv:240.006336, 2024.
[57] Difan Zou and Quanquan Gu. An improved analysis of training over-paramettrized deep neural networks. Advances neural information processing systems, 32, 2019.
[58] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine learning, 109:467-492 2020.
14

---

Appendix
A Conclusions and limitations
Conclusions. In this paper, we provide first full error analysis incorporating both optimization and sampling processes. For the training process, provide first result under a deep neural network and prove the exponential convergence into a neighborhood of minima. At the same time, we extend the current analysis to the variance exploding case for sampling. Moreover, based on the full error analysis, we establish a quantitative understanding of the error bound under the two schedules. Consequently, we conclude with a qualitative illustration of the "bell-shaped" weighting and the choices of schedules under well-trained and less-trained cases.
Limitations. The network architecture used in the model is deep ReLU network. Although being so far the most complicated architecture for theoretical results, it is still far from what is used practice like U-Nets and transformers. Moreover, regarding the full errоr analysis, we only focus on the optimization and sampling error and do not dissect the generalization error. When bridging the theoretical results with practical designs of diffusion models, our results are mostly qualitative and we only compare two existing schedules under two extreme cases, when the network is well-trained and less- less-trained. Thus, theoretical implications on practical designs remain be explored. We will leave these perspectives for future exploration.
B Notations
X Y
Solution of forward dynamics [1)
Solution of backward dynamics
Y
Solution of generation algorithm 10
of
Diffusion coefficient of (T) and
Standard deviation of X, 44
Lconti
Continuous-tine score-mater score-matching objective 3
C
Discrete-time denoising score-matching objective (population version)
Lem
Discrete-time denoising score-matching objective (empirical version)
C₁
Constant between score- coore-matching and denoising score- matching loss time (4)
C
Constant between mote--mtchhnng and denoising score-matching loss over all discrete times
55)
Sample from the initial data distribution Po
Xij
Sample from the distribution Pt at time t (7)
The jth time point for forward process (6)
The jth time point for backward process (10)
The first (last) time point of the forward (backward) dynamics, i.e., to [II
T
Stopping time of the forward dynamics
[11
Difference between backward time points, 7-1 17
(1]
pe
Density of the solution of forward dynamies at time (and backward dynamics at time
- 1TT
ge
Density of the solution the generation algorithm time
wt)
Weighting function (3
Bj
Total weighting, i.e. wltt (t; 1-- 1),o,
BEDM
Total weighting used in EDM 30 12)
max (amin)
Maximum (minimum) of o, (Table
Number of samples from the initial distribution Po
N
Number of time steps when discretizing the forward and backward dynamics (
d
Dimension of input, output data, and the solutions of the dynamics (1 and
Deep ReLU network (parameterization of score function)
0
All the parameters in the network S The weight the ith layer of the network S The kth iteration of the weights / through GD
W.
g(k)
w((k)
The &th iteration of the weights We through GD 9
m
Width of the network
15

---

L
Depth of the network 8
(**())
Index of the largest loss and (t, -t the sth iteration (Theorem Second moment of the initial distribution Po 11
m3
Ej
Initialization error 11
ED
Discretization error
Es
Score error (II)
étrain
Optimization error (Theorem
En
Statistical error (Theorem
Cest
Estimation error (Theorem
Capprox
Approximation error (Theorem 3
Minimum point Ć when Zem (Theorem
0f
Optimal parameter in the function class (Theorem 3
C Derivation of denoising score matching objective
In this section, we will derive the denoising score matching objective, i.e. show the equivalence of and 4 For simplicity, we denote So to be the neural network we use (,).
Consider
E Ex.-P, (S(::, V 1ogp:12 Ex, [1S012- (Se, Vlogpi)) / logp:: where pe is the density of X₁. Since px())) pu(v) (xll)) where (-) is the density of X, then have
(13)
Ex, (So, log
S v logp: pt dx.
f S Vp dx.
ƒ ()V)(=))))o()) drdy
If S7 log (x) drdy
Ex P. Ex (Sa. v Vloga((rela
Then
13
Ex E Q. [150 - ( Ex, log
Ex, Ex Ex, Sq - v log q: 12 +C,
(A)
where Ex./E.VVVV.. logp. 22 Ex, E x, log (x
Moreover, -N(No~N(ET1) Xo, af I). and density function
x - E e y
2e (xly)) (2m²²) 4/2 exp
2
Then
(A) Ex, Ex,
Sq log 1
|| nh Xoll
Ex,,EXIAA X
Sq V.
207
X1 c-M
Ex,Exx
Sq
E1
Exx Be,
So
16

---

Let = f 8,
~ ~O, I). Then
(A) = Ex, Reo 06
o. + El
Ex, Egl + E|1
0,
D Proofs for training
In this section, we will prove Theorem
I
Before introducing the concrete proof, we first redefine deep fully connected feedforward network
Tij.0 Wi.Ki ij. 9:3,0 = (rij,a),
Tij. Wi 4:t (((,)) for L=
X(0;t,, = L-19j,L
where Wo E RRaxd W e Rddm and We E mm a is the ReLUU activation. We also denote Qtj.1 be Xzy.
also follow the notation in Allen-Zhu et al. I and denote Di,e 6 mR to diagonal matrix and (Di,)Ra* 1)) for lo = 1, m,, m. Then
4z Dij.W
For the objective 7 the gradient w.r.t. to the kth row of We for f= 1,",, L is the following
Com
Σ 1=1
w(1)((1)-)) (1-1)
L(W D
WL+1QH, + Eis) 011,e-1 I(Wegy,e))
Rey.te1
Throughout the proof, both Zem (f) and (e) to represent the same value of the loss function, where W (W1,",,W)) and we let b=
Next, we will prove Theorem
Proof of Theurem
First by Lemma 4
Lem (W(1)) = O1220 (o0dda-e(i)))) Ew(())(--- -(-))(()))))//r))))))))))
Also, (omm((0) VL maxi //
Then we have
[W()) 10))
h M Lem [
5
s s0(,
midka-N
max M mmx (1;) -(--)))) hik max max
Lom (W(1)
O
maBa- NL max w(t, (t, 1 d'k v
-tt=)))
Let
N. min, (f, 1)
and
Old 1-g0 n² N log( اه) )),
Cerain
where
Atrain
is some small
constant.
Then W and by Lemma
0 4 2/2 1 /max, wlt, (l, (t, Es -(())((,(()-))/0, miny wott) (t,-))1))t
Ologg( Ctroin
//n
17

---

with probability at least 1 O(nN) (x--))xxp(--((PPoo0)))
Lem (W
Lemm( (Wi())
(
11 Ily Lom
- - 1-)00,, -(( 0
log mm Leom Co (W(K)))
(am 122v Loom
E (0(()))) -()-)))()-)--))) OL2 / V ( p2
md 20-1
$ (1 (gr-1))-- S2
Lem Lom(W(())
N3
+hC m5/6 Ej w (t) t5-1) miny w(t) -tj-1) Lem (W(K)
(logm)) max, (ttt((t) - tg-1) En (ttt)(t; - -t--1)/or, =d
ngN2 )) Lem Lem (W(K))
$ 1- - hoo (1)r -o Ω
where > is some constant, ap (1/2, 1); the second inequality follows from Lemma with
md 20
VV Crm (0(4))
N2 w(( (1.- - Lom(())))). (O(K))
which
is
obtained inductively;
the last
inequality follows
from m
Ω
23/2- 14/3 21 (1
max, ((t,,)((, Es -6 miny ((ts)(tttt)) / (t,)(()
D.1 Proof of lower bound of the gradient at the initialization
In this section, we will show the main part of the convergence analysis, which the following lower bound of the gradient. Lemma (Lower bound). With probability O(.N) exp((S((cRo-)) we have
md 20-2 (((;))(())) -)) Ccm (o(o))
(0 Co
N3N2
where (i" arg arg universal constant.
((tt))(,,-4-1) 0e,
(ō,, WL+1 Ev) < 1, and Co 0 is some
Below is the proof sketch of Lemma]
Proof sketch. We first decompose the gradient of the kth row of Wz Lemz)e ( (tyr - -ty-1) (ōrg. Winner + Eey (n
#. -(t)(t,
(ō,, W2-14gL Eg)
where (i", j") indicates the sample index with the largest loss value.
Then we first fix (115,t1)ss 1, and prove that the index set of both (1 and EG.D 39 (tt)((t) (tt how. 0 is order with high probability.
18

---

Next, we conditioned on the index set we' ve found, then we can decouple each element of V ((WL), Sem with high probability. We prove that with high probability
W fejr, Gerj, WL+1
E
aigbi,,
Σ
(4fStt)) cd
90-1
(1)))(()a))))
(1)))(())))))))
for some constant c and < ap < Based on this, we show that with probability at least 1 O(nN) exp(-O(d)),
P((V1),> 0,
ed ed
for some 0. Then prove that with probability at least exp(=S((md -4 (u+ llk: (W 1) O, (W (" {)
0} O -
with high probability, the event (Vi),> 0 and (V2)s has probability at least of order 12/2 where ag (1/2,1).
Now, we deal with (9t5.L-1)s and prove that if the above results hold for (9tj, L-1) 1, then there exists an index set with cardinality of order m/(nN) such that (Vi)s and (V2)s O also hold in this index set.
In the end, combining all the steps above yields the lower bound.
Here the complete proof.
Proof. The main idea of the proof of lower bound is to decouple the clements in the gradient and incorporate geometric view. We focus on Vw, Cem (0).
Step 1: Rewrite Lom() to be the (i", ,J" 1h term Ir plus the rest n.N 1 terms
(( Let (i" = arg max
(0, WT. 1. + s3)) Let
Dbj.L w(t; - ((ol)=(t-))()))) (or,WL-1955 + Eij) 1-j.-11
Then
V ((w)) (0)
wo(( (1.e -1--- (( (ōr (5
V
Σ -(1)((4---))) (ōx, W W + Eig) 1
+ " (1.))((-)*))
V2
Also define
V1. (o) -( (Wzr) W WL (a Des0
n
Via.a
(tt - 11) (WL+)** Engy (-) 1)
+ n
V12.a
22, Σ w(t;)(t) tz-1) ( (wg ((())(i*>"*)
WL-19H.
VzL.
+- Σ n ((5))((*(1"))
V2z.
19

---

Our goal is to show that with high probability, there at least O md s0- n.
number of rows & such
that VIL. O, V22 O, V21, O, 222 O. Then we can lower bound (V (0) by V |1?, which can eventually lower bounded by (0m
For (92)s. first take (1ij,t-1)ss for all (i,j) + (i" j*). Step 2: Consider (e Then only need to consider
2
E w(()(1) - (WLn) (ōt, W WI-11 (W (1)))
which is independent of For 11, since qe J',L-11 --1 we can also first take (" 1 for all s.
0 which does not affect the sign of this term,
Step 3: We focus on V11 and 22 and we would like to pick the non-zero elements in this two terms. More precisely, let
N1
{8l (1))g) 0,8 8= 00,,=1,..,,1 m,,
N₂
Σ ((t;)(t, 1 (awaas weo(gg,))s > 0, s= 1, m
() 5*)
Let ai w (t;)(ts t 1r, (W29),2-1)) 2 0. Then
Σ w(t;)(t; - - 1 (~AA)))))
AWense wwoll1),L)
Σ ai5(9),))s= E (W
(15)+())))
(1+)+())))))))))))))))))))))))))
Σ (W (W1)s (1))=({P5>)
aiglij,--1 0, 5))
Σ
then there must be at least one pair of (i,j) s.t. an (WL9).L-1)+ (WL4,,L-1))) > 0, which implies E(Ej))(t)))) aiy ())()1),))) O. Therefore, it suffices to consider
N1 (s (g0 = (WL) 0, 1,
N₂- s (WL)s Σ Po 0
(iJ)
Since (g1j,t-1)& 2 0, we have
1 Σ
Aijligti-l 20,
(15)=(1*55)
i.e., 10p0,1111
Σ
0sjlij.L-11
5
(1)))(())))
By Lemma and Proposition 2 we have
Σ a08h)11100
(())=(P)))
(Wr)s [(We)&ll
(Wr)s Σ 0099),2-110) 105 [(WL)al (1)))(()))))))
=
1
4'
Also (WD),' are i.i.d. multivariate Gaussian. By Chernoff bound,
p N(NNnn
(61
7 82 m
) 5 2e (mm)
20

---

for some small 61 $ and 62 4, i.e., [N1 n NgI = (mm) with probability at least 1 - 2e -2(m)
Step Next we condition on N1 n N2 and consider (Wi.1)" Winio f. + (WI.11) )M Eieg
and (WL.1) )* WT.1 E(.))-( (*) anyi,, + (Wi.-1)* E(cs)aCe atylis. where dr aiyloij. We would like to prove that with high probability
< L ger Errge, WL+1 ((())(())))))
Σ
aggiit+
E dsyLty) cd " (1.)))()))))))
for some constant e > and 1 ap <1
First, since Sij L2). by Bernstein's inequality, with probability at least exp((s(dd)), have E (d)) Similarly, since (Wr..1gg,r.)) ~O,, "l by Berstein's inequality and m Lemma with probability at least 1 exp( (d)()) we have [[W++1g,, @ By union bounds, the above holds for all i,j with probability at least - 2n.N exp(-9(d)).
Wa11 Eea.e- st) a.... Eeiane auy
Let v [Wi-1 (e E البلاب and WI jt For notational simplicity, use & to denote Ertge. se. Fix U, and consider the probability of event A
= {v (u + E) s -V1 coddo-1 E
for some Co > and 1/ ag 1.
Then consider the following event that has larger probability than A
(w'(u+ E))² (1 cod 1|u E||?
(14)
(v (1 - codd ||/l| /++ + 2( uu (1 -1 ccndd=--)) (w 2)2222 2 (1 1))
(15)
Since (vTu)²= < |u| where the equality holds when e
" Tol. we have
LHS s cad |u| 2 2(v uu (1 co -oddo-1)u) [+) E)
Also, since = with probability least 1 2nN exp(-x((d)) we have
d2a0
IP (12(0 wvv (1 (1-c)d=-))u)) 2 [ 2exp
-c |/ul
2 2eepp(1p2 -ad
for constant 0.
Therefore, with probability at least On.V) exp(-5((d20011)
LHS of (15 ed 00 (o g)²
for some constant e >> 0.
Then
(u+ - codao-1 u
El)
$
co |u+ -
$ 2 2 (1- c1-
{
-r- E
P
T
V (1 d doo-1)
P
2 (11 cdv0-1)
||[||
= P
(v,
arccos(-1o (1 + d
<(- |ET
2 (arcoos--Vil (1-dcdao-1))
2P
< (w
2 2
() 90
d-1vd
21

---

where the second equality follows from Lemmal 2 the third equality follows from series expansion; the forth equality follows from (T6;; c'.'', C' are some constants.
Thus with probability at least 1
(d vá-iva
v (u+ + {) -V1 coda0-1100 uu+
cd 1p-
i.e. < (v, u
for
Then by Lemma with probability at least - (x(xtt1-0)))
P
2 0) 2 ed cd
for some
Since (W are iid Guassian vectors for k = 1, by Chernoff bound on Bernoulli variable ((WW,, ("2.,,)"22 ( we have, with probability at least - exp(-S(mmd mg-1 ))
k|:: (Wh..1) Ju O, (W (u E) 03| = Omd "²
Step Combining the above 4 steps, we would like to obtain the lower bound of the gradient.
For each k, consider (4ijL-1)s for (1,j) * (i",j") and denote {(bj)L-))))) J*) Let go
(W((W--)) (L)))(())))) (())))))
and To ~ (O, QQ").
where each row of Q is L. for (i,j) (i" (*")))). Thus, qa qo projected to the nonnegative orthant.
Let 1= = (1, 1) RaN-1 Therefore, if (3x. 1) 0 for some 8& R then at least half of the nonnegative orthant is contained in (u E TR=N-11 (3k, v) 0), i.e., there exists a constant C > 0, s.t.
P((Sk, qe) 2 2) 2 ck
min Re Iemm
C₁ > 0, for all s 1.
Then since 3g 1 for k 1, and nN < m, there exists set of indices K. {1, "., m)
with IK] O( aN and set of indices s (1,".,m with |S] = O(m), s.t., (Be, 2 0, for k I,, 8 S.
m
Let d.j.e (gkj,) Then by Bernstein's inequality, we can also obtain that 102,.e112 = ( with probability least 1 n.V exp(-S(dd)).
Combine all of the above and apply the Claim 9.5 in Allen-Zhr et al. 1. we obtain, with probability least 1 (nNN 2x ))),
VV Fem (0 1
n² Cowltz (tz - 12) d I WL+10 Er, 12
ma "d
n.
2 Cemd 20-1 w(1 (1,5 - -1,- 1)- T fem (0(0))
where Co some universal constant, < a0 1, and second inequality follows from the definition of i", j."
D.1.1 Geometric ideas used in the proof
Proposition Consider w ~ (O, I), where w R Then [ll and [[ol] are independent random variables and |=] U))
Lemma 2. Let w Unif (S"-)) where gn-1 (xr E R²|[|=||| 1}.
Then for vectors U1, U2 R.
P(w 01 2 20, w 2 0) < (V1, U2)
2
22

---

Proof. Since w Unif(S" we only need to consider the area of the event. It is obvious that the set f e v;} is semi-hypersphere. Therefore, we only need consider the intersection of two semi-hypersphere, i.e.,
"
area of [w e ""= 21 2 0) n area of (w S""""=""""" 02 0}
P(w' 11 0,0, wwwwu 02
area of the hypersphere
π < (v1, U2)
2m
Next we follow the notations and definitions in Lee and Kim 33 Consider the unit hypersphere in Rd gd-1 {x E 12 ||=| 13. The area of S is
2/2
Aa(1) = I(d/2)
Lemma Fix E gd-1 and let & Unif Un where gd-1 for é [+|[ 1}. Then with probability least exp(-S((d)), have < (21, S 3x
Proof. For any fixed f all the 2 that satisfy (511 E2) 2 क Lee and Kim 33). the area of the hypersperical cap is
0 are on hyperspherical cap. By
dd-1 2
A2(1) = Aa(1)
2
2
Then
(())()))) - 0) AQ(1) sin
d 1
0
α
(16)
Aa(1) 2
2
2
2
VT
1/2
Let 0= -
Then with probability least 1 1-exp(-((d))) we have < <61,(2) 35
D.2 Proofs related to random initialization
Consider W; -W this section.
Lemma 4. If e e (0,1), with probability at least 1 - OnNN) A0min( O ,dttt)) |X 12 e Ille"Mss o² d- cot, "es, "0=""xx," r.|12 + oî d co?,d" dhh for all i == 1, 1,,.n and j 0, 0,,,. N
Moreover, with probability at least 1- - O ((L))-A(eeei) (Im=) over the randomness of W. for s= 0, L,
we have lqs,ell [[X (1-e), -), |X [X,i((( e) for fixed i,j. Therefore, with probability at least
1 1- L-O(nL ( (LL, )dd")) we have R(d') = =lqg,el = =dd").
Proof. Consider Xij "h 91 X d+ll oi,
+ Eij. Since Eij N 1), 2, X 2| follows from the noncentral
x2 distribution and E
"( x; (this includes the time variable at the dth dimension). nc,
By Berstein inequality,
P
X
X
2t exp
-emin
(..1)
ot,
2 - -emin
i.e,, 10
|le 1ee x; + Tz, Eisl 2 3 d lle "es xiil3
02.
5 2exp
7:t)
Therefore, with probability at least 1 -ttin) 2 20 edd")) |X 12 -1e "4 12
o d
coī dbb "e" 1 1 oī, d eσi, ddb) for all = 1,.., n and j 0,.., N - 1, where e E (0, 1). The second part of the Lemma follows the similar proof Lemma 7.1 of Allen-Zhu al. [ The last part follows from union bound and Assumption I
23

---

Lemma (Upper bound). Under the random initialization of Wo i= 0,,,, L, with probability at least 1- O(NN Mmin(mmm)) (L, )) we have
mmd - max w/ ((;) 1 (o
WWW (0(0))
Proof. For any l 1,,, L, we have
VeeLee (0)
Σ ((Wi)eeem(())
1=
Σ (l;) ((-1)
D WL hww.::/wwww..aa.aaaaaaaaa..aaaaa sos D(WWL.1De).WW
N n N
<
Σ i-1
w(t) (tt) (t, - Σ (WW.:Dig...-DD,,,WW...)) WL+19 + Ezy)) 199.
k-1
N
N
s cnd
Σ
w0t, (1; 1)-1) 1 W 1- Etil
d n
=1
$ Cyd M.V
max (1;)(11 --- Lem
d
where the first inequality follows from Young's inequality; the second inequality follows from Lemma and Lemma 7.4 in Allen-Zhu et C7> >
D.3 Proofs related to perturbation
Consider W.Per w W for 1..,, this section. We follow the same idea in Allen-Zhu et al. M to consider the network value of perturbed weights at each layer. We use the superscript "per" denotes the perturbed version, i.e..
per
Xox 1.5. = (r
$j.0
per
wper 1ij.-11- per o for l 1, L
ij.t
S(@per :ty, Xij) WL+1
We also similarly define the diagonal matrix Der for the above network.
The following Lemma measures the perturbation of each layer. The lemma differs from Lemma 8.2 in Allen-Zin Zhu et al. [1 by a scale of d For sake of completeness, we state in the following and the proof can be similarly obtained.
Lemma 6. Let w
C, d for some large C 1.
With probability at least
spp(= f)) any AW s.t. [AW] S w, we have
1. 1. -iije can be decomposed to two part per Tíjee = - ryg.. where Il's (ww and 1r (WL 112 Vlog mdd m m//2).
2. Dper Dis,ello 01m L.) and || [(D Dije)r per sg,ell (1.
3. Ier Tij.ell and l qij.e are (1 Vog md
D.4 Proofs related to the evolution of the algorithm
Lemma (Upper and lower bounds of gradient after perturbation). Let min, (t,) - to-1)0
0
L (log 2 N3d max(max, (t, [,
24

---

Consider goer s.t. |@ner - 811 s w, where 8 follows from the Gaussian initialization. Then with probability least 1 OnNN) -Maa0 ( 1)
V Vw, Frm (0Per)))
(0 N max w/ (g))
(md
(o0l------600 15-1))r, (OPer).
md 2
Vw, WWm (0Per))
N2 ((( - -10-1)00p min(( (0), Rom, (OBer))
for 8 11,,,,
Proof. Consider the following terms
N w (1;)(()) (1-1)
V(Wa)a (0) ΣΣ
ΣΣ
T1 ==1==
x L(WW.. W Wo.)) (or, WL.10,, Eit) dij.-1 ((Wagg)-)) (17)
1 1 Σ
per
(0m =
(t, -t--))
w
(We),
11 s=1s=1
x
(or,, WL+19 per + 91 (Wennse (18)
V(Ws)a (Oper)
-
ΣΣ
(11)((11111)
1=1
X[ Der D..r
( (ōt, WL-19
per lx g Lawprrogg Da20 (19)
Then
VW: Cem (0) -VPe Cer (0)
Σ
Cewi ((ee) Gem(()))
k-1
N
<
ΣΣ
Σ
(t))))(t)))))))))))))) 1
WL.1(95,L g,,t 19.,,-111
F=1
k-1
$ Co
m
n
Σ
w(tt))((; - 1 llā,, W Wz-1 (1 - -.p.))
d
1=11
N
m &C S
N
Σ
(wL Vlo logmat )
d
N3 w(tz (t,. - 10 C
where the first two inequalities follow the same as the proof of Lemma 5 the third inequality follows from Lemma 6 the last inequality follows from the definition of w.
25

---

Also, we have
v per (We).
(om V(W:) Cem (OPCr))
(t,t)(t,-4 - -
L+1 ij.L Gj Wer We.DDgeWW-DDeWWee.) W WL11 + +bg)la
Der "T. ....e
G-je-1 (wwg oit DiS0
per
n
+
w
(ōr, Wi.1g + Erg)la
(1..0-1 ((Wen..-))) 8ig--1 1 (w dewpmappe 2g1l))220)
Then
V
per W.
Com (0) Vw, Cem )
< 2
w(t,) ( - -tz-1 -10
m
( L+1 D 12 W. Weer WL Dis W W par + 61))1?
x
w -1--1))) --g--1----111...-1111111 ( - G 1 (wppar dep Das0 1
Σ (W De.. W,
WL+1 per + E1s))l
x
m N
s Cg(w2 L ld 20 ) (L 3/2 logm m 1/2
w(t,) ((t,)))(t,)))) 2 10,, WL+1 per Easl
1=1 j=1
N Σ
Cg( log mdd
m
w (t, ) (t, - 1 1o W L 1+,. per E,jl?
d
1=1
T N2 w((()) (4, - -(b-))))
s C.
Leme
(OPer)
where the first inequality follows from Young's inequality and the above decomposition; the second inequality follows from Lemma 7.4, 8.7 in Allen-Zhu et al. [T] (with Odd" m and Lemma 6 the last inequality follows from the definition of w.
For upper bound, we only need to consider (W,), Lem (0) and V V(W.), Lem (BPer). (BPPrr) By similar argument as Lemma S with probability at least 1 (nNL)e R(min(mme" / 20 (edd)) we have
yeer
o |/ Iiperr (0) =0mm=0mmmm
(md N Maxx wr,, - tt-1) Cem (0pee)..
Then
|| V w. Lem (fPer)
$ 2/| v per
W, Lem (6) 21 W. per Lem (8) V Vw, Cem (oper)
O mddaa-1 N max w/ (t;;)(t; (z-1) Rom (Oper).
3
26

---

Also,
V Fem (1
2 Vw, fer (OPet) )12
2
V Vw, Lom (0) 1 Wo. V
per Wi
Com (0) |1 per
fem (0) Vw, (Oper)))
3
20- "
=Ω
Na wo(g-) (1z² - 1) Lem (0), (OPEr))
Note when interpolation is not achievable, this lower bound is always away from 0, which means the current technique only evaluate the lower bound outside neighbourhood of the minimizer. More advanced method is needed and we leave it for future investigation.
Lemma 8 semi-smoothnesss. Let Ω. of g(), we have for all s.t. 10 g(0) w. and all gper s.t. ||0per 0| w.
With probability at least 1 ((loggm) over the randomness
Lem (00er) S (0) + (vLem (0), (0), oper 0)
V Cem(()) (0)
Σ (()(t) t5-1) -
log | -- 13
Cem (0)
Σ (()))(( (-1)),,,000 O112 ((122mdd)) ) - 0|
Proof. By definition.
Lem( (OPer) Lem (0) (V (0), Oper 0)
N Σ (1)) - -1--1 W (is)
qU - Σ
(Wpeer 13, We,.0gs.e
8-1
w (tz) - 110111111111 (1-,)) WL W - 21.2))12 2ō,,
Similar to the proof of Theorem in Allen-Zhu et al. [ we obtain the desired bound by using Cauchy- Schwartz incquality. Note. in our case, due to the order of imput data, we choose = O (d' m 2/3 in Allen-Zhu et al. and therefore the bound is slightly different from theirs.
E Proofs for sampling
In this section, we prove Theorem The proof includes two main steps: 1. decomposing (psllggr))) into the initialization error, the score estimation errors and the discretization errors; 2. estimating the initialization error and the discretization error based our assumptions. In the following context, we introduce the proof of these two steps separately.
Proof of Theorem 2 Step The error decomposition follows from the ideas in 12 of studying VPSDE-based diffusion models. According to the chain rule of KL divergence, we have
(L(psggr-6) KL (4)) Ey-pr (KL (-1)))
27

---

Apply the chain rule again for at across the time schedule (T -t DusjeN-, the second term can be written
By-pr
N-1
Bus-Pn-g [KL
11 (-1y)))
5-0
1 N-1
$ -
Σ
- -7 (Y))
2
3-0
N-1
f+
$
Σ
of
B(l: M17) (Y
N-1
Σ
|
of- (Yy v logpr-((Y)))
|=0
where the second inequality follows from Lemma
Therefore, the error decomposition writes
N-1
{
KL(PslqT-o) (L(pl)))) +
Σ
j-0 eej
Σ 3=0
of Vllooprr-r- (Yr7)- (X))
(20)
where the three terms in (20 quantify the initialization error, the score estimation error and the discretization error, respectively.
Step 2: In this step, estimate the three terms in Step 1. First, recall that pr (O, and go (O, la), hence the initialization error (L(Pl))) can be estimated follows,
KL(prl90)
(21)
where the inequality follows from Lemma 10
Hence we recover the term E, in
Next, since or is non-decreasing in f, the score estimation error can be estimated as
N-1
T+: of
Σ
V logpr-23 (2)
Jr
3-0
N-1
Σ
T-
2
E[ B[ Mg) VogpT-1; (Y;;) 133.
(22)
j-0
Hence, we recover the term Eg in
Last, we estimated the discretization error term. Our approach is motivated by analyses of VPSDEs
in B 26 We defines process L, == v log (Y)) Then we can relate discretization error to quantities depending on L,, and therefore bound the discretization error via properties {Le JossTt According to Lemma
12 we have
N-1
Σ
of El logpT (Yr7) - V VlogPr-(X))]
J=U
2d
N-1
Σ
Sr
t
2 of dudt
N-1
Σ
Jog
- E[tr-ETT (Xr-g)))
2
5-0
j-0
N,
N2
N-1
(")m 2
Σ
j=0 J
N₂
28

---

Since & ++ o2 is non-decreasigg and + E[tx(Er-(Xxx-)) is non non-increasing we have
N-1
f of du s 2d
N-1
1T-
N1 2d
Σ
T-5
T-
Σ
9
J--z
o, dt,
5=0
f=0
N-1
N2 Σ
2
oi- BI tr tatt(ttrr EIx-ry (Xr-1) ))];
15
5-0
N-1
N3 S
Σ
o /.
---
(XT-1z,)))
1-0
17
Therefore, we obtain
N-1
Σ
o7
oi. (Y1;) - [x)
j=0
5jj
N-1
T-t5
N-1
Σ J.g of (XI-z;)) ))]
2d
Σ
0, dt +
T-
3-0
j-0
or- o tt (XT-1;,)))
j-0
N-1
T-t
o, dt S² dto
2d
Σ
Nj
JT-77
1-0
N-1
Σ
o di J.y od
([LTT (XT-1)))
(23)
j=1
The above bound depends on (X))),, hence we estimate E(tr(I((X)))) for different values t.
First, we have
m²
Meanwhile,
x, [..] 1
|[[ X.|X,]11
Therefore, E[tr(E.(X:)) s min(m3} $ (1 (1-e- o )m3 + d). Plug this estimation into 23 and we get
N-1
Σ
07..811171omm---1111 (Yiz) (Y))
N-1
T-
0, dt+ S 2 dto m
$ d
Σ
Vs
J
1=0
N-1
- $555)(
( 2f oT-
5 o--10-
(m) + d)
Σ
(1 -
Fl
dd? Σ 0=0
Yj
0, dt m
So" of-ntt of
+ + + d)
N-1
Σ (1-e و.
-
04 04 ---,, ot
of-1z,, 77-17
k=1
where the last inequality follows from the definition of o, and integration by parts. The proof of Theorem 2 is completed.
Lemma Let (YeFoSIsT be the solution to 2 with f. and Pitsl (ly) be the conditional distribution of You given (Y, = y). Let (Yi) be the solution to (IT ( be the conditional
distribution Y..t given (Y, y). Then for any fixed t (0, (0,7)], we have
By-pr KL (+bv)aag + (l1)) oz E[]ss t7 (5,Y)) - (Y)))
29

---

Proof of Lemma
According to [2] Lemma 6], we have
+0 (1v)le +1|5 (-v))
$ 2-20 ʃ Přj +1/; (<lb))l log -1- (xly)
12
20 +1|;; (x)y)
Pr: +2/t (x/y) 905 +0/e7 (x/yy) )]
+ 20 Big (ogg ( s(0: -t: Yo; ) v log
logpr-((
1 $ T BT.,, . (alv) -2)
)
where the last inequality follows from Young's inequality. Therefore, Lemma 9 is proved after taking another expectation.
Lemma 10. For any probability distribution satisfying Assumption 3 and g being a centered multivariate normal distribution with covariance matrix o La. we have
KL(p* * glg) $ m
2o
Proof of Lemma
10
KL(P * gl)) s f Kl.(a(-- w)o(()))()))) = s (TT))(()
2 f d+ tr(Ta) 11v1 p(dy))
m
20
where the inequality follows from convexity of KL(-)g) and the second identity follows from KL- divergence between multivariate normal distributions.
Lemma 11. Let {XiJoSIST be the solution to with ft = 0 and Poj((->) be the conditional distribution of Xo given (X, I)). Define
mx(X.) Ex px Σx(X,) Covx (x))
(24)
Let (Yi)OStsT be the solution to with Si = O and got (+/z) be the conditional distribution of Yo given (Y, x). Define
mY,)) := ( Ex(Y,) Cov CoVx)
(25)
Then have for all i (0,7),
dmi 220
and _ EEE) 20, 20,
Proof of Lemma 11 We first represent Vlogp:(X,) and v (X,,) via mxX)) and Ex))) Since (Xe) (Xi)osesr solves (I. X, Xo + 5&E with (Xo.E) p @ N(, Ia). Therefore, according to
Bayes rule, we have
Vlogp. (x.) f (loppio(X(/)))),.xx.X
Ear-poje (1x,)) [0, (Xx x)]
= m (X,)),
(26)
30

---

where the second identity follows from the fact that Pgo(-/z) = -xx,, The last identity follows from the definition of my (X,) in Lemma 11 Similarly, according to Bayes rule, we can compute
(x(),) I ( X,)
pi())) I (x po,c(r, Xx)dr
(x()) ( = -0 o, (X,,),
f log (ex
(27)
where the second identity follows from the fact that Pojo Nax, and the definition of (x()) Lemma
According to Bayes rule, we have
1 1|,,- - =x| )p(dr
(ojo(d a exp(
--pp--1 2 0i
and
| x
ooj (dx(())) z-1 exp(
)p(xx)
07 1-
1 _ -
1 exp(
|0|
(x, (d)))(d))
= - exp((h( exp(h(()))p(d)))
(28)
where Ze f (x)))(dr) (random) normalization constant. From the above computations, we can see that Goje (dr Yi) PojT- (dx|X(() for all [O, Therefore, we have
m.(()) = Ex- (1Y) [X (XT-t). Σ,YY,) = Covx (xx. (Xr-(xr=);
where the identities hold in distribution. Therefore, to prove the first statement, it suffices to compute dr,, (Y+). To so, we first compute (1), ol h()))) dZ, and dloo Z..
33 - dhv(a) -
o Yi)dt + (0- dYi).
(29)
According to the definition of Y, and 26 we have
dY. 20 logrr- + 2 20
= 2V -Y, dt -207
Therefore
(30)
Apply
29 and 30 and we get
dz. f exp(h(()))) d((())).(=)))))pdz) (agthce))(cchh)))////ddd))ddddd
33 -- y (1x) ( - my (Y)))
=
+ F (
dy
(31)
and
dlog U, = Z, dZ, 2
= - my (Y, ( ( -07.,77..11. (32)
31

---

If we further define (Y,Y Sole(d2(Y))
Txp exp(hh(xx)). We have
p(dx)
dY, dexp(log R.(Y))) R.(Y))d(log R4 (Y.)) 1 + R, (Y.)]
=-Ro(Y)dd(ogg 7+) (a()))d (((())d(A,xx)) - log Ze, (x))) log 7c] (33)
With 29
and
we have
dmp (Yi)
r TR (Y))(())
1 = f x( d(log Ze) + dh((x) + / dd(h(()) - log Zz, 1((x))) log Zo (dx)()))
20
(34)
where most terms cancel in the last identity. Therefore, the first statement is proved. Next, we prove the second statement. We have
d [[E((Y))] (XT=)-(Xx-))-
d
dt B E(ET-((XT-o))
dt d
E Ey-pp
Bo 1= 2] m. m).)
dt d
10 [1 2]
ml (Y; 2]
dl
dl
- 2m (Y, (Y, di) dl.m m.() 2 E[ (Y:)
where the second last identity follows from (34 and the last identity follows from the definition of क.. Last, we reverse the time and get
[X)]
20,0, (x,,)
dt
The proof is completed.
Lemma 12. Under the conditions in Lemma 11 let (Yijusier be the solution (2) with fo O.
Define L, := v log then for any I E (L, (471)) we have
[- 2 oT du कं Elu(Er ([x(Ex-;(XXx-))) ((()))))))
E - Lez [=]]==2 2d
,,
Proof of Lemma
12
First, according to the definition of L, and Y,, follows from Itô's lemma that
dL, v'loopT- Vog.T- + (Y)
(35)
d( V log PT-1) (Y,)
+ ((VlogpT-(Y)))
(36)
dt
(37)
where the last step follows from applying the Fokker Planck equation of I with ft = 0, i.e., O.p. of Apt. Most of the terms cancelled after applying the Fokker Planck equation. Now, for fixed 0 and / >> s, define E,,t := El La 1.] Apply Itô's lemma and 35 we have
d...t 2E L,. dL,)] + d[L.].
-2E[(L, L.. vog.. 207 (o)
(38)
32

---

where | denotes the Frobenius norm of any matrix A. According to (27). we have
dE,,
-2ff. dl
(ar (X--i))
where the last identity follows from the proof of Lemma Therefore, for any Ltj (7.1). have
J or- { (xxx
E 2d
{" 0 ou Elr(TT
5j
2d I' of JT of du 11
[- E(
1o (XT-r)))
E(tr
Elt
+ 4
To T.
.I 6
B (Str
2d
-
["
-. B[tr(Br-ttErr ((Tttrrrr(XXT=)1 (XT=1))]
The proof is completed.
F Sampling error for Gaussian data distributions
In this section, we consider special case when the data distribution is mixture of Gaussians, i.c.,
p(x) = (1; oTa)
(39)
where N (a;:m, the density of Gaussian random vector with mean m and covariance o² T In this case, the score function v log::(2) (x) can be explicitly calculated from any > 0, see Lemma
14 Therefore, the sampling process (II) can be implemented with zero score estimation error via the Tollowing piecewise SDE: for any t [tj ..,1).
dY. 20 llgPr-e (Yg )dt + 2
(40)
The iterates, (Yrz), are all Gaussians with explicit means and covariance matrices, see Lemma quantify the quantity, KL(PslQT-6) in Theorem 2 explicitly since both
As consequence, distributions are Gaussians.
Lemma (KL-divergence error for Gaussian data distribution). Assume the data distribution has density given by (39). Let (Yr; >o be defined by (40) with initial condition Yo Ia)
Denote (Y,) V.r for all 0 st T Then
d (E, (Ea-1- 1E.-1-1 1-g Ea) |m|
(0² 20² = o2 + oz
kL(polqr-6)
E.,
(41)
2
where Eg is a positive constant depending on the variance schedule (ōT- or given by
E, ( (02 + 2222 1=
N-1 (o² o (0) -
(02 -- )²
33

---

Proof of Lemmal 13 ps = p*p (O, (m, and Qr-5 Law(Pr)) = =N(,N, Σn)
with (mN, EN) given in Lemma 15 Therefore, we have
(1(pl KL(N(m, (o oa)+) ΣN))
det(EN)
d
log
tr( +0,) EN)) )+ (mN m) ''N' (m mN)
del((o2 σ [a)
2
(o 00²+0
N-1 (a - 07-05
d
log
Σ
2
(o2 (
( 0 T-1 901
)²
2
d (o o
N-1 (o
-,
2
T-
2
(o² 02
(o² + + I- i1l
)
N-1 (o (--
---
)
2 7
(0²
o 2
301
|m|
(02 T- }
Lemma 14 (Explicit score function for mixture of Gaussian target). Assume the data distribution has density given by (39), then the score function is given by
I
logpe(
(42)
02 + 07
Proof. Since the forward process ① with ft = O is the just process that keeps adding noise, the density Pt along the process is convolution between data density and Gaussian density with mean zero and covariance o? La:
(=)
oa)).).
Therefore, we have
(p.(2) (2 + -0?))
4/2 exp
1x. ar-ml² )(
2- 02 o²
2( + 03)
x; m, (o² oi)
02 + ,
42 follows directly from the above computations.
Lemma 15 (Gaussian iterates along the trajectory). Assume the data distribution has density given
by 39 Let (Yrz) be defined by (40) with initial condition Y, ~ (0,00 12). Then for all 0 s j s N,
Y2 ~ ~N(m,, Σg) with
2-_ to
+ -
mj
m.
(43)
02 + 01-
(0² + + 200
(o o (' -
Σラ =
la.
(44)
(0² 0- T-
(02 + o7 of )2
1
Proof of Lemma 15
According to Lemma
14
40 can be written as
Y - m
= -207-- 0² + 02 :7-15
/
207
34

---

which implies that for any jE 0, ,, 1:
o -
dl
Y1,,, Yr- = -2
(Y²--)) -m) + V1 J,j dt
2
02 o7 5-
,7 07 (-) (Yr-- -m) + v 1---,
0 T- j+1
Uf-1s
2² + 0 T-1
T- tt V15
T-
2
Yiz, (1
m v T-
56 -
Ust1
22 + 1
02 200 o
(45)
where (U)) are i...d standard Gaussian vectors in R. Since Yı5 is Gaussian, by induction, we prove that Y2 is Gaussian for all j = 1,,",,,, .N. Denote Ye. ~ ~N(m,, 2z). According to (45) and the independence between Us+1 and Yoy, we have
2 T- 1j T- 501
07 T- j+1 my
M3+1 (1
m.
r2 + T--5
r2 + o
0² T-
s+1 (m- -m),
Mj+1 m
0² -
0² o - (mo m) m
07 -- -
m, 02 1-15
0² + o
m.
Again, according to 45 and the independence between Uj+1 and V17, we get a relation between consecutive covariance matrices:
Ejs1= = (1
0 T-5
T
jol
2
Eg + (0)-1 - 22 -1,- )aa
σ o7
T
0 T- 8- ldi
Ej+1
(o2 07 T-1j11
)2
(02 + 2
(02 T-1
T-
Σnj
E
j-1 T- -0 ldi
(02 + + )2
(02 +220f 7
(02 o7 )2
T-
Σ, (02 + (0-1(00-0
(0²
)207
j-1 (02+ +
)0 -ot-(a)) Jo: (02 07 o )2
1=0
G Full error analysis
Proof of Theorem 3 We only need to deal with Es. By applying the same schedules to training
objective, we obtain
N-1
((tt-))(t----tt - 1-- 1
Es = Σ
Ex, Eelle, s(f: XeN-,) + El
5-0
w(t-,))
JIN-s
N-1
Σ
w - (N-
j- w(t w(NN=s))
lix-1_
< maxx
(C(W) + C).
w(tN-j)
35

---

Together with
[(W(k))+ + + Lem (W(k)) + (0m -0") + /eem (W(K)) - -Lem(0)) (0))
+ +12(0") - -[(=)) ((0=))++120) +1(00) C
we have the result.
H Proofs for Section 4.1
H.1 Proof of "bell-shaped" curve
Proof of Proposition 7 Fix Ta Eays Oin- By definition of the network S(0; with respect to Xij.
Xzy), (,::),,,))))) continuous
For Xıj = Iz or Eg., and thus S(0: t X1)) is also continuous w.r.t. or,, Also, since oz, E =[0,0ew] there exists Mo > 0, s.t.,
S(0: j. n; + Ety) [- Mo]
Then for any 6 > 0, there exists 8= f VdMa 0, s.t., 0 S 0t, < 51, we have
oo S(0:t,, 24 +z, Eıil 2 ||ز - S(;:t, + 5;)5
2 |ka vdMoor,
2 |k: c1.
For 2, by the positive homogeniety of ReLU,
Iz Eis S(0; f. v, + Ei,) oz, 0,t1. Tt,
Consider oG, 2 M, for some M > 0. Then
|ō, + 0r, 5b) Ssil = o2 S 0:tj: Tt
Di + Eij
+ Eij
2
0:
41 Eij
||
Tz:
2² S 0: Tz,
Etj
- 11
(46)
For any y D(M), where D(M) = (yy R v. [(56)s -1xx:) M/M ())) + +1xx)) -s=
1, d}.
|5(0;4,, v\ 2 2115(0;(,, [S(0: (.))l Since S is differentiable a.e., by the fundamental theorem of calculus.
(47)
S(0:t,,)) - S(0;t,, f (O:t,,,
Then
1
|S(0: f) S(O:;),, s M
M1.
(48)
where M₁ mma.aaaaaa (x;), essSup (0:
for some fixed 0 M2 M.
Combining (46).(47) and (8 we have
Mj
| S(0; 4, E,g + Eiil 2 M (5(0044,Es))
||E
M
M1
[sill
=22
[1S(0;4,
M
M
Mi+ NM de2+
Then V 62 > 0, there exists M max
.M2
> s.t., when oe, > M, we have
20a
10, Gis Eilll M((((((0:),,s)))] -22)
36

---

H.2 Proof of optimal rate
Proof of Corollary 2 If (I(((() : s for all i, and k > K, then by Lemma 1
and 7 we choose the maximum ;(0());,,)) for the lower bound, which is of order (E)) away from the other (( Therefore, can take jk) arg max, f f((())2,)) ;i,j) and absorb the (c))
error in constant factors. Then the result naturally follows.
H.3 Proof of comparisons of Es
Recall that the training objective of EDM is defined in the following
1
(lon w- Tomean
σ² + odata
Ea-peran (G) y1²
aPd
Bxx (0s(;,]) + 2 do.
Z1
020 data
Let Bg Ci BRBDM i.e.,
(log 0e, Fmocan 2
w(()) (t, ==-1)= = Cc
2P2
0t
by data
o
ot,
(loRo15 Pmean)?
0 + daata
w(t,) C1
2p2
ot,
1j 1--1
01 data
EDM. Consider o; and tj = @ (o - 110 N-1 0
for j = 0, N. N. Then
N
(logd - -Pmoon )
t 2 Jaata
w(tt)) Ci t, - tz-1
to
C1
1/p T
1/p N-1 ) min
(oll (o g2/p1 N-j+1
Omax
N
(ngg (nn(0010 1/0 -( 1/2 -o
Up xx -
(o (5
F T )
N-1 2p
252
min
data
8
data
of Then the maximum of w(t,)
4 w(t,)
appears at =
(Prenn log Rmaxy
max w(t))
of w(t))
max
t,
Fmax of dsta
Ci (o max + odua)
Fmax
1/p max
m
3/p
N
σ 2/P \.
min
Song et al. [6]. Consider o, Vi and t, 52
for j 0, .......... Then
(log /55 )
VT
1g o data Vijaaata data
w(tt)) = C1
2n2 and
17
1--1
Emin 7max
N
N-1
mmax
o data
= C1
2P2 sud
max
N-1
N-1+1
22
02
min
22
@ mni
data
max
G
max
Then
(Prsn mmax.
1
1
σ max data e
202
o2
I/N
max
max
mmax Tmax 72 max
w(t (t,)
2w(t,)
C (1 + 22
37

---

I Proofs for Section 4.2
I.1 Proof when F, Ep dominates.
Under the EDM choice of variance, o, = / for all e [0. 7]. and study the optimal time schedule when ED Ey dominates. First, it follows from Theorem that
N-1
75
11
Er + En
d
T2
(T-15)
1=0
2
13 T-r (7 11))
(m² d)( T- 21 (T - (T (T-
Σ
Σ
(T - 17)3
Based on the above time schedule dependent error bound, we quantify the errors under polynomial time schedule and exponential time schedule.
Polynomial time schedule. we consider T 7-1 (81/a + (N -)) with h TV and a 1,
N
7j = = (N- jj h) h for some 9 (0, We have 75/h a(T - -17) a=
and
ma
da²T*
2T
T 8 N2
E1 + ED S T
(m² + d)
S
8 N
Therefore, to obtain E, En 5 e, it suffices to require T O
m2 23rz
and the iteration complexity
N
m2
m
For fixed m2, 8 and E, optimal value of that minimizes the iteration complexity is = In( ae
Once we let mmin Fmax
0 mg and a == p, the iteration complexity is
27/2
N a mi d 12(
max Mp g2 maxx):
Mmin
and it is easy that theoretical result supports what's empirically observed in EDM that there an optimal value of p that minimizes the FID.
Exponential time schedule. we consider 7/5 -(5) with k
(n(T/6)
we have
N
E + ED $ m dln(T/8)2 T2
+m + d) In IT(3)))2 In(3/5)3
+ N
+
N
N2
Therefore, to obtain E₁ Ep c, it suffices require T = O my and the iteration complexity
= (2(
m +
In
m2
When m2 ((vd), the exponential time schedule is asymptotic optimal, hence is better than the polynomial time schedule when the initilization error and discretization error dominate. Once we let ). the iteration complexity
mins mmax O f1/2
m d
N 22
In
Fmax
max
d
Now we adopt the variance schedule in 46), o, Vi for all t [0,7], it follows from Theorem that
m N-1 77 (m + +d) Σ
E1 ED S
d Σ 7-0 (T 1T-
Σ
T- (T -1-1 13 T- (T - )2
Polynomial time schedule. consider T - (81/2 +(N -)) with h 8/0 and
7i = a(81/o + (N -- -00))a-11 h for some 1 E (0, 1). We have 9s/h a(T - t a1 and
2 da²T E ED
(m² d) TT
8.N
8 N'N
38

---

Therefore, to obtain E + E.D & e, it suffices require A m
and the iteration complexity
N sa²
m2
2 d
òs
mi Once we let o² min T' o max = 0 =
and a = p. the iteration complexity is
N my d p²
max 2/0 5
Omin
Compared to exponential time schedule with the EDM choice variance schedule, this iteration complexity is worse up to a factor
Cuuax 1/p mmin
Exponential time schedule. we consider 7/3 = -1) with K
ln(T/5)
we have
N
m
dln(T/5)2
(m² d) In(T/0)2
E ED
T
N
N
Therefore, to obtain Es + ED S e, it suffices to require T O mi
m and the iteration complexity
m2 d
N 02
Once we let _² - = 0² max
0 mi and p, the iteration complexity is
N 2 m d
maax
In
0
max)
d
min
Compared to exponential time schedule with the EDM choice of variance schedule, this iteration complexity has the same dependence on dimension parameters m2, d and the minimal/maximal variance T mins maax-
Optimality Exponential time schedule. For simplicity, we assume m Od). Then under both schedules in [30] and [46], Eys only dependent on T, and are independent of the time schedule. Both Eps satisfy
N-1 Σ 5=0
ED S d
Se
(T t )2
Let Ty In T-r e (0, o0). Then T-E 1 J and Eser-GerT) In(T/8) is fixed. T-
Since (1 2) is convex the domain (0, according the Jensen's inequality, 2
Eser-y <T' (²- reaches its minimum when Ty are constant-valued for all j. which implies the exponential schedule is optimal to minimize Ep, hence optimal to minimize Ep + Ej.
39

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No]" provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] x or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We state it in Section and Appendix A.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 1 and Appendix A.
40

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yea Yes]
Justification: See Appendix and Theorems in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification: No experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
41

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
No experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ipps.c//ublic/guudeee http://s//ooooeuuuuuissssssooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
42

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
43

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp::/neuurpps httpppp.......bbiiiiiiieeeeddaad
Answer: [Yes]
Justification: It has been confirmed by Ethics reviewers.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Section 1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
44

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We show new methods and theorems.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
45

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
46


Reviews:
[{"review_id": "nAY2VMu0ku", "reviewer": "Reviewer_5uDw", "summary": "This paper provides a full error analysis (considering both training and sampling) for a class of score-based SDE diffusion models, and using the results to understand why and when certain time and variance schedules are preferred.", "questions": "- In Remark 1, it was stated that the KL divergence can be explicitly computed when the data distribution is Gaussian, would you provide such result in the appendix?\n\nMinor remarks:\n- typo in line 16: variance -> various \n- missing lemma number in line 201", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The theoretical contributions of the paper are excellent\n- Overall the paper is well organized and the notations are clear", "weaknesses": "- The stated theorems are quite dense and are not easy to parse. The paper would benefit from having informal version of these results highlighting the key qualitative features\n- Only an SDE based sampler (the exponential integrator scheme) is considered, while comparisons with other samplers (such as the Probability Flow ODE) are missing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for all the helpful advice and comments and deeply appreciate the positive evaluation. \n\n>Weakness 1: The stated theorems are quite dense and are not easy to parse...\n\nWe greatly appreciate the comment and will add more intuitive explanation or informal versions of results in our next version.\n\n>Weakness 2: Only an SDE based sampler (the exponential integrator scheme) is considered, while comparisons with other samplers (such as the Probability Flow ODE) are missing.\n\nProbability Flow ODE ([1]) is indeed another implementation. It reverses the diffusion as an ODE. Because of that, common tools like Girsanov no longer applies, and its analysis requires different approaches. However, there are existing anlyses (e.g., [2,3]) that assumes score is already learned within $\\epsilon$ error, and these approaches can be used to replace the sampling component of our full analysis. We also note so far there is no verdict on which one is better, even just for the sampling part. \n\nIn this work we simply chose to analyze the VE-SDE implementation as it was used in many celebrated papers [4,5]. But we agree that other sampling strategies could also be interesting future investigations!\n\n>Question 1: In Remark 1, it was stated that the KL divergence can be explicitly computed when the data distribution is Gaussian, would you provide such result in the appendix?\n\nWhen the target distribution is a Gaussian with mean $m$ and covariance $\\sigma^2 I_d$, the score function can be eaxctly computed explicitly. In this case, $p_\\delta$ and $q_{T-\\delta}$ in Theorem 2 are both Gaussian and $\\text{KL}(p_\\delta|q_{T-\\delta})$ satisfies:\n$$\n\\text{KL}(p_\\delta|q_{T-\\delta})=\\frac{d}{2}\\log M -\\frac{d}{2} +\\frac{d}{2M}+ \\frac{\\lVert m\\rVert^2}{(\\sigma^2+\\bar{\\sigma}_T^2)^2 M }, \n$$\n\nwith $M=\\frac{\\bar{\\sigma}\\_T^2+\\sum_{k=0}^{N-1} \\frac{(\\sigma^2+\\bar{\\sigma}\\_\\delta^2)^2}{(\\sigma^2+\\bar{\\sigma}\\_{T-t_{k+1}}^2)^2}(\\bar{\\sigma}\\_{T-t_k}^2-\\bar{\\sigma}\\_{T-t_{k+1}})}{\\sigma^2+\\bar{\\sigma}\\_\\delta^2}$. We will add the detailed computation in the appendix of our updated manuscript.\n\n[1]  Y. Song et al. “Score-based generative modeling through stochastic differential equations”. In: International Conference on Learning Representations. 2021\n\n[2] Chen, Sitan, et al. \"The probability flow ode is provably fast.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Li, Gen, et al. \"Towards faster non-asymptotic convergence for diffusion-based generative models.\" arXiv preprint arXiv:2306.09251 (2023).\n\n[4] Karras, Tero, et al. \"Elucidating the design space of diffusion-based generative models.\" Advances in neural information processing systems 35 (2022): 26565-26577.\n\n[5] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" arXiv preprint arXiv:2011.13456 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYZGThNlnQ", "reviewer": "Reviewer_Ti47", "summary": "This work studies the training and sampling process and then achieves the end-to-end analysis for diffusion models. For the optimization process, this work uses an over-parameterized NN and gradient descent to prove the convergence rate. For the sampling process, this work provides VESDE results and explains the great performance of SOTA VESDE. After these results, this work explains the “bell shaped” weight function used in application from the theoretical perspective.", "questions": "Please see the Weakness part.\n\nQuestion 1. Can you relax the high-dimensional data assumption by using the low-dimensional manifold assumption [3]?\n\nQuestion 2. The guarantee in [4] is a pure $W_2$ guarantee and can not be directly compared to the $KL$ guarantee in this work. Can you discuss it in detail?\n\nComment 1: It would be better to add a conclusion and limitation part at the end of this work.\n\nTypo 1: In line 201, the reference Lemma is unclear.\n\n[3] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics (pp. 1648-1656). PMLR.", "limitations": "This work does not discuss the limitation and societal impace in a independent paragraph.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThis work is the first one to analyze the optimization process with a deep NN and prove the exponential convergence into a neighborhood of minima. \n2.\tThis work makes the first step to explain the “bell shaped” weight function and the great performance of the state-of-the-art VE-based model from the theoretical perspective.", "weaknesses": "Weakness 1: The high-dimensional data assumption is strong. It would be better to discuss the application of high-dimensional data.\n\nWeakness 2: The technique novelty of Theorem 1 is needed to be discussed in detail. Section 3.1 claims that this work develops a new method for obtaining a lower bound of gradient. However, it does not discuss this method on the main page. It would be helpful to discuss the technique novelty in detail.\n\nWeakness 3: As shown in Section 3.1, this work mentions that the data separability assumption (corresponds to parameter $\\delta$ in [1]) can not be directly used due to the property of diffusion models. This work replaces $\\delta$ with $t_0/T$ when choosing $m$, which seems to be related to the $E_I$ term in Theorem 2. The technique challenge is unclear after replacing $\\delta$ with $t_0/T$. \n\nWeakness 4: As shown in [2], their results can be straightforwardly extended to any linear SDE, including VESDE. Hence, it would be better to discuss the technique of Theorem 2.\t\n\n[1] Allen-Zhu, Z., Li, Y., & Song, Z. (2019, May). A convergence theory for deep learning via over-parameterization. In International conference on machine learning (pp. 242-252). PMLR.\n\n[2] Benton, J., Bortoli, V. D., Doucet, A., & Deligiannidis, G. (2024). Nearly d-linear convergence bounds for diffusion models via stochastic localization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for all the helpful comments. We especially appreciate the comment \"...the first one to analyze the optimization process with a deep NN...the first step to explain the “bell shaped” weight...\". Please kindly see our itemized replies below:\n>Weakness 1\n\nWe agree that this assumption is strong. Therefore, we improved our convergence theory, and the new version now uses a much more relaxed assumption, where the original version $$d=\\Theta(m)$$ is replaced by\n$$d=\\Omega({\\text{poly}}(\\log (nN))).$$\n\nNote in the old version, the input dimension is the same order as the width of the network $m$, while in the new version, it only needs $d$ to be not too small, or equivalently, the number of data points $n$ and the number of time points $N$ should not be exponential in $d$. Please also see a new version of Theorem 1 in our global rebuttal.\n\n\n\n> Weakness 2\n\nThank you for pointing this out. \n\nRoughly speaking, we first decompose the gradient into terms based on one data point that has the largest loss value at the current iteration, and the rest $nN-1$ samples [1]. Then the gradient can be written in the form $w^\\top v+w^\\top u$ for some $w,v,u$. The main novelty of our proof lies in analyzing the angles between $v,u$. This allows us to show the probability of $w^\\top v\\ge 0,w^\\top u\\ge 0$ to be roughly of the order $d^{\\frac{-3}{8}}$. Finally, this leads to a lower bound of the gradient by $w^\\top v$. The proof is more involved but the above is the main technical innovation.\n\n\n\n> Weakness 3\n\n\nWe apologize for a critical confusion: the $\\delta$ used in [1] and the $t_0/T$ in our paper serve different purposes, and we did not replace $\\delta$ by $t_0/T$.\n\nAs the expert reviewer knows, $\\delta$ is a lower bound used in the data separability assumption in [1]. However, in our paper, our proof does not require such an assumption due to our new method and the diffusion model settings, and therefore there is no counterpart of this variable. More precisely, in [1], they require the output data to be in $o(1)$; in this case, the input data have to be well separated so that the gradient is locally strongly convex (obtaining the lower bound). In contrast, denoising diffusion models require Gaussian outputs that cannot be small, which help contribute to the strongly convexity and relax the requirement on the input data.\n\n\nRegarding $t_0/T$, it adds no extra technical challenges and is merely due to the weighted $\\ell_2$ loss for diffusion models, which depends on different time points and variances, compared to the standard $\\ell_2$ loss. Such weighting will enter the constant factors of different bounds and consequently cause the dependency in the width $m$. \n\n\n\n> Weakness 4\n\nResults in [2] indeed can be extended to VESDE, for the sampling part where discretization error is analyzed. However, we respectfully think the detailed calculations still need to be done, and several not-so-straightforward steps we encountered included:\n1) bounding the initialization error: [2] considered the VPSDE and uses KL-decay along the OU-process which is no longer true for VESDE. Therefore we instead handle the initialization error by the convexity property of KL divergence (Lem.10). \n2) bounding the discretization error: we considered a general time-schedule, which makes it harder to bound the term $N_2+N_3$ in line 758-759. We utilize the asymptotic estimation $\\min(\\mathrm{m}_2^2, \\bar{\\sigma}\\_t^2 d)\\lesssim(1-e^{-\\bar{\\sigma}\\_t^2})(\\mathrm{m}_2^2 +d)$ to obtain the upper bound. \n\nThanks to the reviewer, we will include more details of these in a revised version.\n\n\n\n\n> Question 1\n\nWe deeply appreciate the reviewer for proposing a nice possibility to improve our result and will carefully evaluate it. Meanwhile, as indicated in the reply to Weakness 1, we have already improved our results without additional assumptions.\n\n> Question 2\n\nThe reviewer mentioned [4] but didn't give the reference. We are guessing that the reviewer might mean [24 in our manuscript]. $W_2$ is in general not comparable to KL unless if we assume $p_{T-\\delta}$ in Thm.2 satisfies Talagrand inequality (implying $W_2^2\\le C\\cdot\\text{KL}$). The comparison to [24] is actually not fair as [24] assumes strongly log-concavity, which is much much stronger than our situation, where the data distribution can be arbitrarily non-log-concave and multimodal. In the updated version, we compare to [6] (Cor.1), whose convergence is established in TV for compact supported data distributions. In this case, we can see our result has better dimension dependence by Pinsker's inequality $2\\text{TV}^2\\le \\text{KL}$. \n\n\n> Comment 1\n\nThank you very much for this comment. We will add it to our next version.\n\n> typos\n\nWe sincerely appreciate!\n\n\n\n**Additional References**\n\n[5] Eldan. Taming correlations through entropy-efficient measure decompositions with applications to mean-field approximation. PTRF'19.\n\n[6] Yang et al. The Convergence of Variance Exploding Diffusion Models under the Manifold Hypothesis.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xvY3F8691r", "reviewer": "Reviewer_RRb9", "summary": "This work develops theoretical bounds on the training error and sample complexity of score-based diffusion models, which further imply a complete convergence analysis (by combining the both). In addition, the derived bounds shed light on the efficient hyper-parameters selection, including re-weighting functions, time discretization schemes and variance schedules, supporting former empirical works in a principled manner.", "questions": "1. Please provide more details of questions raised in the weaknesses section above.\n2. The scale of model initializations is near-zero (see  Line 181-182), which seems to be one of the typical configurations in the over-parameterization regime (lazy training?). Is this setting reasonable and representative in practice? It would be better to provide (numerical) examples about this, particularly in this generative modeling case. \n3. Details 1: In Line 201, \"... high probability (see Lemma..).\" \n4. Details 2: What is the definition of $a$ in Theorem 1 (see $d^{2a-1.5-4c}$)?\n5. Details 3: In Proposition 1, there seems to be a confusion on the mixed use of $S$ and $S^*$ (particularly, Point 2). \n6. Details 4: In Line 297-299, it states that $\\epsilon_n$ (the distance between the empirical loss and population loss at the GD outputs) can be estimated by central limit theorem. Can the authors provide more details on it (since the learned model parameters are *dependent* of data)? \n7. Details 5: In Table 1, why is the exponential schedule selected as $\\bar{\\sigma}_t=\\sqrt{t}$? In VE-SDE, $\\sigma_t$ is an exponential function of $t$ (and hence $\\bar{\\sigma}_t^2:=2\\int_0^t \\sigma_s^2 ds$). Note that in this case, Assumption 2 (Point 1) may not hold.", "limitations": "Based on the developed theory, it is worthy to further explore new weighting functions, time discretization schemes and variance schedules to better tune diffusion models.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The problem setup is quite general, at least at the current stage.\n2. This work provides a comprehensive and rigorous convergence analysis of diffusion models for *both* training and sampling. The analyzed score network also extends former theoretical literature. \n3. The obtained mathematical bounds also imply beneficial insights to guide the hyper-parameters selection, which coincides with other experimental literature .", "weaknesses": "1. The paper is organized in a almost technical way, and some details are not easy to follow. It would be helpful to improve the readability by adding e.g., (key) notations collections and formulation illustrations. Also, there is no conclusion section.\n2. The derived bounds in main theorems are too technically complex. Is it possible to simplify them to summarize the core algebraic dependence on hyper-parameters (including the data dimension $d$, sample size $n$, model capacity $(m,L)$, discretization steps $N$)? \n3. There are no experiments to support the theory. Particularly, is it feasible to numerically verify the exponential convergence in Theorem 1? \n4. Are there any differences on the training between over-parameterized neural networks applied to normal supervised learning, and here in the generative modeling (as the score network)? Note that this work selects the *same* feed-forward network (without biases) for *all* time steps (see the equation between Line 173 and 174), ignoring the time information in the architecture of score networks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank the reviewer for all the helpful comments and sincerely appreciate the positive evaluation. Here are our itemized responses:\n\n\n> W1\n\nWe sincerely appreciate your comment and have done a major revision of the paper in accordance.\n\n> W2\n\n\nThank you for this advice. It is in general hard to simplify the dependency without choosing time and variance schedules. Therefore, in the following, we show the upper bound in our full error analysis under the time and variance schedules of [31] (a current gold standard in practices): \n\n$KL(p_\\delta|q_{T-\\delta}) \\lesssim  \\frac{\\mathrm{m}_2^2}{T^2} +\\frac{d a^2 T^{\\frac{1}{a}} }{\\delta^{\\frac{1}{a}}N}+(\\mathrm{m}_2^2+d)( \\frac{a^2T^{\\frac{1}{a}}}{\\delta^{\\frac{1}{a}} N}+\\frac{a^3T^{\\frac{2}{a}}}{\\delta^{\\frac{2}{a}}N^2})$\n$+\\frac{C}{N}(C_1+(1-C_2 h (\\frac{md^{\\frac{a_0-1}{2}}}{n^3N^2}))^K),$\n\nwhere $\\mathrm{m}_2^2$ is the second moment of the data distribution and it is of order $d$ if the data distribution is Gaussian; $a$ is chosen to be 7 in [31]; $a_0\\in(1/2,1)$; $C,C_1,C_2$ are some constants; $K$ is the total number of iterations.\n\nPlease see upper bounds of sampling errors under more schedules in Appendix F1. We will try to make it less technical in our next version.\n\n\n\n\n\n> W3\n\nWe would like to thank the reviewer for helping us strengthen the paper. The figure in the attached pdf shows how the actual training is consistent with our theory. More precisely, the $y-$axis is in $\\log$ scale; the training loss decays in a nonlinear way but it is clearly bounded by the exponential bound.\n\n\n\n\n> W4\n\nThank you for a very sharp observation. There are indeed time information in the model of our paper compared to the previous overparameterized neural networks. \n\n\nIn short, $X_{ij}$ in our theory is actually an augmented version that concatenates both the original $X_{ij}$ and $t_j$. This way we no longer need to consider separately an additional time variable. More precisely, due to the data structure $X_{ij}=x_i+\\bar{\\sigma}\\_{t_j}\\xi_{ij}$, if we take $(x_i)\\_d=0$ and $(\\xi_{ij})\\_d=1$, then the last element of this input vector is just $\\bar{\\sigma}\\_{t_j}$, which is a function of time. This can be seen, for example, in Table 2: practical choices of $\\bar{\\sigma}\\_{t_j}$ include $\\bar{\\sigma}\\_{t_j}=t_j$ and $\\bar{\\sigma}\\_{t_j}=\\sqrt{t_j}$.\n\nHowever, we now realize this is a source of confusion. Apology, and it will be clarified in our next version.\n\n\n> Q1\n\nKindly see the above.\n\n> Q2\n\nThank you for the questions. Such initialization is indeed what is used in practice. For example, in EDM [31], which is a gold standard for the design of diffusion models, they provide several choices for initialization, including Xavier and Kaiming initializations using both uniform and Gaussian distributions. When initializing the weights by Gaussian, they rescale the standard normal by $\\sqrt{\\frac{2}{d_{\\rm in}+d_{\\rm out}}}$ and $\\sqrt{\\frac{1}{d_{\\rm in}}}$ for a weight matrix of dimension $d_{\\rm out}\\times d_{\\rm in}$. In our paper, $d_{\\rm out},d_{\\rm in}$ equal to $m$ or $d$ for different layers. Therefore, our initialization of all the weight matrices ($\\mathcal{N}(0,\\frac{2}{m})$ and $\\mathcal{N}(0,\\frac{1}{d})$) matches the order of at least one of those used in practice. \n\n> Q3\n\nApologize for the typo. This should be Lemma 4. We will add it in our next version.\n\n> Q4\n\nWe deeply appreciate that you carefully checked the details and apologize for this typo and the confusion caused. It was $a=1/2$, but we now further improved our rate and this $a$ is no longer needed.\n\n> Q5\n\nWe sincerely apologize for the typo. All the $S^*$ should be just $S$. We have already corrected it in a revised version.\n\n> Q6\n\nWe deeply apologize for this confusion and thank the reviewer again for the careful review. This is actually a mistake that we forgot to correct. The \"central limit theorem\" part should be the law of large numbers, i.e., this $\\epsilon_n$ is the statistical error that converges to 0 as $n\\to \\infty$. We currently do not have refined statistical error estimation, and this limitation will be clarified in a revision. \n\n> Q7\n\n\nThe exponential schedule is the popular version from [52]; it is called so because the time schedule $(t_k)$ decays exponentially fast (see the last column of Table 1). Regarding its noise schedule, since its forward SDE is $\\mathrm{d}X_t= \\sqrt{2}\\mathrm{d}W_t$, $\\sigma_t=1$. Therefore, the variance $\\bar{\\sigma}_t^2=t$, which is not exponential. The Assumption 2 Point 1 is satisfied since we can always choose an appropriate scaled weight $w(t)$ in the training step.\n\n> Limitation\n\n\n\nWe agree. This is actually nontrivial given the complexity (due to its dependence on many design factors) of our bound, and we hope the reviewer could consider the length and scope of a conference paper and allow us to do it properly in a future work because we'd also like to comprehensively verify everything empirically. \n\n[31] Karras et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cq8EEyFk3r", "reviewer": "Reviewer_1csf", "summary": "This paper analyzes both the training and sampling errors of diffusion models. The analysis sheds light on two practically relevant design choices, which are the noise level weighting (during training) and discretization (during sampling).", "questions": "- I don't understand the sentence in lines 296-299, which mentions empirical and population losses even though the paper does not tackle the generalization error. Also, I don't see how the central limit theorem applies here, as individual terms in the empirical average over the training set are not independent since the network parameters themselves depend on the training set. Could the authors please clarify?\n- Do the authors believe that their work can inform practical decisions? If yes, what new predictions can be made using their theoretical results? If no (which does not necessarily imply the results are not significant), I think the paper should be more honest about this.", "limitations": "Contrary to what is stated in the checklist, limitations are not discussed in the paper. There is no discussion section.\n\n- A first limitation is that only the approximation and optimization errors are studied, and generalization is not tackled. This means that the score network learns the empirical distribution of the training set, and thus reproduces training data points during sampling. This is only briefly mentioned in passing in the middle of the text but should be stated in the introduction (and potentially discussion).\n\n- Some assumptions are questionable. First, the first and last layer are not trained and left to their random initialization. Second, and more importantly, the chosen asymptotic setting corresponds to extremely wide networks trained on very few data points (probably so that they can memorize their training set). This thus corresponds to a rather unrealistic setting.\n\nAll in all, I am divided about this paper. On the one hand, it provides a unified analysis of both training and sampling errors, which opens up understanding how to adapt the discretization schedule to score errors. On the other hand, independently of the mathematical correctness of the results (which were not checked), they seem very artificial (due to the restricted setting of the analysis in section 3, and the very qualitative arguments in section 4), and not clearly presented at all. It is thus not clear to me how they can be useful to the theory and practice communities. I recommend rejection, but I am willing to change my decision if the authors convince me that my assessment is incorrect.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Figure 1 presents a clear qualitative picture of how to choose noise level schedules for training and sampling. In particular, this is the first theoretical work (to the best of my knowledge) to show how the sampling discretization schedule should be adapted depending on score errors arising because of imperfect training.\n- The sampling theoretical analysis is thorough and is very up-to-date with the sampling convergence literature. It extends previous results from the variance-preserving to the variance-exploding case, but the extension seems straightforward.\n- I am however not knowledgeable enough about the optimization literature to speak of the novelty of the training error results.", "weaknesses": "All the theoretical results are very hard to even parse, let alone extract intuition or guidance for practice. The authors argue that their results provide justification for the design choices of Karras et al. [31], but it seems to me that they are shoehorned to fit the chosen narrative. In other words, it is not clear to me that the exact same error bounds could not be used to justify any other schedule used in practice. I find Section 4.1 in particular to be very handwavy. I appreciate the attempt to connect theoretical results to practical choices, but I think the connection is too tenuous here.\n\nSome claims made in the text seem unsubstantiated to me:\n- The authors claim that the NTK can only be analyzed in very restricted settings (lines 103-104). I don't see why this is true, as the NTK applies to all architectures in the lazy regime.\n- I don't see how Theorem 1 implies that GD has an exponential convergence (lines 223-224) as product factors depend on $s$.\n- I don't see how Theorem 2 implies that the sampling error is almost linear in $d$ as $\\gamma_k$ and $\\sigma_t$ may depend on $d$.\n\nMinor:\n- line 140 and eq (4): $C$ should be $C_t$ as it depends on $t$\n- line 150: aspectS\n- line 154: by A deep ReLU\n- line 176: introduce $W$ before line 174\n- line 364: we maintain a fixed ~the~ total weighting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for all the helpful comments. Sorry for confusions created by our initial submission. We hope the itemized responses below, can do a better job in making things clearer, and will further accomplish this goal in a revision.\n\n> All the theoretical results are very hard to even parse, let alone extract intuition or guidance for practice. The authors argue that their results provide justification for the design choices of Karras et al. [31], but it seems to me that they are shoehorned to fit the chosen narrative. In other words, it is not clear to me that the exact same error bounds could not be used to justify any other schedule used in practice. I find Section 4.1 in particular to be very handwavy. I appreciate the attempt to connect theoretical results to practical choices, but I think the connection is too tenuous here.\n\n\nWe appreciate an opportunity of explanation. Our biggest contribution is the analysis of the score training process. This is in contrast to the sampling process, which, as pointed out, is by now already much better understood thanks to an entire community. Our 2nd biggest contribution, in our opinion, is the general error bound obtained by fusing both training and sampling, and it applies to all design choices. Due to the complexity of deep learning where data, architecture, nonconvex optimization, and schedules are all mingled together, the theoretical result is indeed complicated, and it might have to, if an informative rate is desired.\n\nBut like the reviewer pointed out, the theoretical result may be indeed difficult to parse due to being fully *quantitative* (it could also be due to our presentation, which we now improved significantly). Understanding that, we thus also tried to illustrate some of its *qualitative* implication by connecting to the practice, whose difficulty we appreciate that the reviewer acknowledged. To that end, we simply would like to understand one thing, namely whether/why the design choice of Karras et al. [31] is good, largely due to its impressive performance and hence popularity. And our theory indeed helped us obtain answers, which can be considered as our no.3 contribution. Regarding \"it is not clear to me that the exact same error bounds could not be used to justify any other schedule used in practice\", our bounds indeed can be used to justify other schedules as well, but we were not able to pack analyses of more schedules due to page limit.\n\n\n\n\n\n\n> The authors claim that the NTK can only be analyzed in very restricted settings (lines 103-104). I don't see why this is true, as the NTK applies to all architectures in the lazy regime.\n\nApology for the confusion. We feel the inconsistent understandings between us and the expert reviewer might be due to different definitions of NTK. The reviewer's interpretation of \"NTK\" seems to be in a broad sense: any models using infinitesimal learning rate in the lazy regime can be referred to as the NTK training regime, which is definitely true. In contrast, \"NTK\" considered in our original version is in a narrow sense following the initial NeurIPS'18 NTK paper, meaning that proof techniques mainly leverage the Gram matrix of the NTK kernel. Based on this definition, current literature only proved the convergence under restricted settings (scalar output, or vector output of a two-layer network with only one layer trained) due to  possibile difficulty lying in the structure of the Gram matrix. Methodologies other than the kernel technique, as discussed and used in this paper, can also reproduce many (narrow-sense) NTK results, but we would like to be precise in citing these literature and distinguishing lazy training from \"NTK\". More clarifications are already made in a revised version, and we appreciate the reviewer points out. \n\n> I don't see how Theorem 1 implies that GD has an exponential convergence (lines 223-224) as product factors depend on $s$.\n\nThanks for an expert question. This $s$ dependence indicates that at each step $s$, the decay ratio between the next step and the current step can be different. However, such ratio has a uniform bound when choosing $j^*(s)= j^*=\\min_j w(t_j)(t_j-t_{j-1})\\bar{\\sigma}\\_{t_j}$, which does not depend on $s$. Therefore, the product term $\\prod_{s=0}^k\\left(1-C_6hw(t_{j^*(s)})(t_{j^*(s)}-t_{j^*(s)-1})\\bar{\\sigma}\\_{t_{j^*(s)}}(\\frac{m\\sqrt{\\log d}d^{2a-1.5-4c}}{n^2N})\\right)$ is further upper bounded by $\\left(1-C_6hw(t_{j^*})(t_{j^*}-t_{j^*-1})\\bar{\\sigma}\\_{t_{j^*}}(\\frac{m\\sqrt{\\log d}d^{2a-1.5-4c}}{n^2N})\\right)^{k+1}$. Please kindly see lines 225-226 for similar illustration.\n\n> I don't see how Theorem 2 implies that the sampling error is almost linear in $d$ as $\\gamma_k$ and $\\sigma_t$ may depend on $d$.\n\nTheorem 2 presents a convergence result of $\\text{KL}(p_\\delta|q_{t_N})$ for general $\\sigma_t$ and $\\gamma_k$. In practice, $\\sigma_t$ is usually chosen to be a function of $t$, independent of $d$, see [31, 52]. Once we determine what $\\sigma_t$ to use, under the exponenital time schedule, the $d$-dependence in the sampling error is always of the form\n$$\nE_I+E_D \\lesssim d \\frac{\\log (T/\\delta)^{k+1} }{N^k},\\quad k=1,2.\n$$\nEven though $T$ is chosen to be of order poly($d$), the overall sampling error is almost linear in $d$. Please refer to Appendix F.1 in the paper for a detailed discussion.\n\n> typos\n\nSincerely appreciated. They will of course be corrected.\n\n--------\n**We would like to kindly direct the reviewer to \"Comment\" for the response to the rest of the review.** We sincerely apologize for exceeding the character limit but we eagerly hope to thoroughly address the reviewer's concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for the helpful comments. We have improved our convergence theory, and the new version now uses a much more relaxed assumption, where the original version $$d=\Theta(m)$$ is replaced by
$$d=\Omega({\text{poly}}(\log (nN))).$$

Note in the old version, the input dimension is the same order as the width of the network $m$, while in the new version, it only needs $d$ to be not too small, or equivalently, the number of data points $n$ and the number of time points $N$ should not be exponential in $d$.

All the other assumptions remain unchanged and we obtain the following new version of the theorem:
> **Theorem 1.** Define a set of indices to be $\mathcal{G}^{(s)}=\{(i,j)|f(\theta^{(s)};i,j)\ge f(\theta^{(s)};i',j')\text{ for all }i',j'\}$.
 Then given Assumption 1 and 2, for any $\epsilon_{\rm train}>0$, there exists some $M(\epsilon_{\rm train})=\Omega\left(\text{poly}\big(n,N,d,L,T/t_0,\log(\frac{1}{\epsilon_{\rm train}})\big)\right)$, s.t., when $m\ge M(\epsilon_{\rm train})$,  $h =\Theta(\frac{nN}{m\min_j w(t_j)(t_j-t_{j-1}) \bar\sigma_{t_j} })$, and $k=\mathcal{O}(d^{\frac{1-a_0}{2}}n^2N\log(\frac{d}{\epsilon_{\rm train}}))$, with probability at least $1-\mathcal{O}(nN)\exp(-\Omega(d^{2a_0-1}))$, we have
    $$
      \bar{\mathcal{L}}\_{\rm em}(\theta^{(k)})\le\prod_{s=0}^{k-1}\left(1-C_5 h \ w(t_{j^*(s)})(t_{j^*(s)}-t_{j^*(s)-1})\bar{\sigma}\_{t_{j^*(s)}} \left(\frac{md^{\frac{a_0-1}{2}}}{n^3N^2}\right)\right)\bar{\mathcal{L}}\_{\rm em}(\theta^{(0)})
       $$
    where the universal constant $C_5>0$, $a_0\in(\frac{1}{2},1)$,  and $(i^*(s),j^*(s))=\arg\max_{(i,j)\in\mathcal{G}^{(s)}}w(t_{j})(t_j-t_{j-1})\bar{\sigma}\_{t_{j}}$. Moreover, when $K=\Theta(d^{\frac{1-a_0}{2}}n^2N\log(\frac{d}{\epsilon_{\rm train}}))$, 
    $$
        \bar{\mathcal{L}}\_{\rm em}(\theta^{(K)})\le \epsilon_{\rm{train}}.
   $$

This bound also has improved dependency on $d$, namely $d^{\frac{a_0-1}{2}}\in(d^{-1/4},1)$ as opposed to the old version, which is $d^{2a-1.5-4c}\sqrt{\log d}\in (d^{-1}\sqrt{\log d}, d^{-9/10}\sqrt{\log d})$.

Meta-review: This paper presents an analysis of diffusion models, taking into account both the training and sampling processes. The authors derive non-asymptotic convergence bounds for denoising score matching under gradient descent and obtain a refined sampling error analysis for variance exploding models. This unified analysis provides insights into the design choices of diffusion models, including the selection of noise distribution, loss weighting functions, and time and variance schedules.

The paper has perhaps potential to guide practical applications. However, the results presented therein are complex and it will be difficult for readers to understand their practical implications. While the paper aims to connect theoretical results with practical design choices, it somewhat lacks in concrete examples and guidelines. This is reinforced by the lack of empirical results. Additionally the authors rely on strong assumptions which may not capture the practice of diffusion models. 

So while I recommend the paper be accepted, the authors are strongly encouraged to enhance the clarity and accessibility of the presentation. This could include providing more intuitive explanations of the theoretical results and their practical implications, offering informal versions of the theorems, summarizing key features etc. A more concrete discussion of practical implications should be included. and the authors should demonstrate how their theoretical findings can help guiding the selection of noise distributions, weighting functions, and time and variance schedules, providing specific examples and applications. Presenting detailed experiments to validate these guidelines would improve significantly the paper's impact.
Decision: Accept (poster)