Title: Evaluating the design space of diffusion-based generative models
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion-based models

denoising score matching

convergence of neural network training

generation error analysis

noising distribution

loss weighting

time schedule

variance schedule
Abstract: Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. In addition, a refined sampling error analysis for variance exploding models is also provided. The combination of these two results yields a full error analysis, which elucidates (again, but this time theoretically) how to design the training and sampling processes for effective generation. For instance, our theory implies a preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in [Karras et al., 2022]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design in [Song et al., 2021] is more preferable, but when it is less trained, the design in [Karras et al., 2022] becomes more preferable.

Evaluating the design space of diffusion-based generative models

Abstract

Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. In addition, a refined sampling error analysis for variance exploding models is also provided. The combination of these two results yields a full error analysis, which elucidates (again, but this time theoretically) how to design the training and sampling processes for effective generation. For instance, our theory implies a preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in Karras et al. [30]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design in Song et al. [46] is more preferable, but when it is less trained, the design in Karras et al. [30] becomes more preferable.

1 Introduction

Diffusion models became a very popular generative modeling approach in various domains, including computer vision [20, 7, 27, 28, 38, 51], natural language processing [6, 34, 37], various modeling tasks [15, 41, 55], and medical, biological, chemical and physical applications [3, 17, 43, 49, 23, 56] (see more surveys in [53, 11, 14]). Karras et al. [30] provided a unified empirical understanding of the derivations of model parameters, leading to new state-of-the-art performance. Karras et al. [31] further upgraded the model design by revamping the network architectures and replacing the weights of the network with an exponential moving average. As diffusion models gain wider usage, efforts to understand and enhance their generation capability become increasingly meaningful.

In fact, a rapidly increasing number of theoretical works already analyzed various aspects of diffusion models [32, 19, 52, 16, 12, 8, 18, 9, 13, 39, 44, 25]. Among them, a majority [32, 19, 52, 16, 12, 8, 18] focus on sampling/inference; more precisely, they assume the score error is within a certain accuracy threshold (i.e. the score function is well trained in some sense), and analyze the discrepancy between the distribution of the generated samples and the true one. Meanwhile, there are a handful of results [44, 25] that aim at understanding different facets of the training process. See more detailed discussions of existing theoretical works in Section 1.1.

Ye He

However, as indicated in Karras et al. [30], the performance of diffusion models also relies on the interaction between design components in both training and sampling, such as the noise distribution, weighting, time and variance schedules, etc. While focusing individually on either the training or generation process provides valuable insights, a holistic quantification of the actual generation capability can only be obtained when both processes are considered altogether. Therefore, motivated by obtaining deeper theoretical understanding of how to maximize the performance of diffusion models , this paper aims at establishing a full generation error analysis, combining both the optimization and sampling processes, to partially investigate the design space of diffusion models.

More precisely, we focus on the variance exploding setting [46], which is also the foundation of continuous forward dynamics in Karras et al. [30]. Our main contributions are summarized as follows:

For denoising score matching objective, we establish the exponential convergence of its gradient descent training dynamics (Theorem 1). We develop a new method for proving a key lower bound of gradient under the semi-smoothness framework [1, 35, 57, 58].

We extend the sampling error analysis in [8] to the variance exploding case (Theorem 2), under only the finite second moment assumption (Assumption 3) of the data distribution. Our result applies to various variance and time schedules, and implies a sharp almost linear complexity in terms of data dimension under optimal time schedule.

We conduct a full error analysis of diffusion models, combining training and sampling (Theorem 3).

We qualitatively derive the theory for choosing the noise distribution and weighting in the training objective, which coincides with Karras et al. [30] (Section 4.1). More precisely, our theory implies that the optimal rate is obtained when the total weighting exhibits a similar 'bell-shaped' pattern used in Karras et al. [30].

We develop a theory of choosing time and variance schedules based on both training and sampling (Section 4.2). Indeed, when the score error dominates, i.e., the neural network is less trained and not very close to the true score, polynomial schedule [30] ensures smaller error; when sampling error dominates, i.e., the score function is well approximated, exponential schedule [46] is preferred.

Conclusions and limitations are in Appendix A.

1.1 Related works

Sampling. There has been significant progress in quantifying the sampling error of the generation process of diffusion models, assuming the score function is already approximated within certain accuracy. Most existing works [e.g., 16, 12, 8] focused on the variance preserving

(VP) SDEs, whose discretizations correspond to DDPM. For example, Benton et al. [8] is one of the latest results for the VPSDE-based diffusion models, and it only needs a very mild assumption: the data distribution has finite second moment. The iteration complexity is shown to be almost linear in the data dimension and polynomial in the inverse accuracy, under exponential time schedule. However, a limited amount of works [32, 24, 54] analyzed the variance exploding (VE) SDEs, whose discretizations correspond to Score Matching with Langevin dynamics (SMLD) [45, 46]. To our best knowledge, Yang et al. [54] obtained the best result so far for VE assuming the data distribution has bounded support: the iteration complexity is polynomial in the data dimension and the inverse accuracy, under the uniform time schedule. In contrast, our work only assumed that the data distribution has finite second moment, and by extending the stochastic localization approach in [8] to VESDE, we obtain an iteration complexity that is polynomial in the data dimension and the inverse accuracy, under more general time schedules as well. Note the improved complexity in terms of the inverse accuracy and the data dimension dependencies; in fact, under the exponential time schedule, our complexity is almost linear in the data dimension, which recovers the state-of-the-art result for VPSDE-based diffusion models.

Training. To our best knowledge, the only works that quantify the training process of the diffusion models are Shah et al. [44] and Han et al. [25]. Shah et al. [44] employed the DDPM formulation and considered data distributions as mixtures of two spherical Gaussians with various scales of separation, together with K spherical Gaussians with a warm start. Then the score function can be analytically solved, and they modeled it in a teacher-student framework solved by gradient descent. They also provided the sample complexity bound under these specific settings. In contrast, our results work for general data distributions for which the true score is unknown, and training analysis is combined with

sampling analysis. Han et al. [25] considered the GD training of a two-layer ReLU neural network with the last layer fixed, and used the neural tangent kernel (NTK) approach to establish a first result on generalization error. They uniformly sampled the time points in the training objective, assumed that the Gram matrix of the kernel is away from 0 (implying a lower bound on the gradient), and lacked a detailed non-asymptotic characterization of the training process. In contrast, we use the deep ReLU network with L layers trained by GD and prove instead of assuming that the gradient is lower bounded by the objective function. Moreover, we obtain a non-asymptotic bound for the optimization error, and our bound is valid for general time and variance schedules, which allows us to obtain a full error analysis.

Convergence of neural networks training. The convergence analysis of neural networks under gradient descent has been a longstanding challenge and has been developed into an extensive field. Here we will only focus on results mostly related to the techniques used in this paper. One line of them is approaches directly based on neural tangent kernel (NTK) [22, 21, 5, 47, 36]. However, existing works in this direction focus more on either scalar output, or vector output but with only one layer trained under two-layer networks, which is insufficient for diffusion models. Another line of research also considers overparameterized models in a regime analogous to NTK, though not necessarily explicitly resorting to kernels. Instead, it directly quantifies the lower bound of the gradient [1, 35, 2, 57, 58] and uses a semi-smoothness property to prove exponential convergence. Our results align with the latter line, but we develop a new method for proving the lower bound of the gradient and adopt assumptions that are closer to the setting of diffusion models. See more discussions in Section 3.1.

1.2 Notations

We denote ∥ ⋅ ∥ to be the ℓ 2 norm for both vectors and matrices, and ∥ ⋅ ∥ F to be the Frobenius norm. For the discrete time points, we use t i to denote the time point for forward dynamics and t ← i for backward dynamics. For the order of terms, we follow the theoretical computer science convention to use O(⋅) , Θ (⋅) , Ω (⋅) . We also denote f ≲ g if f ≤ Cg for some universal constant C .

2 Basics of diffusion-based generative models

In this section, we will introduce the basic forward and backward dynamics of diffusion models, as well as the denoising score matching setting under which a model is trained.

2.1 Forward and backward processes

Consider a forward diffusion process that pushes an initial distribution P 0 to Gaussian √

where dW t is the Brownian motion, X t is a d -dim. random variable, and X t ∼ P t . Under mild assumptions, the process can be reversed and the backward process is defined as follows

where Y 0 ∼ P T , and p t is the density of P t . Then Y T -t and X t have the same distribution with density p t [4], which means the dynamics (2) will push (near) Gaussian distribution back to (nearly) the initial distribution P 0 . To apply the backward dynamics for generative modeling, the main challenge lies in approximating the term ∇ log p T -t ( Y t ) which is called score function . It is common to use a neural network to approximate this score function and learn it via the forward dynamics (1); then, samples can be generated by simulating the backward dynamics (2).

2.2 The training of score function via denoising score matching

In order to learn the score function, a natural starting point is to consider the following score matching objective [e.g., 29]

where S ( θ ; t, X t ) is a θ -parametrized neural network, w ( t ) is some weighting function , and the subscript means this is the continuous setup. Ideally one would like to minimize this objective function to obtain θ ; however, p t in general is unknown, and so is the true score function ∇ x log p t . One of the solutions is denoising score matching proposed by Vincent [48], where one, instead of directly matching the true score, leverages conditional score for which initial condition is fixed so that p t ∣ 0 is analytically known.

More precisely, given the linearity of forward dynamics (1), its exact solution is explicitly known: Let µ t = ∫ t 0 f s ds , and ¯ σ 2 t = 2 ∫ t 0 e 2 µ s -2 µ t σ 2 s ds . Then the solution is X t = e -µ t X 0 + ¯ σ t ξ, where ξ ∼ N ( 0 , I ) . We also have X t ∣ X 0 ∼ N ( e -µ t X 0 , ¯ σ 2 t I ) and g t ( x ∣ y ) = ( 2 π ¯ σ 2 t ) -d / 2 exp (-∥ x -e -µ t y ∥ 2 /( 2¯ σ 2 t )) , which is the density of X t ∣ X 0 . Then the objective can be rewritten as

where C t = E X t ∥∇ log p t ∥ 2 -E X 0 E X t ∣ X 0 ∥∇ log g t ( X t ∣ X 0 )∥ 2 . For completeness, we will provide a detailed derivation of these results in Appendix C and emphasize that it is just a review of existing results in our notation.

Throughout this paper, we adopt the variance exploding (VESDE) setting [46], where f t = 0 and hence µ t = 0 , which also aligns with the setup in the classic of EDM [30].

3 Error analysis for diffusion-based generative models

In this section, we will quantify both the training and sampling processes, and then integrate them into a more comprehensive generation error analysis.

3.1 Training

In this section, we consider a practical implementation of denoising score matching objective, represent the score by a deep ReLU network, and establish the exponential convergence of GD training dynamics.

Training objective function. Consider a quadrature discretization of the time integral in (4) based on deterministic 1 collocation points 0 < t 0 < t 1 < t 2 < /uni22EF < t N = T . Then

Define β j = w ( t j )( t j -t j -1 ) 1 ¯ σ t j to be the total weighting . Consider the empirical version of ¯ L (6) .

Denote the initial data to be { x i } n i = 1 with x i ∼ P 0 , and the noise to be { ξ ij } N j = 1 with ξ ij ∼ N( 0 , I d ) . Then the input data of the neural network is { t j , X ij } n,N i = 1 ,j = 1 = { t j , x i + ¯ σ t j ξ ij } n,N i = 1 ,j = 1 and the output data is { ξ ij / ¯ σ t j } n,N i = 1 ,j = 1 if ¯ σ t j ≠ 0 . Consequently, ¯ L( θ ) (6) can be approximated by the following

We will use (7) as the training objective function in our analysis. For simplicity, we also denote f ( θ ; i, j ) = β j ∥ ¯ σ t j S ( θ ; t j , x i + ¯ σ t j ξ ij ) + ξ ij ∥ 2 and then ¯ L em ( θ ) = 1 2 n ∑ n i = 1 ∑ N j = 1 f ( θ ; i, j ) . Note the time dependence can be absorbed into the X dependence. More precisely, because ¯ σ t is a monotonically increasing function of t , we can replace t j in the inputs by ¯ σ t j to indicate the time dependence. This is then equivalent to augmenting X ij to be d + 1 dimensional with ( x i ) d + 1 ∶= 0 and ( ξ ij ) d + 1 ∶= 1 . For simplified presentation, we will slightly abuse notation and still use d as the input dimension rather than d + 1 .

Architecture. The analysis of diffusion model training is in general very challenging. One obvious factor is the complex score parameterizations used in practice such as U-Net [42] and transformers [40, 34]. In this paper, we simplify the architecture and consider deep feedforward networks. Although it is still far from practical usage, note this simple structure can already provide insights about the design space, as shown in later sections, and is more complicated than existing works [25, 44] related to the training of diffusion models (see Section 1.1). More precisely, we consider the standard deep ReLU network with bias absorbed:

where θ = ( W 0 , /uni22EF , W L + 1 ) , W 0 ∈ R m × d , W L + 1 ∈ R d × m , W ℓ ∈ R m × m for ℓ = 1 , /uni22EF , L , and σ (⋅) is the ReLU activation.

Algorithm. Let θ ( k ) = ( W 0 , W ( k ) 1 , /uni22EF , W ( k ) L , W L + 1 ) . We consider the gradient descent (GD) algorithm as follows

where h > 0 is the learning rate. We fix W 0 and W L + 1 throughout the training process and only update W 1 , /uni22EF , W L , which is a commonly used setting in the convergence analysis of neural networks [1, 10, 25].

Initialization. We employ the same initialization as in Allen-Zhu et al. [1], which is to set ( W ( 0 ) ℓ ) ij ∼ N( 0 , 2 m ) for ℓ = 0 , /uni22EF , L , i, j = 1 , /uni22EF , m , and ( W ( 0 ) L + 1 ) ij ∼ N( 0 , 1 d ) for i = 1 /uni22EF , d , j = 1 /uni22EF , m .

For this setup, the main challenge in our convergence analysis for denoising score matching lies in the nature of the data. 1) The output data that neural network tries to match is an unbounded Gaussian random vector, and cannot be rescaled as assumed in many theoretical works (for example, Allen-Zhu et al. [1] assumed the output data to be of order o ( 1 ) ). 2) The input data X ij is the sum of two parts: x i which follows the initial distribution P 0 , and a Gaussian noise ¯ σ t j ξ ij . Therefore, any assumption on the input data needs to agree with this noisy and unbounded nature, and commonly used assumptions like data separability [1, 35] can no longer be used.

To deal with the above issues, we instead make the following assumptions.

Assumption 1 (On network hyperparameters and initial data of the forward dynamics) . We assume the following holds:

Data scaling: ∥ x i ∥ = Θ ( d 1 / 2 ) for all i .

Input dimension: d = Ω ( poly ( log ( nN ))) .

We remark that the first assumption focuses only on the initial data x i instead of the whole solution of the forward dynamics X ij which incorporates the Gaussian noise. Also, this assumption is indeed not far away from reality; for example, it holds with high (at least 1 -O( exp (-Ω ( d )) ) probability for standard Gaussian random vectors. The requirement for input dimension d is to ensure that d is not too small, or equivalently the number of data points is not exponential in d .

We also make the following assumptions on the hyperparameters of the denoising score matching. Assumption 2 (On the design of diffusion models) . We assume the following holds:

Weighting: ∑ N j = 1 w ( t j )( t j -t j -1 ) ¯ σ t j = O( N ) .

Variance: ¯ σ t 0 > 0 and ¯ σ t N = Θ ( 1 ) .

The first assumption is to guarantee that the weighting function w ( t ) is properly scaled. This expression w ( t j )( t j -t j -1 ) ¯ σ t j is obtained from proving the upper and lower bounds of the gradient of (7), and is different from the total weighting β i defined above. In the second assumption, ¯ σ t 0 > 0 ensures the output ξ ij / ¯ σ t j is well-defined. The ¯ σ t N = Θ ( 1 ) guarantees that the scales of the noise ¯ σ t j ξ ij and the initial data x i are of the same order at the end of the forward process, namely, the initial data x i is eventually push-forwarded to near Gaussian with the proper variance. Therefore, Assumption 2 aligns with what has been used in practice (see Section 4 and Karras et al. [30], Song et al. [46] for examples).

The following theorem summarizes our convergence result for the training of the score function.

Theorem 1 (Convergence of GD) . Define a set of indices to be G ( s ) = {( i, j )∣ f ( θ ( s ) ; i, j ) ≥ f ( θ ( s ) ; i ′ , j ′ ) for all i ′ , j ′ } . Then given Assumption 1 and 2, for any ϵ train > 0 , there exists some M ( ϵ train ) = Ω ( poly ( n, N, d, L, T / t 0 , log ( 1 ϵ train ))) , s.t., when m ≥ M ( ϵ train ) , h = Θ ( nN m min j w ( t j )( t j -t j -1 ) ¯ σ t j ) , and k = O( d 1 -a 0 2 n 2 N log ( d ϵ train )) , with probability at least 1 -O( nN ) exp (-Ω ( d 2 a 0 -1 )) , we have

where the universal constant C 5 > 0 , a 0 ∈ ( 1 2 , 1 ) , and ( i ∗ ( s ) , j ∗ ( s )) = arg max ( i,j )∈G ( s ) w ( t j )( t j -t j -1 ) ¯ σ t j . Moreover, when K = Θ ( d 1 -a 0 2 n 2 N log ( d ϵ train )) ,

The above theorem implies that for denoising score matching objective ¯ L em ( θ ) , GD has exponential convergence. For example, if we simply take j ∗ = min j w ( t j )( t j -t j -1 ) ¯ σ t j , then ¯ L em ( θ ( k + 1 ) ) is further upper bounded by ( 1 -C 6 h w ( t j ∗ )( t j ∗ -t j ∗ -1 ) ¯ σ t j ∗ ( md a 0 -1 2 n 3 N 2 )) k + 1 ¯ L em ( θ ( 0 ) ) . The rate of convergence can be interpreted in the following way: 1) at the k th iteration, we collect all the indices of the time points into G ( k ) where f ( θ ; i, j ) has the maximum value; 2) we then choose the maximum of w ( t j )( t j -t j -1 ) ¯ σ t j among all such indices and denote the index to be j ∗ ( k ) , and obtain the decay

Remark 1 (Can ϵ train be arbitrarily small? Some ramifications of the denoising setting) . Let us first see some facts about ¯ L em and ¯ L . Under minimal assumption of the existence of score function and in the zero-time-discretization-error limit, the score matching objective can be made zero and therefore the denoising score matching objective is bounded below by -¯ C , which is nonnegative and zero only when the data distribution is extremely special (we thus write -¯ C > 0 from hereon unless confusion arises). That is, min θ ¯ L( θ ) ≥ min any function S ¯ L = -¯ C > 0 according to (4) . Since ¯ L em → ¯ L as the sample size of the training data set n →∞ , we have ¯ L em ≥ -¯ C -c n > 0 for some constant c n > 0 and c n → 0 as n →∞ .

However, Theorem 1 seems to imply ¯ L em ( θ ( k ) ) → 0 as k →∞ since ¯ L em ( θ ( K ) ) ≤ ϵ train and ϵ train is arbitrary, and it seems to contradict the -¯ C > 0 lower bound. However, there is no contradiction due to the combination of two facts. First, the theorem states that for arbitrary ϵ train > 0 , there exists a critical size, such that for overparameterized network beyond this size, GD can render the loss ¯ L em ( θ ) eventually no greater than ϵ train . If we fix the network size, i.e., with m,L,d given, then K is given, and Theorem 1 says nothing about GD's behavior after K iterations. That is, we do not know whether limsup k →∞ ¯ L em ( θ ( k ) ) = 0 . Second, our optimization setting requires the sample size n to be smaller than the network width m (Assumption 1). Thus, when m is fixed, the sample size n is upper bounded.

The above discussion implies, within the validity of our theory, for any fixed network width m , if ϵ train is small, the sample size n cannot be too large, meaning ¯ L em ( θ ) -¯ L( θ ) may not be small. Therefore, we can simultaneously have ¯ L em ( θ ) close to 0 and ¯ L( θ ) close to -¯ C > 0 .

Main technical steps for proving Theorem 1. The proof of Theorem 1 is in Appendix D, where the analysis framework is adapted from Allen-Zhu et al. [1]. Roughly speaking, the key proof in this framework is to establish the lower bound of the gradient. Then by integrating it into the semi-smoothness property of the neural network, we can obtain the exponential rate of convergence of gradient descent. For the lower bound of gradient, we develop a new method to deal with the difficulties in the denoising score matching setting (see the discussions earlier in this section).

Our new proof technique adopts a different decoupling of the gradient and leverages a high probability bound based on a high-dimensional geometric idea. See Appendix D.1 for a proof sketch and more details.

3.2 Sampling

In this section, we establish a nonasymptotic error bound of the backward process in the variance exploding setting, which is an extension to Benton et al. [8]. For simplified notations, denote the backward time schedule as { t ← j } 0 ≤ j ≤ N such that 0 = t ← 0 < t ← 1 < /uni22EF < t ← N = T -δ .

Generation algorithm. We consider the exponential integrator scheme for simulating the backward SDE (2) with f t ≡ 0 2 . The generation algorithm can be piecewisely expressed as a continuous-time SDE: for any t ∈ [ t ← j , t ← j + 1 ) , √

Initialization. Denote q t ∶= Law ( ¯ Y t ) for all t ∈ [ 0 , T -δ ] . We choose the Gaussian initialization, q 0 = N( 0 , ¯ σ 2 T ) .

Our convergence result relies on the following assumption.

Assumption 3. The distribution P 0 has a finite second moment: E x ∼ P 0 [∥ x ∥ 2 ] = m 2 2 < ∞ .

Next we state the main convergence result, whose proof is provided in Appendix E.

j + 1 j Theorem 2 is a VESDE-based diffusion model's analogy of what's proved in Benton et al. [8] for VPSDE-based diffusion model, only requiring the data distributions to have finite second moments, and it achieves the sharp almost linear data dimension dependence under the exponential time schedule. The major differences from [8] are (1) the initialization error in the VESDE case is handled differently (see Lemma 10); (2) Theorem 2 applies to varies choices of time schedules, which enables to investigate the design space of the diffusion model, as we will discuss in Section 4. Worth mentioning is, Yang et al. [54] also obtained polynomial complexity results for VESDE-based diffusion models with uniform stepsize, but under stronger data assumption (assuming compact support). Compared to their result, complexity implied by Theorem 2 has better accuracy and data dimension dependencies. A detailed discussion on complexities is given in Appendix I.1.

where γ j ∶= t ← -t ← for all j = 0 , 1 , /uni22EF , N -1 is the stepsize of the generation algorithm in (10) .

Terms E I , E D , E S in (11) represent the three types of errors: initialization error, discretization error, and score estimation error, respectively. Term E I quantifies the error between the initial density of the sampling algorithm q 0 and the ideal initialization p T , which is the density when the forward process stops at time T . Term E D is the error stemming from the discretization of the backward dynamics. Term E S characterizes the error of the estimated score function and the true score, and is related to the optimization error of ¯ L em . Important to note is, in Theorem 2, population loss is needed instead of the empirical version ¯ L em (7). Besides this, the weighting γ j σ 2 T -t ← j is not necessarily the same as the total weighting in ¯ L em (7) β j , depending on choices of w ( t j ) and time and variance schedules (see more discussion in Section 4). We will later on integrate the optimization error (Theorem 1) into this score error E S to obtain a full error analysis in Section 3.3.

Remark 2 (sharpness of dependence in d and m 2 2 ) . In one of the simplest cases, when the data distribution is Gaussian, the score function is explicitly known. Hence KL ( p δ ∣ q T -δ ) can be explicitly computed as well, which verifies that the dependence of parameters d and m 2 2 is sharp in E I and E D .

3.3 Full error analysis

In this section, we combine the analyses from the previous two sections to obtain an end-to-end generation error bound.

Before providing the main result of this section, let us first clarify some terminologies.

Time schedule, variance schedule, and total weighting. The terms time schedule and variance schedule respectively refer to the choice of t ← j and ¯ σ t j in sampling. Meanwhile, note both the training and sampling processes require the proper choices of time and variance, and these choices are not necessarily the same for both processes. For training, the effect of these two is integrated into the total weighting β j , which is also influenced by an additional weighting parameter w ( t j ) . In this theoretical paper, when studying the generation error, we aim to apply the optimization result to better understand the effect of optimization on sampling. Therefore, to simplify the analysis and discussions in Section 4, we choose the same time and variance schedules for both training and sampling.

The main result is stated in the following.

Theorem 3. Under the same conditions as Theorem 1,2, and that K is such that GD reaches ϵ train in at most K th iterations, we have

where E I , E D are defined in Theorem 2, ϵ train is defined in Theorem 1, ϵ n = ∣ ¯ L( θ ( K ) )-¯ L em ( θ ( K ) )+ ¯ L em ( θ ∗ )-¯ L( θ ∗ )∣ , ϵ est = ∣ ¯ L( θ ∗ )-¯ L( θ F )∣ , ϵ approx = ∣ ¯ L( θ F )+ ¯ C ∣ . In these terms, ¯ C is defined in (5) ,

θ ∗ = arg min θ,s.t., ¯ L em ( θ )= 0 ¯ L( θ ) and θ F = arg inf { θ ∶ S ( θ )∈F} ∣ ¯ L( θ ) + ¯ C ∣ with F = { ReLU network function defined in (8) , with d = Ω ( poly ( log ( nN ))) , m = Ω ( poly ( n, N, d, L, T / t 0 ))} .

In this theorem, the discretization error E D and initialization error E I are the same as Theorem 2. For the score error E S , our optimization result is valid for general time schedules and therefore can directly fit into the sampling error analysis, which is in contrast to existing works [25, 44] (see more discussions in Section 1.1). The coefficient max j σ 2 t N -j / w ( t N -j ) results from different weightings in E S and ¯ L em , i.e., γ j σ 2 T -t ← j and β j . We will discuss the effect of max j σ 2 t N -j / w ( t N -j ) under different time and variance schedules in Section 4.

The way we bound E Y t ← j ∼ p T -t ← j [∥ S ( θ ; T -t ← j , Y t ← j ) - ∇ log p T -t ← j ( Y t ← j )∥ 2 ] in E S (see Theorem 2) is to decompose it into the optimization error ϵ train , statistical error ϵ n , estimation error ϵ est , and approximation error ϵ approx . This gives clear intuition to results, but we also note it may not give a tight bound. In fact, we have

ϵ n can still be small if we take n →∞ , but that means ϵ train has to be large, and our generation error bound cannot be made 0. It is unclear yet whether this is due to limitation of our analysis or intrinsic, and will be left for future investigation.

Another related note is, in this paper, we focus on ϵ train and the effect of optimization, but the analyses of ϵ n , ϵ est , and ϵ approx are also important and possible [13, 39, 25, 50]. On the other hand, again, whether it is optimal to decompose the full error into these four is unclear.

To better see the parameter dependence of the error bound in Theorem 3, the following is an example with simplified results, where we employ the schedules in EDM [30].

Corollary 1 (Full error analysis under EDM [30] designs) . Under the same conditions as Theorem 3, we have

where C 8 , C 9 > 0 and a = 7 in [30].

4 Theory-based understanding of the design space and its relation to existing empirical counterparts

This section theoretically explores preferable choices of parameters in both training and sampling, and shows that they agree with the ones used in EDM [30] and Song et al. [46] in different circumstances.

4.1 Choice of total weighting for training

This section develops the optimal total weighting β j for training objective (7). We qualitatively show in two steps that 'bell-shaped' weighting, which is the one used in EDM [30], will lead to the optimal rate of convergence: Step 1) ∥ ¯ σ t j S ( θ ; t j , X ij ) + ξ ij ∥ as a function of j is inversely 'bell-shaped'; Step 2) f ( θ ; i, j ) = β j ∥ ¯ σ t j S ( θ ; t j , X ij ) + ξ ij ∥ should be close to each other for any i, j .

4.1.1 Inversely 'bell-shaped' loss ∥ ¯ σ t j S ( θ ; t j , X ij ) + ξ ij ∥ as a function of time index j Proposition 1. Under the same assumptions as Theorem 1, for any θ and i = 1 , /uni22EF , n , we have

∀ ϵ 1 > 0 , ∃ δ 1 > 0 , s.t., when 0 ≤ ¯ σ t < δ 1 , ∥ ¯ σ t S ( θ ; t j , x i + ¯ σ t ξ ij ) + ξ ij ∥ > ∥ ξ ij ∥ -ϵ 1 .

The above proposition can be interpreted in the following way. Given any network S , when ¯ σ t j is very small, 1 implies that ∥ ¯ σ t j S ( θ ; t j , x i + ¯ σ t j ξ ) + ξ ij ∥ is away from 0 by approximately ∥ ξ ij ∥ which is of order √ d with high probability, i.e., it cannot be small. When ¯ σ t j is large, 2 shows that as it becomes larger and larger, i.e., as M increases, ∥ ¯ σ t j S ( θ ; t j , x i + ¯ σ t j ξ ) + ξ ij ∥ will also increase. Therefore, the function ∥ ¯ σ t j S ( θ ; t j , X ij ) + ξ ij ∥ has most likely an inversely 'bell-shaped' curve in terms of j dependence.

4.1.2 Ensuring comparable values of f ( θ ; i, j ) for optimal rate of convergence

Corollary 2. Under the same conditions as Theorem 1, for some large K ′ > 0 , if ∣ f ( θ ( k + K ′ ) ; i, j ) -f ( θ ( k + K ′ ) ; l, s )∣ ≤ ϵ holds for all k > 0 and all ( i, j ) , ( l, s ) , with some small universal constant ϵ > 0 , then we have, for some constant C 7 > 0 ,

The above corollary shows that if f ( θ ( k ) ; i, j ) 's are almost the same for any i, j , then the decay ratio of the next iteration is minimized. More precisely, the index set G ( k ) defined in Theorem 1 is roughly the whole set { 1 , /uni22EF , N } , and therefore w ( t j ∗ )( t j ∗ -t j ∗ -1 ) ¯ σ t j ∗ can be taken as the maximum value over all j , which consequently leads to the optimal rate.

4.1.3 'Bell-shaped' weighting: our theory and EDM

Combining the above two aspects, the optimal rate of convergence leads to the choice of total weighting β j such that f ( θ ; i, j ) = β j ∥ ¯ σ t j S ( θ ; t j , X t j ) + ξ ij ∥ is close to each other; as a result, the total weighting should be chosen as a 'bell-shaped' curve as a function of j according to the shape of the curve for ∥ ¯ σ t j S ( θ ; t j , X t j ) + ξ ij ∥ .

10

!

2

10 0

10 2

0

1

2

3

4

5

6

7

-

EDM

7

<

Before comparing the preferable weighting predicted by our theory and the intuition-and-empiricsbased one in EDM [30], let us first recall that the EDM training objective 3 can be written as E ¯ σ ∼ p train E y,n λ ( ¯ σ )∥ D θ ( y + n ; ¯ σ ) -y ∥ 2

where Z 1 is a normalization constant, and we denote β EDM ( ¯ σ ) = e -( log ¯ σ -P mean ) 2 2 P 2 std ¯ σ 2 + σ 2 data ¯ σσ 2 data to be the total weighting of EDM. Note the dependence on ¯ σ and time j can be freely switched due to their 1-to-1 correspondence.

Figure 2 plots the total weighting of EDM β EDM as a function of ¯ σ . As is shown in the picture, this is a 'bell-shaped' curve 4 , which coincides with our choice of total weighting in the above theory. When ¯ σ is very small or very large, according to Proposition 1, the lower bound of ∥ ¯ σ t j S ( θ ; t j , X t j ) + ξ ij ∥ cannot vanish and therefore needs the smallest weighting over all ¯ σ . When ¯ σ takes the middle value, the scale of the output data ξ ij / ¯ σ j is roughly the same as the input data X ij and therefore makes it easier for the neural network to fit the data, which admits larger weighting.

4.2 Choice of time and variance schedules

This section will discuss the choice of time and variance schedules based on the three errors E S , E D , E I in the error analysis of Section 3.3. Two situations will be considered based on how well the score function is approximated in training: when the network is less trained, E S dominates and polynomial schedule [30] is preferable; when the score function is well approximated, E D + E I dominates and exponential schedule [46] is better.

4.2.1 When score error E S dominates

As is shown in Theorem 3, the main impact of different time and variance schedules on score error E S appears in the term max j σ 2 t N -j / w ( t N -j ) , when the score function is approximated to a certain accuracy. It remains to compute w ( t ) under various choices of schedules.

General rule of constructing w ( t ) . To ensure fair comparisons between different time and variance schedules, we maintain a fixed total weighting in the training objective. Additionally, to facilitate comparisons with practical usage, we adopt the total weighting in EDM, i.e., β j = C 3 β EDM ( ¯ σ t j ) , for some universal constant C 3 > 0 . The reason for using the EDM total weighting is that according to Section 4.1, our total weighting β j should be 'bell-shaped' as a function of j , which agrees qualitatively with the one used in EDM.

Polynomial schedule [30] vs exponential schedule [46]. We fix ϵ n , ϵ train and apply the two schedules (Table 1) separately to the above total weighting β (hence w ). Then, compute max j σ 2 t N -j / w ( t N -j ) which is a factor in score error E S (Thm.3) in Table 2. The Exp.'s result 1 2 ( ¯ σ max -¯ σ max ( ¯ σ 2 min ¯ σ 2 max ) 1 / N ) is larger 5 than the Poly.'s result ( ¯ σ max -( ¯ σ 1 / ρ max -¯ σ 1 / ρ max -¯ σ 1 / ρ min N ) ρ ) for large N , meaning the poly. time schedule in EDM is better than the exp. schedule in [46]. Note these two terms are both of order 1 / N as N →∞ and therefore the difference lies in their prefactors.

4.2.2 When discretization error E D and initialization error E I dominate

In this section, we compare the two different schedules in Table 1 by studying the iteration complexity of the sampling algorithm, i.e., number of time points N , when E D + E I dominates.

General rules of comparison. We consider the case when the discretization and initialization errors are bounded by the same quantity ϵ , i.e., E I + E D ≲ ε. Then according to Theorem 2 and Theorem 3, we compute the iteration complexity for achieving this error using the two schedules in Table 1. To make the comparison more straightforward, we adopt T = t N = Θ ( poly ( ε -1 )) and therefore ¯ σ max = Θ ( ε -1 / 2 ) . More details are provided in Appendix I.1.

Polynomial schedule [30] vs exponential schedule [46]. As is shown in the last column of Table 2, the iteration complexity under exponential schedule [46] has the poly-logarithmic dependence on the ratio between maximal and minimal variance ( ¯ σ max / ¯ σ min ) 6 , which is better than the complexity under polynomial schedule [30], which is polynomially dependent on ¯ σ max / ¯ σ min . Both complexities are derived from Theorem 2 by choosing different parameters.

Remark 3 (The existence of optimal ρ in the polynomial schedule [30]) . For fixed ¯ σ max and ¯ σ min , the optimal ρ that minimizes the iteration complexity is ρ = 1 2 ln ( ¯ σ max ¯ σ min ) . In [30], it was empirically observed that with fixed iteration complexity, there is an optimal value of ρ that minimizes the FID. Our result indicates that, for fixed ¯ σ max and ¯ σ min , hence the desired accuracy in KL divergence being fixed, there is an optimal value of ρ that minimizes the iteration complexity to reach the fixed accuracy. Even though we consider a different metric/divergence instead of FID, our result still provides a quantitative support to the existence of optimal ρ observed in [30].

Acknowledgments and Disclosure of Funding

The authors are grateful for the partially support by NSF DMS-1847802, Cullen-Peck Scholarship, and GT-Emory Humanity.AI Award. We thank the anonymous reviewers for their helpful comments.

References

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International conference on machine learning , pages 242-252. PMLR, 2019.

Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. On the convergence rate of training recurrent neural networks. Advances in neural information processing systems , 32, 2019.

Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019 , 2022.

Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=SlxSY2UZQT .

Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations , 2024.

Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv preprint arXiv:2002.00107 , 2020.

Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems , 32, 2019.

Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering , 2024.

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning , pages 4735-4763. PMLR, 2023.

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning , pages 4672-4712. PMLR, 2023.

Minshuo Chen, Song Mei, Jianqing Fan, and Mengdi Wang. An overview of diffusion models: Applications, guided generation, statistical rates and optimization. arXiv preprint arXiv:2404.07771 , 2024.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=NsMLjcFaO8O .

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. ICLR , 2023.

Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis , 80:102479, 2022.

Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffusion models without early stopping: finite fisher information is all you need. arXiv preprint arXiv:2308.12240 , 2023.

Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. TMLR , 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning , pages 1675-1685. PMLR, 2019.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Chenru Duan, Yuanqi Du, Haojun Jia, and Heather J Kulik. Accurate transition state generation with an object-aware equivariant elementary reaction diffusion model. Nature Computational Science , 3(12):1045-1055, 2023.

Xuefeng Gao and Lingjiong Zhu. Convergence analysis for general probability flow odes of diffusion models in wasserstein distances. arXiv preprint arXiv:2401.17958 , 2024.

Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=h8GeqOxtd4 .

Ye He, Kevin Rojas, and Molei Tao. Zeroth-order sampling methods for non-log-concave distributions: Alleviating metastability by denoising diffusion. arXiv preprint arXiv:2402.17886 , 2024.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research , 23(47):1-33, 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems , 35:8633-8646, 2022.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35: 26565-26577, 2022.

Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. arXiv preprint arXiv:2312.02696 , 2023.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems , 35:22870-22882, 2022.

Yongjae Lee and Woo Chang Kim. Concise formulas for the surface area of the intersection of two hyperspherical caps. KAIST Technical Report , 2014.

Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems , 35:4328-4343, 2022.

Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems , 31, 2018.

Xin Liu, Zhisong Pan, and Wei Tao. Provable convergence of nesterov's accelerated gradient method for over-parameterized neural networks. Knowledge-Based Systems , 251:109277, 2022.

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=aBsCjcPu_tE .

Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning , pages 26517-26582. PMLR, 2023.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Arne Schneuing, Yuanqi Du, Charles Harris, Arian Jamasb, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, et al. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.13695 , 2022.

Kulin Shah, Sitan Chen, and Adam Klivans. Learning mixtures of gaussians using the ddpm objective. Advances in Neural Information Processing Systems , 36:19636-19649, 2023.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations , 2021.

Zhao Song and Xin Yang. Quadratic suffices for over-parametrization via matrix chernoff bound. arXiv preprint arXiv:1906.03593 , 2019.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature , 620(7976):1089-1100, 2023.

Andre Wibisono, Yihong Wu, and Kaylee Yingxi Yang. Optimal score estimation via empirical bayes smoothing. arXiv preprint arXiv:2402.07747 , 2024.

Junde Wu, RAO FU, Huihui Fang, Yu Zhang, Yehui Yang, Haoyi Xiong, Huiying Liu, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. In Medical Imaging with Deep Learning , 2023. URL https://openreview.net/forum?id= Jdw-cm2jG9 .

Kaylee Yingxi Yang and Andre Wibisono. Convergence in KL and Rényi divergence of the unadjusted langevin algorithm using estimated score. NeurIPS Workshop on Score-Based Methods , 2022.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys , 56(4):1-39, 2023.

Ruofeng Yang, Zhijie Wang, Bo Jiang, and Shuai Li. The convergence of variance exploding diffusion models under the manifold hypothesis. OpenReview , 2024.

Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning , pages 12062-12072. PMLR, 2021.

Yuchen Zhu, Tianrong Chen, Evangelos A Theodorou, Xie Chen, and Molei Tao. Quantum state generation with structure-preserving diffusion model. arXiv preprint arXiv:2404.06336 , 2024.

Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks. Advances in neural information processing systems , 32, 2019.

Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes overparameterized deep relu networks. Machine learning , 109:467-492, 2020.

Appendix

A Conclusions and limitations

Conclusions. In this paper, we provide a first full error analysis incorporating both optimization and sampling processes. For the training process, we provide a first result under a deep neural network and prove the exponential convergence into a neighborhood of minima. At the same time, we extend the current analysis to the variance exploding case for sampling. Moreover, based on the full error analysis, we establish a quantitative understanding of the error bound under the two schedules. Consequently, we conclude with a qualitative illustration of the "bell-shaped" weighting and the choices of schedules under well-trained and less-trained cases.

Limitations. The network architecture we used in the model is a deep ReLU network. Although being so far the most complicated architecture for theoretical results, it is still far from what is used in practice like U-Nets and transformers. Moreover, regarding the full error analysis, we only focus on the optimization and sampling error and do not dissect the generalization error. When bridging the theoretical results with practical designs of diffusion models, our results are mostly qualitative and we only compare two existing schedules under two extreme cases, when the network is well-trained and less-trained. Thus, theoretical implications on practical designs remain to be explored. We will leave these perspectives for future exploration.

B Notations

C Derivation of denoising score matching objective

In this section, we will derive the denoising score matching objective, i.e. show the equivalence of (3) and (4). For simplicity, we denote S θ to be the neural network we use S ( θ ; t, X t ) . Consider

where p t is the density of X t

Since p t ( x ) = ∫ p 0 ( y ) q t ( x ∣ y ) dy , where q t (⋅) is the density of X t ∣ X 0 , then we have

Then

where C = E X t ∥∇ log p t ∥ 2 -E X 0 E X t ∣ X 0 ∥∇ log q t ( x t ∣ x 0 )∥ 2 .

Moreover, X t ∣ X 0 ∼ N( e -µ t X 0 , ¯ σ 2 t I ) , and its density function is

Then

Let ξ = ϵ t ¯ σ t ∼ N( 0 , I ) . Then

D Proofs for training

In this section, we will prove Theorem 1.

Before introducing the concrete proof, we first redefine the deep fully connected feedforward network

where W 0 ∈ R m × d , W L + 1 ∈ R d × m and W ℓ ∈ R m × m ; σ is the ReLU activation. We also denote q ij, -1 to be X ij .

We also follow the notation in Allen-Zhu et al. [1] and denote D i,ℓ ∈ R m × m to be a diagonal matrix and ( D i,ℓ ) kk = ✶ ( W ℓ q ij,ℓ -1 ) k > 0 for k = 1 , /uni22EF , m . Then q ij,ℓ = D ij,ℓ W ℓ q ij,ℓ -1

For the objective (7), the gradient w.r.t. to the k th row of W ℓ for ℓ = 1 , /uni22EF , L is the following

Throughout the proof, we use both ¯ L em ( θ ) and ¯ L em ( W ) to represent the same value of the loss function, where W = ( W 1 , /uni22EF , W L ) , and we let a = b = 1 2 . Next, we will prove Theorem 1.

Proof of Theorem 1. First by Lemma 4,

Also, ∥∇ ¯ L em ( θ )∥ ≤ √ L max ℓ ∥∇ W ℓ ¯ L em ( θ )∥ . Then we have

Let h = Θ ( nN m min j w ( t j )( t j -t j -1 ) ¯ σ t j ) and k = O( d 1 -a 0 2 n 2 N log ( d ϵ train )) , where ϵ train > 0 is some small constant. Then ω = O( log ( d ϵ train ) d 1 -a 0 2 n 3 N 5 / 2 L 1 / 2 √ m √ max j w ( t j )( t j -t j -1 ) ¯ σ t j ∑ k w ( t j )( t j -t j -1 )/ ¯ σ t j min j w ( t j )( t j -t j -1 ) ¯ σ t j ) and by Lemma 8,

with probability at least 1 -O( nN ) exp (-Ω ( d 2 a 0 -1 )) ,

where C > 0 is some constant, a 0 ∈ ( 1 / 2 , 1 ) ; the second inequality follows from Lemma 7 with

D.1 Proof of lower bound of the gradient at the initialization

In this section, we will show the main part of the convergence analysis, which is the following lower bound of the gradient.

Lemma 1 (Lower bound) . With probability 1 -O( nN ) exp (-Ω ( d 2 a 0 -1 )) , we have

where ( i ∗ , j ∗ ) = arg max ∥ √ w ( t j )( t j -t j -1 ) ¯ σ t j ( ¯ σ t j W L + 1 q ij,L + ξ ij )∥ , 1 2 < a 0 < 1 , and C 6 > 0 is some universal constant.

Below is the proof sketch of Lemma 1.

Proof sketch. We first decompose the gradient of the k th row of W L ∇ ( W L ) k ¯ L em ( θ ) 1 n w ( t j ∗ )( t j ∗ -t j ∗ -1 )( W L + 1 ) k ⊺ ( ¯ σ t j ∗ W L + 1 q i ∗ j ∗ ,L + ξ i ∗ j ∗ ) q i ∗ j ∗ ,L -1 1 ( W L q i ∗ j ∗ ,L -1 ) k > 0

where ( i ∗ , j ∗ ) indicates the sample index with the largest loss value.

Then we first fix ( q ij,L -1 ) s = 1 , and prove that the index set of both ( q i ∗ j ∗ ,L ) s > 0 and ∑ ( i,j )≠( i ∗ ,j ∗ ) w ( t j )( t j -t j -1 ) ¯ σ t j 1 ( W L q ij,L -1 ) k > 0 ( q ij,L ) s > 0 is order m with high probability.

=

Next, we conditioned on the index set we've found, then we can decouple each element of ∇ ( W L ) k ¯ L em with high probability. We prove that with high probability

for some constant c > 0 and 1 2 < a 0 < 1 . Based on this, we show that with probability at least 1 -O( nN ) exp (-Ω ( d )) ,

for some c > 0 . Then we prove that with probability at least 1 -exp (-Ω ( md a 0 -1 2 ))

with high probability, the event (∇ 1 ) s > 0 and (∇ 2 ) s > 0 has probability at least of order d ( a 0 -1 )/ 2 where a 0 ∈ ( 1 / 2 , 1 ) .

Now, we deal with ( q ij,L -1 ) s and prove that if the above results hold for ( q ij,L -1 ) s = 1 , then there exists an index set with cardinality of order m /( nN ) such that (∇ 1 ) s > 0 and (∇ 2 ) s > 0 also hold in this index set.

In the end, combining all the steps above yields the lower bound.

Here is the complete proof.

Proof. The main idea of the proof of lower bound is to decouple the elements in the gradient and incorporate geometric view. We focus on ∇ W L ¯ L em ( θ ) .

Step 1 : Rewrite ∇ ( W L ) k ¯ L em ( θ ) to be the ( i ∗ , j ∗ ) th term g 1 plus the rest nN -1 terms g 2 .

Then

Also define

Our goal is to show that with high probability, there are at least O( md a 0 -1 2 nN ) number of rows k such that ∇ 11 ,s ≥ 0 , ∇ 12 ,s ≥ 0 , ∇ 21 ,s ≥ 0 , ∇ 22 ,s ≥ 0 . Then we can lower bound ∥∇ ( W L ) k ¯ L em ( θ )∥ 2 by ∥∇ 1 ∥ 2 , which can be eventually lower bounded by ¯ L em ( θ ) .

Step 2 : Consider [∇ ( W L ) k ¯ L em ( θ )] s . For ( g 2 ) s , first take ( q ij,L -1 ) s = 1 for all ( i, j ) ≠ ( i ∗ , j ∗ ) . Then we only need to consider

which is independent of s . For ∇ 1 , since q i ∗ ,j ∗ ,L -1 ≥ 0 which does not affect the sign of this term, we can also first take ( q i ∗ ,j ∗ ,L -1 ) s = 1 for all s .

Step 3 : We focus on ∇ 11 and ∇ 21 and we would like to pick the non-zero elements in this two terms. More precisely, let

Let α ij = w ( t j )( t j -t j -1 ) ¯ σ t j ( W L q ij,L -1 ) k > 0 ≥ 0 . Then

If

then there must be at least one pair of ( i, j ) s.t. α ij ( W L q ij,L -1 ) s = α ij σ ( W L q ij,L -1 ) s > 0 , which implies ∑ ( i,j )≠( i ∗ ,j ∗ ) α ij ( q ij,L ) s > 0 . Therefore, it suffices to consider

Since ( q ij,L -1 ) s ≥ 0 , we have

By Lemma 2 and Proposition 2, we have

Also ( W L ) s 's are i.i.d. multivariate Gaussian. By Chernoff bound,

for some small δ 1 ≤ 1 4 and δ 2 ≤ 4 , i.e., ∣ N 1 ∩ N ′ 2 ∣ = Θ ( m ) with probability at least 1 -2 e -Ω ( m ) . Step 4 : Next we condition on N 1 ∩ N ′ 2 and consider ( W L + 1 ) k ⊺ W L + 1 ¯ σ t j ∗ q i ∗ j ∗ ,L +( W L + 1 ) k ⊺ ξ i ∗ j ∗ and ( W L + 1 ) k ⊺ W L + 1 ∑ ( i,j )≠( i ∗ ,j ∗ ) α ij q ij,L +( W L + 1 ) k ⊺ ∑ ( i,j )≠( i ∗ ,j ∗ ) ¯ α ij ξ ij , where ¯ α ij = α ij / ¯ σ ij . We would like to prove that with high probability

for some constant c > 0 and 1 2 < a 0 < 1 .

First, since ξ ij ∼ N( 0 , I d ) , by Bernstein's inequality, with probability at least 1 -exp (-Ω ( d )) , we have ∥ ξ ij ∥ 2 = Θ ( d ) . Similarly, since ( W L + 1 q ij,L ) s ∼ N( 0 , 2 ∥ q ij,L ∥ 2 m ) , by Berstein's inequality and Lemma 4, with probability at least 1 -exp (-Ω ( d )) , we have ∥ W L + 1 q ij,L ∥ 2 = Θ ( d ) . By union bounds, the above holds for all i, j with probability at least 1 -2 nN exp (-Ω ( d )) .

Let v = W L + 1 ∑ ( i,j )≠( i ∗ ,j ∗ ) α ij q ij,L +∑ ( i,j )≠( i ∗ ,j ∗ ) ¯ α ij ξ ij ∥ W L + 1 ∑ ( i,j )≠( i ∗ ,j ∗ ) α ij q ij,L +∑ ( i,j )≠( i ∗ ,j ∗ ) ¯ α ij ξ ij ∥ and u = W L + 1 ¯ σ t j ∗ q i ∗ j ∗ ,L . For notational simplicity, we use ξ to denote ξ i ∗ j ∗ . Fix v, u and consider the probability of event A

for some c 0 > 0 and 1 2 < a 0 < 1 .

Then consider the following event that has larger probability than A

Since ( v ⊺ u ) 2 ≤ ∥ u ∥ 2 where the equality holds when v = u ∥ u ∥ , we have

Also, since ∥ u ∥ 2 = O( d ) with probability at least 1 -2 nN exp (-Ω ( d )) , we have

for some constant c > 0 .

Therefore, with probability at least 1 -O( nN ) exp (-Ω ( d 2 a 0 -1 ))

for some constant c > 0 .

Then

where the second equality follows from Lemma 2; the third equality follows from series expansion; the forth equality follows from (16); c ′ , c ′′ , C > 0 are some constants.

Thus with probability at least 1 -C ( d 1 -a 0 2 ) d -1 √ d ,

for some c > 0 .

Then by Lemma 2, with probability at least 1 -O( nN ) exp (-Ω ( d )) ,

for some c > 0 .

Since ( W k L + 1 ) are iid Guassian vectors for k = 1 , /uni22EF , m , by Chernoff bound on Bernoulli variable ✶ {( W k L + 1 ) ⊺ v ≥ 0 , ( W k L + 1 ) ⊺ ( u + ξ )≥ 0 } , we have, with probability at least 1 -exp (-Ω ( md a 0 -1 2 )) ∣{ k ∶ ( W k L + 1 ) ⊺ v ≥ 0 , ( W k L + 1 ) ⊺ ( u + ξ ) ≥ 0 }∣ = Θ ( md a 0 -1 2 ) .

Step 5: Combining the above 4 steps, we would like to obtain the lower bound of the gradient.

For each k , consider ( q ij,L -1 ) s for ( i, j ) ≠ ( i ∗ , j ∗ ) and denote q s = {( q ij,L -1 ) s } ( i,j )≠( i ∗ ,j ∗ ) = { σ (( W L -1 ) s q ij,L -2 )} ( i,j )≠( i ∗ ,j ∗ ) . Let ¯ q s = {( W L -1 ) s q ij,L -2 } ( i,j )≠( i ∗ ,j ∗ ) and ¯ q s ∼ N( 0 , QQ ⊺ ) , where each row of Q is q ⊺ ij,L -2 for ( i, j ) ≠ ( i ∗ , j ∗ ) . Thus, q s is ¯ q s projected to the nonnegative orthant.

Let 1 = ( 1 , 1 , /uni22EF , 1 ) ∈ R nN -1 . Therefore, if ⟨ β k , 1 ⟩ ≥ 0 for some β k ∈ R nN -1 , then at least half of the nonnegative orthant is contained in { v ∈ R nN -1 ∶ ⟨ β k , v ⟩ ≥ 0 } , i.e., there exists a constant c k > 0 , s.t.

Then since β k ∈ R nN -1 for k = 1 , /uni22EF , m and nN ≪ m , there exists a set of indices K ⊆ { 1 , /uni22EF , m } with ∣K∣ = Θ ( m nN ) and a set of indices S ⊆ { 1 , /uni22EF , m } with ∣S∣ = Θ ( m ) , s.t., ⟨ β k , q s ⟩ ≥ 0 , for k ∈ K , s ∈ S .

Let q K ij,ℓ = ( q ij,ℓ ) k ∈K . Then by Bernstein's inequality, we can also obtain that ∥ q K ij,ℓ ∥ 2 = Θ ( d ) with probability at least 1 -nN exp (-Ω ( d )) .

Combine all of the above and apply the Claim 9.5 in Allen-Zhu et al. [1], we obtain, with probability at least 1 -O( nN ) exp (-Ω ( d 2 a 0 -1 )) ,

where C 6 > 0 is some universal constant, 1 2 < a 0 < 1 , and the second inequality follows from the definition of i ∗ , j ∗ .

D.1.1 Geometric ideas used in the proof

Proposition 2. Consider w ∼ N( 0 , I ) , where w ∈ R n . Then ∥ w ∥ and w ∥ w ∥ are independent random variables and w ∥ w ∥ ∼ Unif ( S n -1 ) .

Lemma 2. Let w ∼ Unif ( S n -1 ) , where S n -1 = { x ∈ R n ∣∥ x ∥ = 1 } . Then for two vectors v 1 , v 2 ∈ R n ,

Proof. Since w ∼ Unif ( S n -1 ) , we only need to consider the area of the event. It is obvious that the set { w ∈ S n -1 ∣ w ⊺ v i } is a semi-hypersphere. Therefore, we only need to consider the intersection of two semi-hypersphere, i.e.,

Next we follow the notations and definitions in Lee and Kim [33]. Consider the unit hypersphere in R d , S d -1 = { x ∈ R d ∣ ∥ x ∥ = 1 } . The area of S d -1 is

Lemma 3. Fix ξ 1 ∈ S d -1 and let ξ 2 ∼ Unif ( S d -1 ) , where S d -1 = { x ∈ R d ∣∥ x ∥ = 1 } . Then with probability at least 1 -exp (-Ω ( d )) , we have ∠( ξ 1 , ξ 2 ) ≤ 3 π 4 .

Proof. For any fixed ξ 1 , all the ξ 2 's that satisfy ∠( ξ 1 , ξ 2 ) ≥ π -θ are on a hyperspherical cap. By Lee and Kim [33], the area of the hypersperical cap is

Then

Let θ = π 4 < 1 . Then with probability at least 1 -exp (-Ω ( d )) , we have ∠( ξ 1 , ξ 2 ) ≤ 3 π 4 .

D.2 Proofs related to random initialization

Consider W i = W ( 0 ) i in this section.

Lemma 4. If ϵ ∈ ( 0 , 1 ) , with probability at least 1 - O( nN ) e -Ω ( min ( ϵ 2 d 4 b -1 ,ϵd 2 b )) , ∥ X ij ∥ 2 ∈ [∥ e -µ t j x i ∥ 2 + ¯ σ 2 t j d -ϵ ¯ σ 2 t j d 2 b , ∥ e -µ t j x i ∥ 2 + ¯ σ 2 t j d + ϵ ¯ σ 2 t j d 2 b ] for all i = 1 , /uni22EF , n and j = 0 , /uni22EF , N -1 . Moreover, with probability at least 1 -O( L ) e -Ω ( mϵ 2 / L ) over the randomness of W s for s = 0 , /uni22EF , L , we have ∥ q ij,ℓ ∥ ∈ [∥ X ij ∥( 1 -ϵ ) , ∥ X ij ∥( 1 + ϵ )] for fixed i, j . Therefore, with probability at least 1 -O( nNL ) e -Ω ( min ( mϵ 2 / L,ϵ 2 d 4 b -1 ,ϵd 2 b )) , we have Ω ( d b ) = ∥ q ij,ℓ ∥ = O( d a ) .

Proof. Consider 1 ¯ σ t j X ij = e -µt j ¯ σ t j x i + ξ ij . Since ξ ij ∼ N( 0 , I ) , ∥ 1 ¯ σ t j X ij ∥ 2 follows from the noncentral χ 2 distribution and E ∥ 1 ¯ σ t j X ij ∥ 2 = d +∥ e -µt j ¯ σ t j x i ∥ 2 (this includes the time variable at the d th dimension). By Berstein inequality,

Therefore, with probability at least 1 -O( nN ) e -Ω ( min ( ϵ 2 d 4 b -1 ,ϵd 2 b )) , ∥ X ij ∥ 2 ∈ [∥ e -µ t j x i ∥ 2 + ¯ σ 2 t j d -ϵ ¯ σ 2 t j d 2 b , ∥ e -µ t j x i ∥ 2 + ¯ σ 2 t j d + ϵ ¯ σ 2 t j d 2 b ] for all i = 1 , /uni22EF , n and j = 0 , /uni22EF , N -1 , where ϵ ∈ ( 0 , 1 ) . The second part of the Lemma follows the similar proof in Lemma 7.1 of Allen-Zhu et al. [1]. The last part follows from union bound and Assumption 1.

Lemma 5 (Upper bound) . Under the random initialization of W i for i = 0 , /uni22EF , L , with probability at least 1 -O( nNL ) e -Ω ( min ( mϵ 2 / L,ϵ 2 d 4 b -1 ,ϵd 2 b )) , we have

Proof. For any ℓ = 1 , ⋅ , L , we have

where the first inequality follows from Young's inequality; the second inequality follows from Lemma 4 and Lemma 7.4 in Allen-Zhu et al. [1]; C 7 > 0

D.3 Proofs related to perturbation

Consider W per i = W ( 0 ) i + W ′ i for i = 1 , /uni22EF , L in this section. We follow the same idea in Allen-Zhu et al. [1] to consider the network value of perturbed weights at each layer. We use the superscript ' per ' to denotes the perturbed version, i.e.,

We also similarly define the diagonal matrix D per ij,ℓ for the above network.

The following Lemma measures the perturbation of each layer. The lemma differs from Lemma 8.2 in Allen-Zhu et al. [1] by a scale of d a . For sake of completeness, we state it in the following and the proof can be similarly obtained.

Lemma 6. Let ω ≤ 1 C 7 L 9 / 2 ( log m ) 3 d a for some large C > 1 . With probability at least 1 -exp (-Ω ( d a mω 2 / 3 L )) , for any ∆ W s.t. ∥ ∆ W ∥ ≤ ω , we have

D.4 Proofs related to the evolution of the algorithm

Lemma 7 (Upper and lower bounds of gradient after perturbation) . Let

Consider θ per s.t. ∥ θ per -θ ∥ ≤ ω , where θ follows from the Gaussian initialization. Then with probability at least 1 -O( nN ) e -Ω ( d 2 a 0 -1 ) ,

for ℓ = 1 , /uni22EF , L .

Proof. Consider the following terms

Then

where the first two inequalities follow the same as the proof of Lemma 5; the third inequality follows from Lemma 6; the last inequality follows from the definition of ω .

Also, we have

Then

where the first inequality follows from Young's inequality and the above decomposition; the second inequality follows from Lemma 7.4, 8.7 in Allen-Zhu et al. [1] (with s = O( d a mω 2 / 3 L ) ) and Lemma 6; the last inequality follows from the definition of ω .

For upper bound, we only need to consider ∇ per ( W ℓ ) k ¯ L em ( θ ) and ∇ ( W ℓ ) k ¯ L em ( θ per ) . By similar argument as Lemma 5, with probability at least 1 -O( nNL ) e -Ω ( min ( mϵ 2 / L,ϵ 2 d 4 b -1 ,ϵd 2 b )) , we have

Then

Also,

Note when interpolation is not achievable, this lower bound is always away from 0, which means the current technique can only evaluate the lower bound outside a neighbourhood of the minimizer. More advanced method is needed and we leave it for future investigation.

Lemma8 (semi-smoothness) . Let ω = Ω . With probability at least 1 -e -Ω ( log m ) over the randomness of θ ( 0 ) , we have for all θ s.t. ∥ θ -θ ( 0 ) ∥ ≤ ω , and all θ per s.t. ∥ θ per -θ ∥ ≤ ω ,

Proof. By definition,

Similar to the proof of Theorem 4 in Allen-Zhu et al. [1], we obtain the desired bound by using CauchySchwartz inequality. Note, in our case, due to the order of input data, we choose s = O( d a mω 2 / 3 L ) in Allen-Zhu et al. [1] and therefore the bound is slightly different from theirs.

E Proofs for sampling

In this section, we prove Theorem 2. The proof includes two main steps: 1. decomposing KL ( p δ ∣ q T -δ ) into the initialization error, the score estimation errors and the discretization errors; 2. estimating the initialization error and the discretization error based on our assumptions. In the following context, we introduce the proof of these two steps separately.

Proof of Theorem 2. Step 1: The error decomposition follows from the ideas in [12] of studying VPSDE-based diffusion models. According to the chain rule of KL divergence, we have

Apply the chain rule again for at across the time schedule ( T -t ← j ) 0 ≤ j ≤ N -1 , the second term can be written as

where the second inequality follows from Lemma 9. Therefore, the error decomposition writes as

where the three terms in (20) quantify the initialization error, the score estimation error and the discretization error, respectively.

Step 2: In this step, we estimate the three error terms in Step 1. First, recall that p T = p ∗N( 0 , ¯ σ 2 T I d ) and q 0 = N( 0 , ¯ σ 2 T I d ) , hence the initialization error KL ( p T ∣ q 0 ) can be estimated as follows,

where the inequality follows from Lemma 10. Hence we recover the term E I in (11). Next, since σ t is non-decreasing in t , the score estimation error can be estimated as

Hence, we recover the term E S in (11).

Last, we estimated the discretization error term. Our approach is motivated by analyses of VPSDEs in [8, 26]. We defines a process L t ∶= ∇ log p T -t ( Y t ) . Then we can relate discretization error to quantities depending on L t , and therefore bound the discretization error via properties of { L t } 0 ≤ t ≤ T . According to Lemma 12, we have

Since t ↦ σ t is non-decreasing and t ↦ E [ tr ( Σ T -t ( X T -t ))] is non-increasing, we have

Therefore, we obtain

The above bound depends on E [ tr ( Σ t ( X t ))] , hence we estimate E [ tr ( Σ t ( X t ))] for different values of t .

First, we have

Meanwhile,

Therefore, E [ tr ( Σ t ( X t ))] ≤ min ( m 2 2 , ¯ σ 2 t d ) ≲ ( 1 -e -¯ σ 2 t )( m 2 2 + d ) . Plug this estimation into (23) and we get

where the last inequality follows from the definition of ¯ σ t and integration by parts. The proof of Theorem 2 is completed.

Lemma 9. Let { Y t } 0 ≤ t ≤ T be the solution to (2) with f t ≡ 0 and p ← t + s ∣ s (⋅∣ y ) be the conditional distribution of Y s + t given { Y s = y } . Let { ¯ Y t } 0 ≤ t ≤ T be the solution to (11) q t + s ∣ s (⋅∣ y ) be the conditional distribution of ¯ Y s + t given { ¯ Y s = y } . Then for any fixed t ∈ ( 0 , γ j ] , we have

Proof of Lemma 9. According to [12, Lemma 6], we have

where the last inequality follows from Young's inequality. Therefore, Lemma 9 is proved after taking another expectation.

Lemma 10. For any probability distribution p satisfying Assumption 3 and q being a centered multivariate normal distribution with covariance matrix σ 2 I d , we have

Proof of Lemma 10.

where the inequality follows from convexity of KL (⋅∣ q ) and the second identity follows from KLdivergence between multivariate normal distributions.

Lemma 11. Let { X t } 0 ≤ t ≤ T be the solution to (1) with f t ≡ 0 and p 0 ∣ t (⋅∣ x ) be the conditional distribution of X 0 given { X t = x } . Define

Let { Y t } 0 ≤ t ≤ T be the solution to (2) with f t ≡ 0 and q 0 ∣ t (⋅∣ x ) be the conditional distribution of Y 0 given { Y t = x } . Define

Then we have for all t ∈ ( 0 , T ) ,

Proof of Lemma 11. We first represent ∇ logp t ( X t ) and ∇ 2 log p t ( X t ) via m t ( X t ) and Σ t ( X t ) . Since { X t } 0 ≤ t ≤ T solves (1), X t = X 0 + ¯ σ t ξ with ( X 0 , ξ ) ∼ p ⊗N( 0 , I d ) . Therefore, according to Bayes rule, we have

where the second identity follows from the fact that p t ∣ 0 (⋅∣ x ) = N( x, ¯ σ 2 t I d ) . The last identity follows from the definition of m t ( X t ) in Lemma 11. Similarly, according to Bayes rule, we can compute

where the second identity follows from the fact that p t ∣ 0 (⋅∣ x ) = N( x, ¯ σ 2 t I d ) and the definition of Σ t ( X t ) in Lemma 11.

According to Bayes rule, we have

and

where Z t = ∫ exp ( h t ( x )) p ( dx ) is a (random) normalization constant. From the above computations, we can see that q 0 ∣ t ( dx ∣ Y t ) = p 0 ∣ T -t ( dx ∣ X T -t ) for all t ∈ [ 0 , T ] . Therefore, we have

where the identities hold in distribution. Therefore, to prove the first statement, it suffices to compute d ¯ m t ( Y t ) . To do so, we first compute dh t ( x ) , d [ h ( x ) , h ( x )] t , dZ t and d log Z t .

According to the definition of Y t and (26), we have

Therefore

Apply (29) and (30) and we get

and

If we further define R t ( Y t ) ∶= q 0 ∣ t ( dx ∣ Y t ) p ( dx ) = Z -1 t exp ( h t ( x )) . We have

With (29), (30), (31), (32) and (33), we have

where most terms cancel in the last identity. Therefore, the first statement is proved. Next, we prove the second statement. We have

where the second last identity follows from (34) and the last identity follows from the definition of ¯ σ t . Last, we reverse the time and get

The proof is completed.

Lemma 12. Under the conditions in Lemma 11, let { Y t } 0 ≤ t ≤ T be the solution to (2) with f t ≡ 0 . Define L t ∶= ∇ log p T -t ( Y t ) , then for any t ∈ [ t ← j , t ← j + 1 ) , we have

Proof of Lemma 12. First, according to the definition of L t and Y t , it follows from Itô's lemma that

where the last step follows from applying the Fokker Planck equation of (1) with f t ≡ 0 , i.e., ∂ t p t = σ 2 t ∆ p t . Most of the terms are cancelled after applying the Fokker Planck equation. Now, for fixed s > 0 and t > s , define E s,t ∶= E [∥ L t -L s ∥ 2 ] . Apply Itô's lemma and (35), we have

where ∥ A ∥ F denotes the Frobenius norm of any matrix A . According to (27), we have

where the last identity follows from the proof of Lemma 11. Therefore, for any t ∈ [ t ← j , t ← j + 1 ) , we have

The proof is completed.

F Sampling error for Gaussian data distributions

In this section, we consider a special case when the data distribution is a mixture of Gaussians, i.e.,

where N( x ; m,σ 2 I d ) is the density of Gaussian random vector with mean m and covariance σ 2 I d . In this case, the score function ∇ log p t ( x ) can be explicitly calculated from any t > 0 , see Lemma 14. Therefore, the sampling process (11) can be implemented with zero score estimation error via the following piecewise SDE: for any t ∈ [ t ← j , t ← j + 1 ) ,

The iterates, ( ¯ Y t ← j ) , are all Gaussians with explicit means and covariance matrices, see Lemma 15. As a consequence, we can quantify the quantity, KL ( p δ ∣ q T -δ ) in Theorem 2 explicitly since both distributions are Gaussians.

Lemma 13 (KL-divergence error for Gaussian data distribution) . Assume the data distribution has density given by (39) . Let ( ¯ Y t ← j ) N j = 0 be defined by (40) with initial condition ¯ Y 0 ∼ N( 0 , ¯ σ 2 T I d ) . Denote q t = Law ( ¯ Y t ) for all 0 ≤ t ≤ T -δ . Then

where E σ is a positive constant depending on the variance schedule ( ¯ σ T -t ← j ) N j = 0 , given by

Proof of Lemma 13. p δ = p ∗N( 0 , ¯ σ 2 δ ) = N( m, ( σ 2 + ¯ σ 2 δ ) I d ) and q T -δ = Law ( ¯ Y t ← N ) = N( m N , Σ N ) with ( m N , Σ N ) given in Lemma 15. Therefore, we have

Lemma 14 (Explicit score function for mixture of Gaussian target) . Assume the data distribution has density given by (39) , then the score function is given by

Proof. Since the forward process (1) with f t ≡ 0 is the just a process that keeps adding noise, the density p t along the process is a convolution between data density and a Gaussian density with mean zero and covariance ¯ σ 2 t I d :

Therefore, we have

(42) follows directly from the above computations.

Lemma 15 (Gaussian iterates along the trajectory) . Assume the data distribution has density given by (39) . Let ( ¯ Y t ← j ) be defined by (40) with initial condition ¯ Y 0 ∼ N( 0 , ¯ σ 2 T I d ) . Then for all 0 ≤ j ≤ N , ¯ Y t ← j ∼ N( m j , Σ j ) with

Proof of Lemma 15. According to Lemma 14, (40) can be written as

which implies that for any j ∈ 0 , 1 , /uni22EF , N -1 :

where ( U j ) N j = 1 are i.i.d. standard Gaussian vectors in R d . Since ¯ Y t ← 0 is Gaussian, by induction, we prove that ¯ Y t ← j is Gaussian for all j = 1 , /uni22EF , N . Denote ¯ Y t ← j ∼ N( m j , Σ j ) . According to (45) and the independence between U j + 1 and ¯ Y t ← j , we have

Again, according to (45) and the independence between U j + 1 and ¯ Y t ← j , we get a relation between consecutive covariance matrices:

G Full error analysis

Proof of Theorem 3. We only need to deal with E S . By applying the same schedules to training objective, we obtain

Together with

H Proofs for Section 4.1

H.1 Proof of 'bell-shaped' curve

Proof of Proposition 1. Fix x i , ξ ij , ¯ σ t N . By definition of the network S ( θ ; t j , X ij ) , it is continuous with respect to X ij .

For 1, X ij = x i + ¯ σ t j ξ ij , and thus S ( θ ; t j , X ij ) is also continuous w.r.t. ¯ σ t j . Also, since ¯ σ t j ∈ [ 0 , ¯ σ t N ] , there exists M 0 > 0 , s.t.,

Then for any ϵ 1 > 0 , there exists δ = ϵ 1 √ dM 0 > 0 , s.t., ∀ 0 ≤ ¯ σ t j < δ 1 , we have

For 2, by the positive homogeniety of ReLU,

Consider ¯ σ t j ≥ M , for some M > 0 . Then

For any y ∈ D( M ) , where D( M ) = { y ∈ R d ∶ y s ∈ [( ξ ij ) s - ∣( x i ) s ∣/ M, ( ξ ij ) s + ∣( x i ) s ∣/ M ] , ∀ s = 1 , /uni22EF , d } ,

Since S is differentiable a.e., by the fundamental theorem of calculus,

where M 1 = max s ( x i ) s ⋅ ess sup x ∈D( M 2 ) ∥ S ′ x ( θ ; t j , x )∥ < +∞ for some fixed 0 < M 2 < M .

Combining (46),(47), and (48), we have

Then ∀ ϵ 2 > 0 , there exists M = max { M 1 + √ M 2 1 + 4 ϵ 2 ∥ ξ ij ∥ 2 ϵ 2 , M 2 } > 0 , s.t., when ¯ σ t j > M , we have

∥ ¯ σ t j S ( θ ; t j , x i + ¯ σ t j ξ ij ) + ξ ij ∥ ≥ M 2 (∥ S ( θ ; t j , ξ ij )∥ -ϵ 2 ) .

Then

H.2 Proof of optimal rate

Proof of Corollary 2. If ∣ f ( θ ( k ) ; i, j ) -f ( θ ( k ) ; l, s )∣ ≤ ϵ for all i, j, l, s and k > K , then by Lemma 1 and 7, we choose the maximum f ( θ ( k ) ; i, j ) for the lower bound, which is of order O( ϵ ) away from the other f ( θ ( k ) ; i, j ) ′ s . Therefore, we can take j ∗ ( k ) = arg max j f ( θ ( k ) ; i, j ) and absorb the O( ϵ ) error in constant factors. Then the result naturally follows.

H.3 Proof of comparisons of E S

Recall that the training objective of EDM is defined in the following

Let β j = C 1 β EDM , i.e.,

EDM. Consider ¯ σ t = t and t j = ( ¯ σ 1 / ρ max -( ¯ σ 1 / ρ max -¯ σ 1 / ρ min ) N -j N ) ρ for j = 0 , /uni22EF , N . Then

Then the maximum of σ 2 t j w ( t j ) = t j w ( t j ) appears at j = N

Song et al. [46]. Consider ¯ σ t = √ t and t j = ¯ σ 2 max ( ¯ σ 2 min ¯ σ 2 max ) N -j N for j = 0 , /uni22EF , N . Then

Then

I Proofs for Section 4.2

I.1 Proof when E I + E D dominates.

Under the EDM choice of variance, ¯ σ t = t for all t ∈ [ 0 , T ] , and study the optimal time schedule when E D + E I dominates. First, it follows from Theorem 2 that

Based on the above time schedule dependent error bound, we quantify the errors under polynomial time schedule and exponential time schedule.

Polynomial time schedule. we consider T -t ← j = ( δ 1 / a +( N -j ) h ) a with h = T 1 / a -δ 1 / a N and a > 1 , γ j = a ( δ 1 / a +( N -j -ϑ ) h ) a -1 h for some ϑ ∈ ( 0 , 1 ) . We have γ j / h ∼ a ( T -t ← j ) a -1 a and

Therefore, to obtain E I + E D ≲ ε , it suffices to require T = Θ ( m 2 ε 1 / 2 ) and the iteration complexity

For fixed m 2 , δ and ε , optimal value of a that minimizes the iteration complexity N is a = 1 2 ln ( m 2 δε 1 / 2 ) . Once we let δ = ¯ σ min , T = ¯ σ max = Θ ( m 2 ε 1 / 2 ) and a = ρ , the iteration complexity is

and it is easy to see that our theoretical result supports what's empirically observed in EDM that there is an optimal value of ρ that minimizes the FID.

Exponential time schedule. we consider γ j = κ ( T -t ← j ) with κ = ln ( T / δ ) N , we have

Therefore, to obtain E I + E D ≲ ε , it suffices to require T = Θ ( m 2 ε 1 2 ) and the iteration complexity

When m 2 ≤ O ( √ d ) , the exponential time schedule is asymptotic optimal, hence it is better than the polynomial time schedule when the initilization error and discretization error dominate. Once we let δ = ¯ σ min , T = ¯ σ max = Θ ( m 2 ε 1 / 2 ) , the iteration complexity is

Now we adopt the variance schedule in [46], ¯ σ t = √ t for all t ∈ [ 0 , T ] , it follows from Theorem 2 that

Polynomial time schedule. we consider T -t ← j = ( δ 1 / a +( N -j ) h ) a with h = T 1 / a -δ 1 / a N and a > 1 , γ j = a ( δ 1 / a +( N -j -ϑ ) h ) a -1 h for some ϑ ∈ ( 0 , 1 ) . We have γ j / h ∼ a ( T -t ← j ) a -1 a and

Therefore, to obtain E I + E D ≲ ε , it suffices to require T = Θ ( m 2 2 ε ) and the iteration complexity

Once we let δ = ¯ σ 2 min , T = ¯ σ 2 max = Θ ( m 2 2 ε ) and a = ρ , the iteration complexity is

Compared to exponential time schedule with the EDM choice of variance schedule, this iteration complexity is worse up to a factor ( ¯ σ max ¯ σ min ) 1 / ρ .

Exponential time schedule. we consider γ j = κ ( T -t ← j ) with κ = ln ( T / δ ) N , we have

Therefore, to obtain E I + E D ≲ ε , it suffices to require T = Θ ( m 2 2 ε ) and the iteration complexity

Once we let δ = ¯ σ 2 min , T = ¯ σ 2 max = Θ ( m 2 2 ε ) and a = ρ , the iteration complexity is

Compared to exponential time schedule with the EDM choice of variance schedule, this iteration complexity has the same dependence on dimension parameters m 2 , d and the minimal/maximal variance ¯ σ min , ¯ σ max .

Optimality of Exponential time schedule. For simplicity, we assume m 2 2 = O( d ) . Then under both schedules in [30] and [46], E I s only dependent on T , and are independent of the time schedule. Both E D s satisfy

Let τ j = ln ( T -t ← j T -t ← j + 1 ) ∈ ( 0 , ∞) . Then γ j T -t ← j = 1 -e -τ j and ∑ δ < T -t ← j < T τ j = ln ( T / δ ) is fixed. Since x ↦ ( 1 -e -x ) 2 is convex on the domain x ∈ ( 0 , ∞) , according the Jensen's inequality, ∑ δ < T -t ← j < T γ 2 j ( T -t ← j ) 2 reaches its minimum when τ j are constant-valued for all j , which implies the exponential schedule is optimal to minimize E D , hence optimal to minimize E D + E I .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state it in Section 1 and Appendix A.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 1 and Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer:[Yes]

Justification: See Appendix and Theorems in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: It has been confirmed by Ethics reviewers.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Section 1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We show new methods and theorems.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes both the training and sampling errors of diffusion models. The analysis sheds light on two practically relevant design choices, which are the noise level weighting (during training) and discretization (during sampling).\n\nSTRENGTHS:\n- Figure 1 presents a clear qualitative picture of how to choose noise level schedules for training and sampling. In particular, this is the first theoretical work (to the best of my knowledge) to show how the sampling discretization schedule should be adapted depending on score errors arising because of imperfect training.\n- The sampling theoretical analysis is thorough and is very up-to-date with the sampling convergence literature. It extends previous results from the variance-preserving to the variance-exploding case, but the extension seems straightforward.\n- I am however not knowledgeable enough about the optimization literature to speak of the novelty of the training error results.\n\nWEAKNESSES:\nAll the theoretical results are very hard to even parse, let alone extract intuition or guidance for practice. The authors argue that their results provide justification for the design choices of Karras et al. [31], but it seems to me that they are shoehorned to fit the chosen narrative. In other words, it is not clear to me that the exact same error bounds could not be used to justify any other schedule used in practice. I find Section 4.1 in particular to be very handwavy. I appreciate the attempt to connect theoretical results to practical choices, but I think the connection is too tenuous here.\n\nSome claims made in the text seem unsubstantiated to me:\n- The authors claim that the NTK can only be analyzed in very restricted settings (lines 103-104). I don't see why this is true, as the NTK applies to all architectures in the lazy regime.\n- I don't see how Theorem 1 implies that GD has an exponential convergence (lines 223-224) as product factors depend on $s$.\n- I don't see how Theorem 2 implies that the sampling error is almost linear in $d$ as $\\gamma_k$ and $\\sigma_t$ may depend on $d$.\n\nMinor:\n- line 140 and eq (4): $C$ should be $C_t$ as it depends on $t$\n- line 150: aspectS\n- line 154: by A deep ReLU\n- line 176: introduce $W$ before line 174\n- line 364: we maintain a fixed ~the~ total weighting\n\nQUESTIONS:\n- I don't understand the sentence in lines 296-299, which mentions empirical and population losses even though the paper does not tackle the generalization error. Also, I don't see how the central limit theorem applies here, as individual terms in the empirical average over the training set are not independent since the network parameters themselves depend on the training set. Could the authors please clarify?\n- Do the authors believe that their work can inform practical decisions? If yes, what new predictions can be made using their theoretical results? If no (which does not necessarily imply the results are not significant), I think the paper should be more honest about this.\n\nLIMITATIONS:\nContrary to what is stated in the checklist, limitations are not discussed in the paper. There is no discussion section.\n\n- A first limitation is that only the approximation and optimization errors are studied, and generalization is not tackled. This means that the score network learns the empirical distribution of the training set, and thus reproduces training data points during sampling. This is only briefly mentioned in passing in the middle of the text but should be stated in the introduction (and potentially discussion).\n\n- Some assumptions are questionable. First, the first and last layer are not trained and left to their random initialization. Second, and more importantly, the chosen asymptotic setting corresponds to extremely wide networks trained on very few data points (probably so that they can memorize their training set). This thus corresponds to a rather unrealistic setting.\n\nAll in all, I am divided about this paper. On the one hand, it provides a unified analysis of both training and sampling errors, which opens up understanding how to adapt the discretization schedule to score errors. On the other hand, independently of the mathematical correctness of the results (which were not checked), they seem very artificial (due to the restricted setting of the analysis in section 3, and the very qualitative arguments in section 4), and not clearly presented at all. It is thus not clear to me how they can be useful to the theory and practice communities. I recommend rejection, but I am willing to change my decision if the authors convince me that my assessment is incorrect.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops theoretical bounds on the training error and sample complexity of score-based diffusion models, which further imply a complete convergence analysis (by combining the both). In addition, the derived bounds shed light on the efficient hyper-parameters selection, including re-weighting functions, time discretization schemes and variance schedules, supporting former empirical works in a principled manner.\n\nSTRENGTHS:\n1. The problem setup is quite general, at least at the current stage.\n2. This work provides a comprehensive and rigorous convergence analysis of diffusion models for *both* training and sampling. The analyzed score network also extends former theoretical literature. \n3. The obtained mathematical bounds also imply beneficial insights to guide the hyper-parameters selection, which coincides with other experimental literature .\n\nWEAKNESSES:\n1. The paper is organized in a almost technical way, and some details are not easy to follow. It would be helpful to improve the readability by adding e.g., (key) notations collections and formulation illustrations. Also, there is no conclusion section.\n2. The derived bounds in main theorems are too technically complex. Is it possible to simplify them to summarize the core algebraic dependence on hyper-parameters (including the data dimension $d$, sample size $n$, model capacity $(m,L)$, discretization steps $N$)? \n3. There are no experiments to support the theory. Particularly, is it feasible to numerically verify the exponential convergence in Theorem 1? \n4. Are there any differences on the training between over-parameterized neural networks applied to normal supervised learning, and here in the generative modeling (as the score network)? Note that this work selects the *same* feed-forward network (without biases) for *all* time steps (see the equation between Line 173 and 174), ignoring the time information in the architecture of score networks.\n\nQUESTIONS:\n1. Please provide more details of questions raised in the weaknesses section above.\n2. The scale of model initializations is near-zero (see  Line 181-182), which seems to be one of the typical configurations in the over-parameterization regime (lazy training?). Is this setting reasonable and representative in practice? It would be better to provide (numerical) examples about this, particularly in this generative modeling case. \n3. Details 1: In Line 201, \"... high probability (see Lemma..).\" \n4. Details 2: What is the definition of $a$ in Theorem 1 (see $d^{2a-1.5-4c}$)?\n5. Details 3: In Proposition 1, there seems to be a confusion on the mixed use of $S$ and $S^*$ (particularly, Point 2). \n6. Details 4: In Line 297-299, it states that $\\epsilon_n$ (the distance between the empirical loss and population loss at the GD outputs) can be estimated by central limit theorem. Can the authors provide more details on it (since the learned model parameters are *dependent* of data)? \n7. Details 5: In Table 1, why is the exponential schedule selected as $\\bar{\\sigma}_t=\\sqrt{t}$? In VE-SDE, $\\sigma_t$ is an exponential function of $t$ (and hence $\\bar{\\sigma}_t^2:=2\\int_0^t \\sigma_s^2 ds$). Note that in this case, Assumption 2 (Point 1) may not hold.\n\nLIMITATIONS:\nBased on the developed theory, it is worthy to further explore new weighting functions, time discretization schemes and variance schedules to better tune diffusion models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the training and sampling process and then achieves the end-to-end analysis for diffusion models. For the optimization process, this work uses an over-parameterized NN and gradient descent to prove the convergence rate. For the sampling process, this work provides VESDE results and explains the great performance of SOTA VESDE. After these results, this work explains the “bell shaped” weight function used in application from the theoretical perspective.\n\nSTRENGTHS:\n1.\tThis work is the first one to analyze the optimization process with a deep NN and prove the exponential convergence into a neighborhood of minima. \n2.\tThis work makes the first step to explain the “bell shaped” weight function and the great performance of the state-of-the-art VE-based model from the theoretical perspective.\n\nWEAKNESSES:\nWeakness 1: The high-dimensional data assumption is strong. It would be better to discuss the application of high-dimensional data.\n\nWeakness 2: The technique novelty of Theorem 1 is needed to be discussed in detail. Section 3.1 claims that this work develops a new method for obtaining a lower bound of gradient. However, it does not discuss this method on the main page. It would be helpful to discuss the technique novelty in detail.\n\nWeakness 3: As shown in Section 3.1, this work mentions that the data separability assumption (corresponds to parameter $\\delta$ in [1]) can not be directly used due to the property of diffusion models. This work replaces $\\delta$ with $t_0/T$ when choosing $m$, which seems to be related to the $E_I$ term in Theorem 2. The technique challenge is unclear after replacing $\\delta$ with $t_0/T$. \n\nWeakness 4: As shown in [2], their results can be straightforwardly extended to any linear SDE, including VESDE. Hence, it would be better to discuss the technique of Theorem 2.\t\n\n[1] Allen-Zhu, Z., Li, Y., & Song, Z. (2019, May). A convergence theory for deep learning via over-parameterization. In International conference on machine learning (pp. 242-252). PMLR.\n\n[2] Benton, J., Bortoli, V. D., Doucet, A., & Deligiannidis, G. (2024). Nearly d-linear convergence bounds for diffusion models via stochastic localization.\n\nQUESTIONS:\nPlease see the Weakness part.\n\nQuestion 1. Can you relax the high-dimensional data assumption by using the low-dimensional manifold assumption [3]?\n\nQuestion 2. The guarantee in [4] is a pure $W_2$ guarantee and can not be directly compared to the $KL$ guarantee in this work. Can you discuss it in detail?\n\nComment 1: It would be better to add a conclusion and limitation part at the end of this work.\n\nTypo 1: In line 201, the reference Lemma is unclear.\n\n[3] Tang, R., & Yang, Y. (2024, April). Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics (pp. 1648-1656). PMLR.\n\nLIMITATIONS:\nThis work does not discuss the limitation and societal impace in a independent paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper provides a full error analysis (considering both training and sampling) for a class of score-based SDE diffusion models, and using the results to understand why and when certain time and variance schedules are preferred.

STRENGTHS:
- The theoretical contributions of the paper are excellent
- Overall the paper is well organized and the notations are clear

WEAKNESSES:
- The stated theorems are quite dense and are not easy to parse. The paper would benefit from having informal version of these results highlighting the key qualitative features
- Only an SDE based sampler (the exponential integrator scheme) is considered, while comparisons with other samplers (such as the Probability Flow ODE) are missing

QUESTIONS:
- In Remark 1, it was stated that the KL divergence can be explicitly computed when the data distribution is Gaussian, would you provide such result in the appendix?

Minor remarks:
- typo in line 16: variance -> various 
- missing lemma number in line 201

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13