Title: Improved Sample Complexity Bounds for Diffusion Model Training
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Learning Theory

Sample Complexity
Abstract: Diffusion models have become the most popular approach to deep generative modeling of images, largely due to their empirical performance and reliability. From a theoretical standpoint, a number of recent works [CCL+23, CCSW22, BBDD24] have studied the iteration complexity of sampling, assuming access to an accurate diffusion model. In this work, we focus on understanding the *sample complexity* of training such a model; how many samples are needed to learn an accurate diffusion model using a sufficiently expressive neural network? Prior work [BMR20] showed bounds polynomial in the dimension, desired Total Variation error, and Wasserstein error. We show an *exponential improvement* in the dependence on Wasserstein error and depth, along with improved dependencies on other relevant parameters.

Improved Sample Complexity Bounds for Diffusion Model Training

Abstract

Diffusion models have become the most popular approach to deep generative modeling of images, largely due to their empirical performance and reliability. From a theoretical standpoint, a number of recent works [CCL + 23b, CCSW22, BBDD24] have studied the iteration complexity of sampling, assuming access to an accurate diffusion model. In this work, we focus on understanding the sample complexity of training such a model; how many samples are needed to learn an accurate diffusion model using a sufficiently expressive neural network? Prior work [BMR20] showed bounds polynomial in the dimension, desired Total Variation error, and Wasserstein error. We show an exponential improvement in the dependence on Wasserstein error and depth, along with improved dependencies on other relevant parameters.

1 Introduction

Score-based diffusion models are currently the most successful methods for image generation, serving as the backbone for popular text-to-image models such as stable diffusion [RBL + 22], Midjourney, and DALL·E [RDN + 22] as well as achieving state-of-the-art performance on other audio and image generation tasks [SDWMG15, HJA20, JAD + 21, SSXE22, DN21].

The goal of score-based diffusion is to produce a generative model for a possibly complicated distribution q 0 . This involves two components: training a neural network using true samples from q 0 to learn estimates of its (smoothed) score functions, and sampling using the trained estimates. To this end, consider the following stochastic differential equation, often referred to as the forward SDE:

where B t represents Brownian motion. Here, x 0 is a sample from the original distribution q 0 over R d , while the distribution of x t can be computed to be

for σ 2 t = 1 -e -2 t . Note that this distribution approaches N (0 , I d ) , the stationary distribution of (1), exponentially fast.

Let q t be the distribution of x t , and let s t ( y ) := ∇ log q t ( y ) be the associated score function. We refer to q t as the σ t -smoothed version of q 0 . Then, starting from a sample x T ∼ q T , there is a reverse

SDE associated with the above forward SDE in (1) [And82]:

That is to say, if we begin at a sample x T ∼ q T , following the reverse SDE in (2) back to time 0 will give us a sample from the original distribution q 0 . This suggests a natural strategy to sample from q 0 : start at a large enough time T and follow the reverse SDE back to time 0 . Since x T approaches N (0 , I d ) exponentially fast in T , our samples at time 0 will be distributed close to q 0 . In particular, if T is large enough-logarithmic in m 2 ε -then samples produced by this process will be ε -close in TV to being drawn from q 0 . Here m 2 2 is the second moment of q 0 , given by m 2 2 := E x ∼ q 0 [ ∥ x ∥ 2 ] .

In practice, this continuous reverse SDE (2) is approximated by a time-discretized process. That is, the score s T -t is approximated at some fixed times 0 = t 0 ≤ t 1 ≤ · · · ≤ t k < T , and the reverse process is run using this discretization, holding the score term constant at each time t i . This algorithm is referred to as 'DDPM', as defined in [HJA20].

Chen et al. [CCL + 23b] proved that as long as we have access to sufficiently accurate estimates for each score s t at each discretized time, this reverse process produces accurate samples. Specifically, for any d -dimensional distribution q 0 supported on the Euclidean Ball of radius R , the reverse process can sample from a distribution ε -close in TV to a distribution ( γ · R )-close in 2-Wasserstein to q 0 in poly( d, 1 /ε, 1 /γ ) steps, as long as the score estimates ̂ s t used at each step are ˜ O ( ε 2 ) accurate in squared L 2 . That is, as long as

In this work, we focus on understanding the sample complexity of learning such score estimates. Specifically, we ask:

How many samples are required for a sufficiently expressive neural network to learn accurate score estimates that generate high-quality samples using the DDPM algorithm?

We consider a training process that employs the empirical minimizer of the score matching objective over a class of neural networks as the score estimate. More formally, we consider the following setting:

Setting 1.1. Let F ( D,P, Θ) be the class of functions represented by a fully connected neural network with ReLU activations and depth D , with P parameters, each bounded by Θ . Let { t k } be some time discretization of [0 , T ] . Given m i.i.d. samples x i ∼ q 0 , for each t ∈ { t k } , we take m Gaussian samples z i ∼ N (0 , σ 2 t I d ) . We take ̂ s t to be the minimizer of the score-matching objective:

We then use { ̂ s t k } as the score estimates in the DDPM algorithm.

This is the same setting as is used in practice, except that in practice (4) is optimized with SGD rather than globally. As in [CCL + 23b], we aim to output samples from a distribution that is ε -close in TV to a distribution that is γR or γm 2 -close in Wasserstein to q 0 . We thus seek to bound the number of samples m to get such a good output distribution, in terms of the parameters of Setting 1.1 and ε, γ .

Block et al. [BMR20] first studied the sample complexity of learning score estimates using the empirical minimizer of the score-matching objective (4). They showed sample complexity bounds that depend on the Rademacher complexity of F . Applied to our setting and using known bounds on Rademacher complexity of neural networks, in Setting 1.1 their result implies a sample complexity bound of ˜ O ( d 5 / 2 γ 3 ε 2 (Θ 2 P ) D √ D ) . See Appendix E for a detailed discussion.

Following the analysis of DDPM by [CCL + 23b], more recent work on the iteration complexity of sampling [CLL22, BBDD24] has given an exponential improvement on the Wasserstein accuracy γ , as well as replacing the uniform bound R by m 2 -the square root of the second moment. In

particular, [BBDD24] show that ˜ O ( d ε 2 log 2 1 γ ) iterations suffice to sample from a distribution that is γ · m 2 close to q 0 in 2 -Wasserstein, as long as the score estimates are ˜ O ( ε 2 /σ 2 t ) accurate, i.e.,

Inspired by these works, we ask: is it possible to achieve a similar exponential improvement in the sample complexity of learning score estimates using a neural network?

1.1 Our Results

We give a new sample complexity bound of ˜ O ( d 2 ε 3 PD log Θ log 3 1 γ ) for learning scores to sufficient accuracy for sampling via DDPM. Learning is done by optimizing the same score matching objective that is used in practice. Compared to [BMR20], our bound has exponentially better dependence on Θ , γ , and D , and a better polynomial dependence on d and P , at the cost of a worse polynomial dependence in ε .

As discussed above, for the sampling process, it suffices for the score estimate s t at time t to have error ˜ O ( ε 2 /σ 2 t ) . This means that scores at larger times need higher accuracy, but they are also intuitively easier to estimate because the corresponding distribution is smoother. Our observation is that the two effects cancel out: we show that the sample complexity to achieve this accuracy for a fixed t is independent of σ t . However, this only holds once we weaken the accuracy guarantee slightly (from L 2 error to ' L 2 error over a 1 -δ fraction of the mass'). We show that this weaker guarantee is nevertheless sufficient to enable accurate sampling. Our approach lets us run the SDE to a very small final σ t = γ , which yields a final γ dependence of O (log 3 1 γ ) via a union bound over the times t that we need score estimates s t for. In contrast, the approach in [BMR20] gets the stronger L 2 -accuracy, but requires a poly( 1 σ t ) dependence in sample complexity for each score s t ; this leads to their poly( 1 γ ) sample complexity overall.

To state our results formally, we make the following assumptions on the data distribution and the training process:

A1 The second moment m 2 2 of q 0 is between 1 / poly( d ) and poly( d ) .

A2 For the score s t used at each step, there exists some function f ∈ F ( D,P, Θ) (as defined in Setting 1.1) such that the L 2 error, E x ∼ q t [ ∥ f ( x ) -s t ( x ) ∥ 2 ] , is sufficiently small.

That is: the data is somewhat normalized, and the smoothed scores can be represented well in the function class. Our main theorem is as follows:

Theorem 1.2. In Setting 1.1, suppose assumptions A1 and A2 hold. For any γ > 0 , consider the score functions trained from

i.i.d. samples of q 0 . With 99% probability, DDPM using these score functions can sample from a distribution ε -close in TV to a distribution γm 2 -close to q in 2-Wasserstein.

We remark that assumption A1 is made for a simpler presentation of our theorem; the (logarithmic) dependence on the second moment is analyzed explicitly in Theorem C.2. The quantitative bound for A2-exactly how small the L 2 error needs to be-is given in detail in Theorem C.3.

Barrier for L 2 accuracy. As mentioned before, previous works have been using L 2 accurate score estimation either as the assumption for sampling or the goal for training. Ideally, one would like to simply show that the ERM of the score matching objective will have bounded L 2 error of ε 2 /σ 2 with a number of samples that scales polylogarithmically in 1 σ . Unfortunately, this is false . In fact, it is information-theoretically impossible to achieve this in general without poly( 1 σ ) samples. Since sampling to γm 2 Wasserstein error needs to consider a final σ t = γ , this leads to a poly( 1 γ ) dependence. See Figure 1, or the discussion in Section 4, for a hard instance.

-R

0

R

η

√

2

π

1

-

η

√

2

π

p

1

p

2

R

0

R

s

1

s

2

10

1

10

2

10

3

0

.

0

0

.

2

0

.

4

0

.

6

Failure probability

L

2

metric

D δ

p

metric

-

m

In the example in Figure 1, score matching + DDPM still works to sample from the distribution with sample complexity scaling with poly(log 1 γ ) ; the problem lies in the theoretical justification for it. Given that it is impossible to learn the score in L 2 to sufficient accuracy with fewer than poly( 1 γ ) samples, such a justification needs a different measure of estimation error. We will introduce such a measure, showing (1) that it will be small for all relevant times t after a number of samples that scales polylogarithmically in 1 γ , and (2) that this measure suffices for fast sampling via the reverse SDE.

The problem with measuring error in L 2 comes from outliers: rare, large errors can increase the L 2 error while not being observed on the training set. We proved that we can relax the L 2 accuracy requirement for diffusion model to the (1 -δ ) -quantile error: For distribution p , and functions f, g , we say that

We adapt [BBDD24] to show that learning the score in our new outlier-robust sense also suffices for sampling, if we have accurate score estimates at each relevant discretization time.

Lemma 1.3. Let q be a distribution over R d with second moment m 2 2 between 1 / poly ( d ) and poly ( d ) . For any γ > 0 , there exist N = ˜ O ( d ε 2 + δ 2 log 2 1 γ ) discretization times 0 = t 0 < t 1 < · · · < t N < T such that if the following holds for every k ∈ { 0 , . . . , N -1 } :

then DDPM can sample from a distribution that is within ˜ O ( δ + ε √ log ( d/γ )) in TV distance to q γ in N steps.

With this relaxed requirement, our main technical lemma shows that for each fixed time, a good (1 -δ ) -quantile accuracy can be achieved with a small number of samples, independent of σ t .

Lemma 1.4 (Main Lemma) . In Setting 1.1, suppose assumptions A1 and A2 hold. By taking m i.i.d. samples from q 0 to train score function ̂ s t , when

with probability 1 -δ train, the score estimate ̂ s t satisfies

Combining Lemma 1.4 with Lemma 1.3 gives us Theorem 1.2. The proof is detailed in Appendix C.

2 Related Work

Score-based diffusion models were first introduced in [SDWMG15] as a way to tractably sample from complex distributions using deep learning. Since then, many empirically validated techniques

have been developed to improve the sample quality and performance of diffusion models [HJA20, ND21, SE20, SSDK + 21, SDME21]. More recently, diffusion models have found several exciting applications, including medical imaging and compressed sensing [JAD + 21, SSXE22], and text-toimage models like DALL · E 2 [RDN + 22] and Stable Diffusion [RBL + 22].

Recently, a number of works have begun to develop a theoretical understanding of diffusion. Different aspects have been studied - the sample complexity of training with the score-matching objective [BMR20], the number of steps needed to sample given accurate scores [CCL + 23b, CLL22, CCSW22, CCL + 23a, BBDD24, BTHD21, LLT23], and the relationship to more traditional methods such as maximum likelihood [PRS + 23, KHR23].

On the training side, [BMR20] showed that for distributions bounded by R , the score-matching objective learns the score of q γ in L 2 using a number of samples that scales polynomially in 1 γ . On the other hand, for sampling using the reverse SDE in (2), [CLL22, BBDD24] showed that the number of steps to sample from q γ scales polylogarithmically in 1 γ given L 2 approximations to the scores.

Our main contribution is to show that while learning the score in L 2 requires a number of samples that scales polynomially in 1 γ , the score-matching objective does learn the score in a weaker sense with sample complexity depending only polylogarithmically in 1 γ . Moreover, this weaker guarantee is sufficient to maintain the polylogarithmic dependence on 1 γ on the number of steps to sample with γ · m 2 2-Wasserstein error.

Our work, as well as [BMR20], assumes the score can be accurately represented by a small function class such as neural networks. Another line of work examines what is possible for more general distributions [CHZW23, OAS23]. For example, [CHZW23] shows that for general subgaussian distributions, the scores can be learned to L 2 error ε/σ t with ( d/ε ) O ( d ) samples. Our approach avoids this exponential dependence, but assumes that neural networks can represent the score.

3 Proof Overview

In this section, we outline the proofs for two key lemmas: Lemma 1.4 in Section 3.1 and Lemma 1.3 in Section 3.2. The complete proofs for these lemmas are provided in Appendix A and Appendix B respectively.

3.1 Training

We show that the score-matching objective (4) concentrates well enough that the ERM is close to the true minimizer. Prior work on sampling [BBDD24] shows that estimating the σ -smoothed score to L 2 2 error of ε 2 σ 2 suffices for sampling; our goal is to get something close to this with a sample complexity independent of σ .

Background: Minimizing the true expectation gives the true score. In this section, we show that if we could compute the true expectation of the score matching objective, instead of just the empirical expectation, then the true score would be its minimizer. For a fixed t , let σ = σ t and p be the distribution of e -t x for x ∼ q 0 . We can think of a joint distribution of ( y, x, z ) where y ∼ p and z ∼ N (0 , σ 2 I d ) are independent, and x = y + z is drawn according to q t . With this change of variables, the score matching objective used in (4) is

Because x = y + z for Gaussian z , Tweedie's formula states that the true score s ∗ = s t is given by

Define ∆ = s ∗ ( x ) --z σ 2 , so E [∆ | x ] = 0 . Therefore for any x ,

The third term does not depend on s , and so does not affect the minimizer of this loss function. Also, for every x , the second term is zero on average over ( z | x ) , so we have

This shows that the score matching objective is indeed minimized by the true score. Moreover, an ε -approximate optimizer of l ( s ) will be close in L 2 , as needed by prior samplers.

Understanding the ERM. The algorithm chooses the score function s minimizing the empirical loss,

Again, the ˆ E [ ∥ ∆ ∥ 2 ] term is independent of s , so it has no effect on the minimizer and we can drop it from the loss function. We thus define

that satisfies l ′ ( s ∗ , x, z ) = 0 and E [ l ′ ( s, x, z )] = E [ ∥ s ( x ) -s ∗ ( x ) ∥ 2 ] . Our goal is now to to show that

for all candidate score functions s that are 'far' from s ∗ . This would ensure that the empirical minimizer of the score matching objective is not 'far' from s ∗ . To do this, we show that (10) is true with high probability for each individual s , then take a union bound over a net.

Boundedness of ∆ . Now, z ∼ N (0 , σ 2 I d ) is technically unbounded, but is exponentially close to being bounded: ∥ z ∥ ≲ σ √ d with overwhelming probability. So for the purpose of this proof overview, imagine that z were drawn from a distribution of bounded norm, i.e., ∥ z ∥ ≤ Bσ always; the full proof (given in Appendix A) needs some exponentially small error terms to handle the tiny mass the Gaussian places outside this ball. Then, since ∆ = z σ 2 -E z | x [ z σ 2 ] , we get ∥ ∆ ∥ ≤ 2 B/σ .

Warmup: poly( R/σ ) . As a warmup, consider the setting of prior work [BMR20]: (1) ∥ x ∥ ≤ R always, so ∥ s ∗ ( x ) ∥ ≲ R σ 2 ; and (2) we only optimize over candidate score functions s with value clipped to within O ( R σ 2 ) , so ∥ s ( x ) -s ∗ ( x ) ∥ ≲ R σ 2 . With both these restrictions, then, | l ′ ( s, x, z ) | ≤ R 2 σ 4 + RB σ 3 . Wecan then apply a Chernoff bound to show concentration of l ′ : for poly( ε, R σ , B, log 1 δ train ) samples, with 1 -δ train probability we have

which is greater than zero if E [ ∥ s ( x ) -s ∗ ( x ) ∥ 2 ] > ε 2 σ 2 . Thus the ERM would reject each score function that is far in L 2 . However, as we show in Section 4, restrictions (1) and (2) are both necessary: the score matching ERM needs a polynomial dependence on both the distribution norm and the candidate score function values to learn in L 2 .

The main technical contribution of our paper is to avoid this polynomial dependence on 1 /σ . To do so, we settle for rejecting score functions s that are far in our stronger distance measure D δ score p , i.e., for which

Approach. We want to show (10), which is a concentration over x and z . Now, x is somewhat hard to control, because it depends on the unknown distribution, but z ∼ N (0 , σ 2 I d ) is very well behaved. This motivates breaking up the expectation over x, z into an expectation over x and z | x . Following this approach, we could try to show that

However, this is not possible to show; the problem is that this could be unbounded, since s ( x ) is an arbitrary neural network that could have extreme outliers. So, we instead show this is true with high probability if we clip the internal value, making it

Note that A x is a function of the empirical samples x . If s is a score that we want to reject under (11), then we know that A x is an empirical average of values that are at least ε 2 σ 2 with probability δ score . It therefore holds that, for m>O ( log 1 δ train δ score ) , we will with 1 -δ train probability over the samples x have

Concentration about the intermediate notion. Finally, we show that for every set of samples x i satisfying (12), we will have

as needed. For each sample x , we split our analysis of ˆ E z | x [ l ( s, x, z )] into two cases:

Case 1: ∥ s ( x ) -s ∗ ( x ) ∥ > O ( B σ ) . In this case, by Cauchy-Schwarz and the assumption that ∥ ∆ ∥ ≤ 2 B/σ ,

so these x will contribute the maximum possible value to A x , regardless of z (in its bounded range).

Case 2: ∥ s ( x ) -s ∗ ( x ) ∥ < O ( B σ ) . In this case, | l ′ ( s, x, z ) | ≲ B 2 /σ 2 and

so for these x , as a distribution over z , l ′ is bounded with bounded variance.

In either case, the contribution to A x is bounded with bounded variance; this lets us apply Bernstein's inequality to show, if m>O ( B 2 log 1 δ train σ 2 A ) ≈ O ( B 2 log 1 δ train ε 2 δ score ) , for every x we will have

This then implies with 1 -δ train probability.

Conclusion. Suppose m>O ( B 2 log 1 δ train ε 2 δ score ) . Then with 1 -δ train probability we will have (12); and conditioned on this, with 1 -δ train probability we will have ˆ E x,z [ l ′ ( s, x, z )] > 0 . Hence this m suffices to distinguish any candidate score s that is far from s ∗ . For finite hypothesis classes we can take the union bound, incurring a log |H| loss (this is given as Theorem A.2). Lemma 1.4 follows from applying this to a net over neural networks, which has size log H ≈ PD log Θ .

3.2 Sampling

Now we overview the proof of Lemma 1.3, i.e., why having an ε/σ T -t k accuracy in the D δ q sense is sufficient for accurate sampling.

To practically implement the reverse SDE in (2), we discretize this process into N steps and choose a sequence of times 0 = t 0 < t 1 < · · · < t N < T . At each discretization time t k , we use our score estimates ̂ s t k and proceed with an approximate reverse SDE using our score estimates, given by the following. For t ∈ [ t k , t k +1 ] ,

This is almost the reverse SDE that would give exactly correct samples, with two sources of error: it starts at N (0 , I d ) rather than q T , and it uses ̂ s T -t k rather than the s T -t . The first error is negligible, since q T is e -T -close to N (0 , I d ) . But how much error does the switch from s to ̂ s introduce?

Let Q be the law of the reverse SDE using s , and let ̂ Q be the law of (13). Then Girsanov's theorem states that the distance between Q and ̂ Q is defined by the L 2 error of the score approximations:

The first line is an equality . The second line comes from approximating x T -t by x T -t k , which for small time steps is quite accurate. So in previous work, good L 2 approximations to the score mean (14) is small, and hence KL ( Q ∥ ̂ Q ) is small. In our setting, where we cannot guarantee a good L 2 approximation, Girsanov actually implies that we cannot guarantee that KL ( Q ∥ ̂ Q ) is small.

However, since we finally hope to show closeness in TV , we can circumvent the above as follows. We define an event E to be the event that the score is bounded well at all time steps x T -t k , i.e.,

If we have a D δ/N q t k accuracy for each score, we have 1 -P [ E ] ≤ δ . Therefore, if we look at the TV error between Q and ̂ Q , instead of bounding KL ( Q ∥ ̂ Q ) . The TV between Q and ̂ Q can be then bounded by

The second term TV (( Q | E ) , ( ̂ Q | E )) is bounded because after conditioning on E , the score error is always bounded, so now we can use (14) to bound the KL divergence between Q and ̂ Q , then use Pinsker's inequality to translate KL into TV distance.

4 Hardness of Learning in L 2

In this section, we give concrete examples where it is difficult to learn the score in L 2 , even though learning to sufficient accuracy for sampling is possible. Previous works, such as [BBDD24], require the L 2 error of the score estimate s t to be bounded by ε/σ t . We demonstrate that achieving this guarantee is prohibitively expensive: sampling from a σ t -smoothed distribution requires at least poly(1 /σ t ) samples. Thus, sampling from a distribution γ -close in 2-Wasserstein to q 0 requires polynomially many samples in 1 γ .

To show this, we demonstrate two lower bound instances. Both of these instances provide a pair of distributions that are hard to distinguish in L 2 , and emphasize different aspects of this hardness:

The first instance shows that even with a polynomially bounded set of distributions, it is information theoretically impossible to learn a score with small L 2 error with high probability, with fewer than poly(1 /γ ) samples.

0

R

η

√

2

π

1

√

2

π

True Dist.

Alternate Dist.

0

10

√

log

m

R

True Score

Alternate Score

In the second instance, we show that even with a simple true distribution, such as a single Gaussian, distinguishing the score matching loss of the true score function from one with high L 2 error can be challenging with fewer than poly(1 /γ ) samples if the hypothesis class is large, such as with neural networks.

Information theoretically indistinguishable distributions: For our first example, consider the two distributions (1 -η ) N (0 , σ 2 )+ η N ( ± R,σ 2 ) , where R is polynomially large. Even though these distributions are polynomially bounded, it is impossible to distinguish these in L 2 given significantly fewer than 1 η samples. However, the L 2 error in score incurred from picking the score of the wrong distribution is large. In Figure 1, the rightmost plot shows a simulation of this example, and demonstrates that the L 2 error remains large even after many samples are taken. Formally, we have:

Lemma 4.1. Let R be sufficiently larger than σ . Let p 1 be the distribution (1 -η ) N (0 , σ 2 ) + η N ( -R,σ 2 ) with corresponding score function s 1 , and let p 2 be (1 -η ) N (0 , σ 2 ) + η N ( R,σ 2 ) with score s 2 , such that η = ε 2 σ 2 R 2 . Then, given m < R 2 ε 2 σ 2 samples from either distribution, it is impossible to distinguish between p 1 and p 2 with probability larger than 1 / 2 + o m (1) . But,

Simple true distribution: Now, consider the true distribution being N (0 , σ 2 ) , and, for large S , let ̂ s be the score of the mixture distribution η N (0 , σ 2 ) + (1 -η ) N ( S, σ 2 ) , as in Figure 2. This score will have practically the same score matching objective as the true score for the given samples with high probability, as shown in Figure 2, since all m samples will occur in the region where the two scores are nearly identical. However, the squared L 2 error incurred from picking the wrong score function ̂ s is large We formally show this result in the following lemma:

N -poly ( m )

Lemma 4.2. Let S be sufficiently large. Consider the distribution ̂ p = η N (0 , σ 2 ) +(1 -η ) N ( S, σ 2 ) for η = Se -S 2 2 +10 √ log m · S 10 √ log m , and let ̂ s be its score function. Given m samples from the standard Gaussian p ∗ = (0 , σ 2 ) with score function s ∗ , with probability at least 1 1 ,

Together, these examples show that even with reasonably bounded or well-behaved distributions, it is difficult to learn the score in L 2 with fewer than poly( R/γ ) samples, motivating our (1 -δ ) -quantile error measure.

5 Conclusion and Future Work

In this work, we have addressed the sample complexity of training the scores in diffusion models. We showed that a neural networks, when trained using the standard score matching objective, can

be used for DDPM sampling after ˜ O ( d 2 PD ε 3 log Θ log 3 1 γ ) training samples. This is an exponentially better dependence on the neural network depth D and Wasserstein error γ than given by prior work. To achieve this, we introduced a more robust measure, the 1 -δ quantile error, which allows for efficient training with poly(log 1 γ ) samples using score matching. By using this measure, we showed that standard training (by score matching) and sampling (by the reverse SDE) algorithms achieve our new bound.

One caveat is that our results, as well as those of the prior work [BMR20] focus on understanding the statistical performance of the score matching objective: we show that the empirical minimizer of the score matching objective over the class of ReLU networks approximates the score accurately. We do not analyze the performance of Stochastic Gradient Descent (SGD), commonly used to approximate this empirical minimizer in practice. Understanding why SGD over the class of neural networks performs well is perhaps the biggest problem in theoretical machine learning, and we do not address it here.

Acknowledgments

We thank Syamantak Kumar and Dheeraj Nagaraj for pointing out an error in a previous version of this work, and for providing a fix (Lemma F.7). This project was funded by NSF award CCF-1751040 (CAREER) and the NSF AI Institute for Foundations of Machine Learning (IFML).

6 References

[And82] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

[BBDD24] Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly $d$-linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations , 2024.

[BMR20] Adam Block, Youssef Mroueh, and Alexander Rakhlin. Generative modeling with denoising auto-encoders and langevin sampling. arXiv:2002.00107 [cs, math, stat] , June 2020. arXiv: 2002.00107.

[BTHD21] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

[CCL + 23a] Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ODE is provably fast. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 68552-68575. Curran Associates, Inc., 2023.

[CCL + 23b] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations , 2023.

[CCSW22] Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. Improved analysis for a proximal algorithm for sampling. In Conference on Learning Theory , pages 2984-3014. PMLR, 2022.

[CHZW23] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

[CLL22] Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning , 2022.

[DN21] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 8780-8794. Curran Associates, Inc., 2021.

[GLP23] Shivam Gupta, Jasper C.H. Lee, and Eric Price. High-dimensional location estimation via norm concentration for subgamma vectors. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 12132-12164. PMLR, 23-29 Jul 2023.

[HJA20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

[HKZ12] Daniel Hsu, Sham Kakade, and Tong Zhang. A tail inequality for quadratic forms of subgaussian random vectors. Electronic Communications in Probability , 17:1-6, 2012.

[JAD + 21] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 14938-14954. Curran Associates, Inc., 2021.

[KHR23] Frederic Koehler, Alexander Heckett, and Andrej Risteski. Statistical efficiency of score matching: The view from isoperimetry. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

[KN] Syamantak Kumar and Dheeraj Nagaraj. Personal communication.

[LLT23] Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory , pages 946-985. PMLR, 2023.

[ND21] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. CoRR , abs/2102.09672, 2021.

[OAS23] Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distribution estimators. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

[PRS + 23] Chirag Pabbaraju, Dhruv Rohatgi, Anish Prasad Sevekari, Holden Lee, Ankur Moitra, and Andrej Risteski. Provable benefits of score matching. CoRR , abs/2306.01993, 2023.

[RBL + 22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022.

[RDN + 22] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR , abs/2204.06125, 2022.

[SDME21] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

[SDWMG15] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

[SE20] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1243812448. Curran Associates, Inc., 2020.

[Sel24] Mark Sellke. On size-independent sample complexity of relu networks. Information Processing Letters , 186:106482, 2024.

[SSDK + 21] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

[SSXE22] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. In International Conference on Learning Representations , 2022.

A Sample Complexity to Achieve (1 -δ ) -Quantile Accuracy

The goal of this section is to prove our main lemma, the sample complexity bound to learn score at a single time. More specifically, we will give a quantitative version of Lemma 1.4.

Lemma A.1 (Main Lemma, Quantitative Version) . Let q 0 be a distribution with second moment m 2 2 . Let ϕ θ ( · ) be the fully connected neural network with ReLU activations parameterized by θ , with P total parameters and depth D . Let Θ > 1 . Suppose there exists some weight vector θ ∗ with ∥ θ ∗ ∥ ∞ ≤ Θ such that

for a sufficiently large constant C . By taking m i.i.d. samples from q 0 to train score s t , when

the empirical minimizer ϕ ̂ θ of the score matching objective used to estimate s t (over ϕ θ with ∥ θ ∥ ∞ ≤ Θ ) satisfies with probability 1 -δ train .

In order to prove the lemma, we first consider the case when learning the score over a finite function class H .

A.1 Score Estimation for Finite Function Class

The main result (Lemma A.2) of this section shows that if there is a function in the function class H that approximates the score well in our D δ p sense (see (6)), then the score-matching objective can learn this function using a number of samples that is independent of the domain size or the maximum value of the score.

Notation. Fix a time t . For the purposes of this section, let q := q t be the distribution at time t , let σ := σ t be the smoothing level for time t , and let s := s t be the score function for time t . For m samples y i ∼ q 0 and z i ∼ N (0 , σ 2 ) , let x i = e -t y i + z i ∼ q t .

We use ̂ E [ f ( x, y, z )] to denote the empirical expectation 1 m ∑ m i =1 f ( x i , y i , z i ) .

We now state the score matching algorithm.

Algorithm 1 Empirical score estimation for s

Input: Distribution q 0 , y 1 , . . . , y m ∼ q 0 , set of hypothesis score function H = { ˜ s i } , smoothing level σ .

Take m independent samples z i ∼ N (0 , σ 2 I d ) , and let x i = e -t y i + z i .

For each ˜ s ∈ H , let

Let ˆ s = arg min ˜ s ∈H l (˜ s )

Return ˆ s

Lemma A.2 (Score Estimation for Finite Function Class) . For any distribution q 0 and time t > 0 , consider the σ t -smoothed version q t with associated score s t . For any finite set H of candidate score functions. If there exists some s ∗ ∈ H such that

for a sufficiently large constant C , then using m > ˜ O ( 1 ε 2 δ score ( d +log 1 δ train ) log |H| δ train ) samples, the empirical minimizer ˆ s of the score matching objective used to estimate s t satisfies with probability 1 -δ train .

Proof. Per the notation discussion above, we set s = s t and σ = σ t .

Denote

We have

Note that by Markov's inequality, with probability 1 -δ train / 3 ,

By Lemma A.3, with probability 1 -δ train / 3 ,

We will show that for all ˜ s such that D δ score q (˜ s, s ) > ε/σ , with probability 1 -δ train ,

so that the empirical minimizer ˆ s is guaranteed to have

Also, by Corollary A.5, with probability 1 -δ train / 3 , for all ˜ s ∈ H that satisfy D δ score q (˜ s, s ) > ε/σ simultaneously,

Plugging in everything into equation (16), we have, with probability 1 -δ train , for all ˜ s ∈ H with D δ score q (˜ s, s ) > ε/σ simultaneously,

as required.

Lemma A.3. For any distribution q 0 and time t > 0 , consider the σ t -smoothed version q t with associated score s t . Suppose s ∗ is such that

for a sufficiently large constant C . Then, using m> ˜ O ( 1 ε 2 δ score ( d +log 1 δ train )) samples ( x i , y i , z i ) where y i ∼ q 0 , z i ∼ N (0 , σ 2 I d ) and x i = e -t y i + z i ∼ q t , we have with probability 1 -δ train ,

Proof. Note that s ( x ) = E z | x [ -z σ 2 ] so that

Also, for any δ , with probability 1 -δ , by Lemmas F.4 and F.5,

for some constant C . Let E be the event that ∥ s ( x ) --z σ 2 ∥ 2 ≤ C d +log d ε 2 δ score δ train σ 2 . Since ∥ s ( x ) --z σ 2 ∥ 2 is subexponential with parameter σ 2 and has mean O ( d σ 2 ) by the above, by Lemma F.7, we have

Then, since P [ E ] ≥ 1 -ε 2 δ score δ train 100( d +log d ε 2 δ score δ train ) ≥ 1 2 and E [ ∥ s ∗ ( x ) -s ( x ) ∥ 2 | ¯ E ] ≤ E [ s ∗ ( x ) s ( x ) 2 ] / P [ ¯ E ] , we have

Moreover,

So, by Chebyshev's inequality, with probability 1 -δ train / 2 ,

for our choice of m . Since E holds except with probability ε 2 δ score δ train 100( d +log d ε 2 δ score δ train ) ≪ δ train, we have with probability 1 -δ train in total,

for sufficiently large constant C .

Lemma A.4. Consider any set F of functions f : R d → R d such that for all f ∈ F ,

Then, with m > ˜ O ( 1 ε 2 δ score ( d +log 1 δ train ) log |F| δ train ) samples drawn in Algorithm 1, we have with probability 1 -δ train ,

holds for all f ∈ F .

Proof. Define

We want to show that h f has

for all f ∈ F with probability 1 -δ train .

Let B = O ( √ d +log m εδ score δ train σ ) . For f ∈ F , let

be a clipped version of h f ( x, z ) . We will show that for our chosen number of samples m , the following hold with probability 1 -δ train simultaneously:

For all i , ∥ ∥ -z i σ 2 ∥ ∥ ≤ B .

For all i , ∥ ∥ E [ -z σ 2 | x i ] ∥ ∥ ≤ B

To show that these together imply (17), note that whenever g f ( x i , z i ) = h f ( x i , z i ) , ∥ f ( x i ) ∥ ≥ 10 B . So, since ∥ -z i σ 2 ∥ ≤ B and ∥ E [ -z σ 2 | x i ] ∥ ≤ B ,

̸

So under conditions 1 , 2 , 3 , for all f ∈ F ,

So it just remains to show that conditions 1 , 2 , 3 hold with probability 1 -δ train simultaneously.

For all i, ∥ ∥ -z i σ 2 ∥ ∥ ≤ B . Holds with probability 1 -δ train / 3 by Lemma F.5 and the union bound.

For all i, ∥ ∥ E [ -z σ 2 | x i ]∥ ∥ ≤ B . Holds with probability 1 -δ train / 3 by Lemma F.6 and the union bound.

ˆ E [ g f ( x, z )] ≥ δ score ε 2 16 σ 2 for all f ∈ F .

Let E be the event that 1 . and 2 . hold. Let a i = min( ∥ f ( x i ) ∥ , 10 B ) . We proceed in multiple steps.

Conditioned on E , | g f ( x i , z i ) | ≲ B 2 . If ∥ f ( x i ) ∥ ≥ 10 B , | g f ( x i , z i ) | = B 2 by definition. On the other hand, when ∥ f ( x i ) ∥ < 10 B , since we condition on E ,

E [ g f ( x i , z i ) | E,a i ] ≳ a 2 i -O ( δ train B 2 ) . First, note that by definition of g f ( x, z ) , for a i = 10 B ,

Now, for a i < 10 B ,

Now let g clip f ( x i , z i ) be a clipped version of g f ( x i , z i ) , clipped to ± CB 2 for sufficiently large constant C . We have, by above,

But,

Now, note that

So, for a < 10 B

Var ( g f ( x i , z i ) | a i , E ) ≲ a 2 i B 2 . For a i = 10 B , we have, by definition of g f ( x, z ) ,

On the other hand, for a i < 10 B ,

With probability 1 -δ train / 3 , for all f ∈ F , ̂ E [ g f ( x i , z i )] ≳ Ω ( ε 2 δ score σ 2 ) Using the above, by Bernstein's inequality, with probability 1 -δ train / 6 ,

Now, note that since P x ∼ p σ [ ∥ f ( x ) ∥ > ε/σ ] ≥ δ score, we have with probability 1 -δ train / 6 , for n > O ( log 1 δ train δ score )

So, for n > O ( B 2 · σ 2 log 1 δ train ε 2 δ score ) , we have, with probability 1 -δ train / 3 ,

Rescaling so that δ train ≤ O ( ε 2 δ score σ 2 · B 2 ) , for n > O ( B 2 · σ 2 log B 2 · σ 2 ε 2 δ score δ train ε 2 · δ score ) , we have, with probability 1 -δ train / 3 ,

Combining with 1. and 2. gives the claim for a single f ∈ F . Union bounding over the size of F gives the claim.

Corollary A.5. Let H bad be a set of score functions such that for all ˜ s ∈ H bad ,

Then, for m > ˜ O ( 1 ε 2 δ score ( d +log 1 δ train ) log |H bad | δ train ) samples drawn by Algorithm 1, we have with probability 1 -δ train ,

for all ˜ s ∈ H bad .

Proof. We have, for f ( x ) := ˜ s ( x ) -s ( x ) ,

since s ( x ) = E [ -z σ 2 | x ] by Lemma F.1. Then, by definition, for s ∈ H bad, for the associated f , P [ ∥ f ( x ) ∥ > ε/σ ] > δ score. So, by Lemma A.4, the claim follows.

A.2 Score Training for Neural Networks

Now we are ready to apply the finite function class to neural networks by a net argument. In particular, we first prove a version that uses a Frobenious norm to bound the weight vector.

Lemma A.6. Let q 0 be a distribution with second moment m 2 2 . Let ϕ θ ( · ) be the fully connected neural network with ReLU activations parameterized by θ , with P total parameters and depth D . Let Θ > 1 . Suppose there exists some weight vector θ ∗ with ∥ θ ∗ ∥ F ≤ Θ such that

for a sufficiently large constant C . By taking m i.i.d. samples from q 0 to train score s t , when

the empirical minimizer ϕ ̂ θ of the score matching objective used to estimate s t (over ϕ θ with ∥ θ ∥ F ≤ Θ ) satisfies with probability 1 -δ train .

Proof. Per the notation discussion above, we set s = s t and σ = σ t . Note that σ < 1 since σ 2 t = 1 -e -2 t .

For any function f denote

we will show that for every ˜ θ with ∥ ˜ θ ∥ F ≤ Θ such that D δ score q ( ϕ ˜ θ , s ) > ε/σ , with probability 1 -δ train , so that the empirical minimizer ϕ ̂ θ is guaranteed to have

First, note that since the ReLU activation is contractive, the total Lipschitzness of ϕ θ is at most the product of the spectral norm of the weight matrices at each layer. For any θ , consider ˜ θ such that

Let M 1 , . . . , M D be the weight matrices at each layer of the neural net ϕ θ , and let ˜ M 1 , . . . , ˜ M D be the corresponding matrices of ϕ ˜ θ .

We now show that ∥ ∥ ϕ ˜ θ ( x ) -ϕ θ ( x ) ∥ ∥ is small, using a hybrid argument. Define y i to be the output of a neural network with weight matrices M 1 , . . . , M i , ˜ M i +1 , . . . , ˜ M D on input x , so y 0 = ϕ ˜ θ ( x ) and y D = ϕ θ ( x ) . Then we have

and so

Note that the dimensionality of θ is P . So, we can construct a τ σD Θ D -1 -net N over the set { θ : ∥ θ ∥ F ≤ Θ } of size O ( σD Θ D -1 τ ) P , so that for any θ with ∥ θ ∥ F ≤ Θ , there exists ˜ θ ∈ N with

Let H = { ϕ ˜ θ : ˜ θ ∈ N } . Then, we have that for every θ with ∥ θ ∥ F ≤ Θ , there exists h ∈ H such that

Now, choose any ˜ θ with ∥ ˜ θ ∥ F ≤ Θ and D δ score q ( ϕ ˜ θ , s ) > ε/σ , and let ˜ h ∈ H satisfy the above for ˜ θ . We will set

for small enough constant C ′ . So, |H| = O ( σD Θ D -1 τ ) P < O ( D Θ 2 D -1 ( m · m 2 2 +( d +log δ m train ) ε 2 δ score ) P , since σ < 1 .

So, our final choice of m satisfies m> ˜ O ( 1 ε 2 δ score ( d +log 1 δ train ) log |H| δ train ) . We have

Now, by Corollary A.5, for our choice of m , with probability 1 -δ train / 4 for every h ∈ H with D δ score / 2 q ( h, s ) > ε/ (2 σ ) simultaneously,

By Markov's inequality, with probability 1 -δ train / 4 ,

By Lemma A.3, with probability 1 -δ train / 4 ,

Plugging into (20) we have that with probability 1 -3 δ train / 4 , for every ˜ h satisfying D δ score / 2 q ( ˜ h, s ) > ε/ (2 σ ) ,

Now, we will show that D δ score / 2 q ( ˜ h, s ) > ε/ (2 σ ) , as well as bound the last term above, for every ˜ θ satisfying D δ score q ( ϕ ˜ θ , s ) > ε/σ and ˜ h ∈ H the rounding of ϕ ˜ θ .

By the fact that q 0 has second moment m 2 2 , we have that with probability 1 -δ over x ,

Now since D δ score q ( ϕ ˜ θ , s ) > ε/σ , and ∥ ˜ h ( x ) -ϕ ˜ θ ( x ) ∥ ≤ ( τ/σ ) · ∥ x ∥ , we have, with probability at least δ score / 2 ,

for our choice of τ as in (19). So, we have shown that D δ score / 2 q ( ˜ h, s ) > ε/ (2 σ ) .

Finally, we bound the last term in (21) above. We have by (18) and a union bound, with probability 1 δ / 8 ,

-train

Similarly, with probability 1 -δ train / 8 ,

So, putting the above together, with probability 1 -δ train / 4 ,

So, with probability 1 -δ train / 4 , for some small enough constant C ′ ,

So finally, combining with (21), we have with probability 1 -δ train

So, we have shown that with probability 1 -δ train, for every ˜ θ with ∥ ˜ θ ∥ F ≤ Θ and D δ score q ( ϕ ˜ θ , s ) > ε/σ ,

so that the empirical minimizer ϕ ̂ θ is guaranteed to have

Then we have our main lemma as a direct corollary:

Proof of Lemma A.1. For every ˜ θ with ∥ ˜ θ ∥ ∞ ≤ Θ , we have

Then, by applying Lemma A.6 directly, we prove the lemma.

B Sampling with our score estimation guarantee

In this section, we show that diffusion models can converge to the true distribution without necessarily adhering to an L 2 bound on the score estimation error. A high probability accuracy of the score is sufficient.

In order to simulate the reverse process of (1) in an actual algorithm, the time was discretized into N steps. The k -th step ends at time t k , satisfying 0 ≤ t 0 < t 1 < · · · < t N = T -γ . The algorithm stops at t N and outputs the final state x T -t N .

To analyze the reverse process run under different levels of idealness, we consider these four specific path measures over the path space C ([0 , T -γ ]; R d ) :

Let Q be the measure for the process that

Let Q dis be the measure for the process that for t ∈ [ t k , t k +1 ] ,

Let Q be the measure for the process that for t ∈ [ t k , t k +1 ] ,

Let ̂ Q be the measure for the process that for t ∈ [ t k , t k +1 ] ,

To summarize, Q represents the perfect reverse process of (1), Q dis is the discretized version of Q , Q runs Q dis with an estimated score, and ̂ Q starts Q at N (0 , I d ) -effectively the actual implementable reverse process.

Recent works have shown that under the assumption that the estimated score function is close to the real score function in L 2 , then the output of ̂ Q will approximate the true distribution closely. Our next theorem shows that this assumption is in fact not required, and it shows that our score assumption can be easily integrated in a black-box way to achieve similar results.

Lemma B.1 (Score Estimation guarantee) . Consider an arbitrary sequence of discretization times 0 = t 0 < t 1 < · · · < t N = T -γ , and let σ t := √ 1 -e -2 t . Assume that for each k ∈ { 0 , . . . , N -1 } , the following holds:

Then, we have

Proof. Since random variable x T -t k follows distribution q T -t k under Q , for each k ∈ { 0 , . . . , N -1 } , we have

Using a union bound over all N different σ values, it follows that with probability at least 1 -δ over Q , the inequality

is satisfied for every k ∈ { 0 , . . . , N -1 } . Under this condition, we have

Hence, we find that

Lemma B.2 (Score estimation error to TV) . Let q be an arbitrary distribution. If the score estimation satisfies that

then the output distribution p T -t N of ̂ Q satisfies

Proof. We will start by bounding the TV distance between Q dis and Q . We will proceed by defining ˜ Q and arguing that both TV ( Q dis , ˜ Q ) and TV ( ˜ Q,Q ) are small. By the triangle inequality, this will imply that Q and Q are close in TV distance.

Defining ˜ Q . For k ∈ { 0 , . . . , N -1 } , consider event

which represents that the accumulated score estimation error along the path is at most ε 2 for a discretized diffusion process.

Given E k , we define a version of Q dis that is forced to have a bounded score estimation error. Let ˜ Q over C ((0 , T ] , R d ) be the law of a modified reverse process initialized at x T ∼ q T , and for each t ∈ [ t k , t k +1 ) , where

This SDE guarantees that once the accumulated score error exceeds ε 2 score ( E k fails to hold), we switch from the true score to the estimated score. Therefore, we have that the following inequality always holds:

Q dis and ˜ Q are close. By (22), we have

Note that when a path ( x T -t ) t ∈ [0 ,t N ] satisfies E 0 ∧··· ∧ E N -1 , its probability under ˜ Q is at least its probability under Q dis. Therefore, we have

˜ Q and Q are close. Inspired by [CCL + 23b], we utilize Girsanov's theorem (see Theorem F.8) to help bound this distance. Define

where k is index such that r ∈ [ t k , t k +1 ) . We apply the Girsanov's theorem to ( ˜ Q, ( b r )) . By Eq. (24), we have

This satisfies Novikov's condition and tells us that for

under measure ˜ Q ′ := E ( L ) t N ˜ Q , there exists a Brownian motion ( ˜ B t ) t ∈ [0 ,t N ] such that

and thus for t ∈ [ t k , t k +1 ) ,

Plug this into (23) and we have that for t ∈ [ t k , t k +1 )

This equation depicts the distribution of x , and this exactly matches the definition of Q . Therefore, Q = ˜ Q ′ = E ( L ) t N ˜ Q , and we have

Then by using (24), we have

Therefore, we can apply Pinsker's inequality and get

Putting things together. Using the data processing inequality, we have

Combining these results, we have

Since q γ is the distribution for x T -t N under Q and p T -t N is the distribution for x T -t N under ̂ Q , we have

Lemma B.3. Consider an arbitrary sequence of discretization times 0 = t 0 < t 1 < · · · < t N = T -γ . Assume that for each k ∈ { 0 , . . . , N -1 } , the following holds:

Then, the output distribution ̂ q T -t N satisfies

Proof. Follows by Lemma B.1 and Lemma B.2.

The next two lemmas from existing works show that the discretization error, TV ( Q,Q dis ) , is relatively small. Furthermore, as T increases, q T converges exponentially towards N (0 , I d ) .

Lemma B.4 (Discretization Error, Corollary 1 and eq. (17) in [BBDD24]) . For any T ≥ 1 , γ < 1 and N ≥ log(1 /γ ) , there exists a sequence of N discretization times such that

Lemma B.5 (TV between true Gaussian and q T for large T , Proposition 4 in [BBDD24]) . Let q be a distribution with a finite second moment of m 2 2 . Then, for T ≥ 1 we have

Combining Lemma B.4 and Lemma B.5 with Lemma B.3, we have the following result:

Corollary B.6. Let q be a distribution with finite second moment m 2 2 . For any T ≥ 1 , γ < 1 and N ≥ log(1 /γ ) , there exists a sequence of discretization times 0 = t 0 < t 1 < · · · < t N = T -γ such that if the following holds for each k ∈ { 0 , . . . , N -1 } :

then there exists a sequence of N discretization times such that

This implies our main theorem of this section as a corollary.

Theorem B.7. Let q be a distribution with finite second moment m 2 2 . For any γ > 0 , there exist N = ˜ O ( d ε 2 + δ 2 log 2 d + m 2 γ ) discretization times 0 = t 0 < t 1 < · · · < t N < T such that if the following holds for every k ∈ { 0 , . . . , N -1 } ,

then DDPM can produce a sample from a distribution that is within ˜ O ( δ + ε √ log (( d + m 2 ) /γ )) in TV distance of q γ in N steps.

Proof. By setting T = log( √ d + m 2 ε + δ ) and N = d ( T +log(1 /γ )) 2 ε 2 + δ 2 in Corollary B.6, we have

Furthermore, we present our theorem under the case when m 2 lies between 1 / poly( d ) and poly ( d ) to provide a clearer illustration.

Lemma 1.3. Let q be a distribution over R d with second moment m 2 2 between 1 / poly ( d ) and poly ( d ) . For any γ > 0 , there exist N = ˜ O ( d ε 2 + δ 2 log 2 1 γ ) discretization times 0 = t 0 < t 1 < · · · < t N < T such that if the following holds for every k ∈ { 0 , . . . , N -1 } :

then DDPM can sample from a distribution that is within ˜ O ( δ + ε √ log ( d/γ )) in TV distance to q γ in N steps.

C Sample Complexity of Training Diffusion Model

In this section, we present our main theorem that combines our score estimation result in the new (6) sense with prior sampling results (from [BBDD24]) to show that the score can be learned using a number of samples scaling polylogarithmically in 1 γ , where γ is the desired sampling accuracy.

Lemma C.1. Let q 0 be a distribution with second moment m 2 2 . Let ϕ θ ( · ) be the fully connected neural network with ReLU activations parameterized by θ , with P total parameters and depth D . Let Θ > 1 . For discretization times 0 = t 0 < t 1 < · · · < t N , let s T -t k be the true score function of q T -t k . If for each t k , there exists some weight vector θ ∗ with ∥ θ ∗ ∥ ∞ ≤ Θ such that

Then, if we take m > ˜ O ( N ( d +log 1 δ train ) · PD ε 2 δ score · log ( max( m 2 , 1) · Θ δ train ) ) samples, then with probability

1 -δ train, each score ˆ s T -t k learned by score matching satisfies

Proof. Note that for each t k , q T -t k is a σ T -t k -smoothed distribution. Therefore, we can use Lemma A.1 by taking δ train /N into δ train and taking δ score /N into δ score . We have that for each t k , with probability 1 -δ train /N the following holds:

By a union bound over all the steps, we conclude the proposed statement.

Now we present our main theorem.

Theorem C.2 (Main Theorem, Full Version) . Let q be a distribution of R d with second moment m 2 2 . Let ϕ θ ( · ) be the fully connected neural network with ReLU activations parameterized by θ , with P total parameters and depth D . Let Θ > 1 . For any γ > 0 , there exist N = ˜ O ( d ε 2 + δ 2 log 2 m 2 +1 /m 2 γ ) discretization times 0 = t 0 < · · · < t N < T such that if for each t k , there exists some weight vector θ ∗ with ∥ θ ∗ ∥ ∞ ≤ Θ such that

for sufficiently large constant C , then consider the score functions trained from

i.i.d. samples of q , with 1 -δ probability, DDPM can sample from a distribution ε -close in TV to a distribution γm 2 -close in 2-Wasserstein to q in N steps.

Proof. Note that for an arbitrary t > 0 , the 2-Wasserstein distance between q and q t is bounded by O ( tm 2 + √ td ) . Therefore, by choosing t N = T -min( γ, γ 2 m 2 2 /d ) , Theorem B.7 shows that by choosing N = ˜ O ( d ε ′ 2 + δ 2 log 2 d + m 2 min( γ,γ 2 m 2 2 /d ) ) , we only need

then DDPM can produce a sample from a distribution within ˜ O ( ε ′ ) in TV distance to a distribution γm 2 -close in 2-Wasserstein to q in N steps. Note that

Therefore, we only need to take ˜ O ( d ε ′ 2 + δ 2 log 2 m 2 +1 /m 2 γ ) steps. Therefore, to achieve this, we set δ train = δ , δ score = ε ′ , and ε = ε ′ / √ log d + m 2 +1 /m 2 γ ≲ ε ′ / √ log d + m 2 min( γ,γ 2 m 2 2 /d ) in Lemma C.1. This gives us the result that with

samples, we can satisfy the score requirement given the assumption in the statement.

For cleaner statement, we present this theorem under the case when m 2 lies between 1 / poly( d ) and poly( d ) and achieving training success probability of 99% . This gives the quantitative version of Theorem 1.2.

Theorem C.3 (Main Theorem, Quantitative Version) . Let q be a distribution of R d with second moment m 2 2 . Let ϕ θ ( · ) be the fully connected neural network with ReLU activations parameterized by θ , with P total parameters and depth D . Let Θ > 1 . For any γ > 0 , there exist N = ˜ O ( d log 2 1 γ ) discretization times 0 = t 0 < · · · < t N < T such that if for each t k , there exists some weight vector θ ∗ with ∥ θ ∗ ∥ ∞ ≤ Θ such that

for a sufficiently large constant C , then consider the score functions trained from

i.i.d. samples of q . With 99% probability, DDPM using these score functions can sample from a distribution ε -close in TV to a distribution γm 2 -close in 2-Wasserstein to q in N steps.

D Hardness of Learning in L 2

In this section, we give proofs of the hardness of the examples we mention in Section 4.

Lemma 4.1. Let R be sufficiently larger than σ . Let p 1 be the distribution (1 -η ) N (0 , σ 2 ) + η N ( -R,σ 2 ) with corresponding score function s 1 , and let p 2 be (1 -η ) N (0 , σ 2 ) + η N ( R,σ 2 ) with score s 2 , such that η = ε 2 σ 2 R 2 . Then, given m < R 2 ε 2 σ 2 samples from either distribution, it is impossible to distinguish between p 1 and p 2 with probability larger than 1 / 2 + o m (1) . But,

Proof.

For x ≤ 2 √ log m ,

So,

On the other hand,

So, it is impossible to distinguish between p 1 and p 2 with fewer than O ( 1 η ) samples with probability 1 / 2 + o m (1) .

2

The score L bound follows from calculation.

Lemma 4.2. Let S be sufficiently large. Consider the distribution ̂ p = η N (0 , σ 2 ) +(1 -η ) N ( S, σ 2 ) for η = Se -S 2 2 +10 √ log m · S 10 √ log m , and let ̂ s be its score function. Given m samples from the standard Gaussian p ∗ = N (0 , σ 2 ) with score function s ∗ , with probability at least 1 -1 poly ( m ) ,

Proof. Let X 1 , . . . , X m ∼ p ∗ be the m samples from N (0 , σ 2 ) . With probability at least 1 -1 poly ( m ) , every X i ≤ 2 σ √ log m . Now, the score function of the mixture ̂ p is given by

E Discussion of [BMR20]

Here, we present a brief, self-contained discussion of the prior work [BMR20]. The main result of that work on score estimation is as follows.

Theorem E.1 (Proposition 9 of [BMR20], Restated) . Let F be a class of R d -valued functions, all of with are L 2 -Lipschitz, with values supported on the Euclidean ball with radius R , and containing uniformly good approximations of the true score s t on this ball. Given n samples from q t , if we let ̂ s t be the empirical minimizer of the score-matching loss, then with probability at least 1 -δ ,

Here, B is a bound on ∥ s t (0) ∥ , and R n ( F ) is the Rademacher complexity of F over n samples.

In the setting we consider, for the class of neural networks with weight vector θ such that ∥ θ ∥ ∞ ≤ Θ with P parameters and depth D , it was shown in [Sel24] that

Moreover, the true score is R σ t Lipschitz, so that if we further restrict ourselves to neural networks have the same Lipschitzness, L ≂ R σ t . B can be bounded by √ d σ t .

Thus, an application of the theorem to our setting gives a sample complexity bound of ˜ O ( R 6 d + R 2 d 2 ε 2 · (Θ 2 P ) D √ D log 1 δ ) for squared L 2 score error O ( ε 2 σ 2 t ) . In order to support σ t -smoothed scores, we need to set R ≳ √ d σ t , for a sample complexity of ˜ O ( d 5 / 2 R 3 σ 3 t ε 2 (Θ 2 P ) D √ D log 1 δ ) to learn an approximation to s t with squared L 2 error O ( ε 2 σ 2 ) .

So, we obtain the following corollary.

Corollary E.2. Let q be a distribution over R d supported over the Euclidean ball with radius R ≤ O ( √ d/σ t ) . Let F be the set of neural networks ϕ θ ( · ) , with weight vector θ , P parameters and depth D , with ∥ θ ∥ ∞ ≤ Θ and values supported on the ball of radius R , and that are O ( R/σ t ) -Lipschitz. Suppose F contains a network with weights θ ∗ such that ϕ θ ∗ provides uniformly good approximation of the true score s t over the ball of radius R . Given n samples from q t , if we let ̂ s t be the empirical minimizer of the score-matching loss, then with probability 1 -δ ,

To learn score approximations in L 2 for all the N relevant timesteps in order to achieve TV error ε and Wasserstein error γ · R , this implies a sample complexity bound of ˜ O ( d 5 / 2 γ 3 ε 2 (Θ 2 P ) D √ D log N δ ) = ˜ O ( d 5 / 2 γ 3 ε 2 (Θ 2 P ) D √ D log 1 δ ) for our final choice of N .

F Utility Results

Lemma F.1 (From [GLP23]) . Let f be an arbitrary distribution on R d , and let f Σ be the Σ -smoothed version of f . That is, f Σ ( x ) = E y ∼ f [ (2 π ) -d/ 2 det(Σ) -1 / 2 exp ( -1 2 ( x -Y ) T Σ -1 ( x -Y ) )] . Let s Σ be the score function of f Σ . Let ( X,Y,Z Σ ) be the joint distribution such that Y ∼ f , Z Σ ∼ N (0 , Σ) are independent, and X = Y + Z Σ ∼ f Σ . We have for ε ∈ R d , so that

t

Lemma F.2 (From [HKZ12], restated) . Let x be a mean-zero random vector in R d that is Σ -subgaussian. That is, for every vector v ,

Then, with probability 1 -δ ,

Lemma F.3. For s Σ the score function of an Σ -smoothed distribution where Σ = σ 2 I , we have that v T s Σ ( x ) is O (1 /σ 2 ) -subgaussian, when x ∼ f Σ and ∥ v ∥ = 1 .

Proof. We have by Lemma F.1 that

So,

The claim follows.

Lemma F.4. Let Σ = σ 2 I , and let x ∼ f Σ . We have that with probability 1 -δ ,

Proof. Follows from Lemmas F.3 and F.2.

Lemma F.5. For z ∼ N (0 , σ 2 I d ) , with probability 1 -δ ,

Proof. Note that ∥ z ∥ 2 is chi-square, so that we have for any 0 ≤ λ < σ 2 / 2 ,

The claim then follows by the Chernoff bound.

Lemma F.6. For x ∼ f Σ , with probability 1 -δ ,

Proof. Since Z Σ ∼ N (0 , σ 2 I d ) so that ∥ Z Σ ∥ 2 is chi-square, we have that for any 0 ≤ λ < σ 2 / 2 , by Jensen's inequality,

The claim then follows by the Chernoff bound. That is, setting λ = σ 2 / 4 , for any t > 0 ,

For t = O ( d +log 1 δ σ 2 ) , this is less than δ .

Lemma F.7 ([KN]) . Let X be a non-negative random variable such that for every t ≥ 0 , P [ X ≥ t ] ≤ exp( -λt ) for some constant λ ≥ 0 . Then, for K ≥ 0 ,

Proof. Let p := P [ X ≥ K ] and denote the distribution of X as P . Consider the sequence of iid samples { X i } i ≥ 0 sampled from P . Let T be the first index i , where X i ≥ K . Then the distribution of X T is the conditional distribution of X given X ≥ K . For any m> 0 ,

Now, 1 (( k -1) m<T ) and sup ( k -1) m<i ≤ km X i are independent since 1 (( k -1) m<T ) depends only on { X i } i ∈ [1 , ( k -1) m ] . Thus,

Choosing m = 1 p so that (1 -p ) m ≤ 1 e gives the claim.

Theorem F.8 (Girsanov's theorem) . For t ∈ [0 , T ] , let L t = ∫ t 0 b s d B s where B is a Q -Brownian motion. Assume Novikov's condition is satisfied:

is a Brownian motion under P where P := E ( L ) T Q , the probability distribution with density E ( L ) T w.r.t. Q.

Then is a Q -martingale and

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims are all reflected in the scope and contributions, as discussed in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations of our assumptions in the introduction, after we state them, as well as in the related work and conclusion sections.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The main paper contains proof overviews, and the appendices contain the full rigorous versions of all proofs of all stated lemmas and theorems.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: We do not have any experiments. We have a small simulation that is just a illustration of a lower bound, which is very small in scope.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: We do not have any experiments. We have a small simulation that is just a illustration of a lower bound, which is very small in scope.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: We do not have any experiments. We have a small simulation that is just a illustration of a lower bound, which is very small in scope.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: We do not have any experiments. We have a small simulation that is just a illustration of a lower bound, which is very small in scope.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: We do not have any experiments. We have a small simulation that is just a illustration of a lower bound, which is very small in scope.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors read through and reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Since this work is entirely theoretical, there are no societal impacts, positive or negative.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not have any data or models to release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: There are no assets that are used in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates theoretical guarantees for the performance of diffusion models. More precisely, while the previous works have mainly focused on the iteration complexity by assuming to have access to an accurate diffusion model, this paper targets the question of the sample complexity for learning an accurate diffusion model (score function). The main contribution is an exponential improvement of the existing bound with respect to Wasserstein error and depth.\n\nSTRENGTHS:\nThe main contribution is to improve the existing sample complexity for learning the score function, exponentially with respect to some on $\\Theta$, $\\gamma$, and $D$. This may contribute to a better theoretical understanding of diffusion-based models.\n\nWEAKNESSES:\n- The impact of the results obtained and the scope of the theoretical insights are limited. I am not sure that the results of this paper bring concrete new implications/understandings for diffusion models.\n\n- Some parts of the paper are not clear enough (see Questions section)\n\nQUESTIONS:\n1. Can the authors develop more why and when the dependence on $\\theta$, $\\gamma$, $D$ and $P$ is more important than the dependence on $\\epsilon$? Can a numerical estimate be provided here?\n\n\n2. Can the authors discuss assumption A2 in more detail? Is it true that the assumption implies $\\inf_{f} E_{X \\sim q_t} [||f(x) - s_t(x)||_2]=0$? Does this assumption implicitly assume that the score function is \"simple\" enough to belong to the family $\\mathcal{F}$? Does it lead to any limitation in practice?  \n\n3. What are $\\gamma$ and $R$ in line 41? They seem to be undefined up to that point.\n\n4.  For clarity, it may be better to replace $z_i$ with $z_{i,t}$ in line 54 to show that such samples are distinct and i.i.d. for each sample and time index.\n\n5. The paper and the considered setting 1.1. depend strongly on the DDPM algorithm. To make the paper self-contained, it is necessary to describe this algorithm.\n\n6. In Setting 1.1, does $P$ denote the number of parameters per layer?\n\nLIMITATIONS:\nNot Applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze the sample complexity of training a score-based diffusion model. They show that, with a sufficiently expressive neural network,  \\tilde O(d^2 P D log \\Theta \\log^3 \\frac{1}{\\gamma} / \\epsilon^3) samples are needed to learn an accurate diffusion model. Compared to the existing result, their bound has exponentially better dependence on \\Theta, \\gamma and D, and a better polynomial dependence on d and P, at the cost of a worse polynomial dependence in \\epsilon.\n\nSTRENGTHS:\nThe paper improves the existing bound for training a diffusion model with sufficiently expressive neural network. They analyze the barrier for L_2 accuracy as a criterion and propose to use a (1-\\delta)-quantile error in their analysis, which would finally scale polylogarithmically in 1/\\gamma, and suffices for fast sampling via the reverse SDE. Basically, the organization of this paper is good, and the proof sketch is clear, though it is a hard to check all the details.\n\nWEAKNESSES:\n1. Generally, this paper discusses the training sample complexity of a score-based diffusion model. The assumption is somehow strong, e.g., the neural networks can represent the score. Besides, the author do not include the optimization error in the analyses.\n\nQUESTIONS:\n1. As the bound in lemma 1.4 is with probability and in lemma 1.3, the requirement must hold for all N,  what if we make sampling step N \\to \\infty in Theorem 1.2? Does the bound in lemma 4 for each t independent or is controlled by the same stochasticity?\n\n2. What is the quantitive requirement on \\mathcal{F} in Assumption 2? What does the \"sufficiently small\" mean?\n\n3.Why is it required to constraint the functions represented by a fully connected neural network with ReLU activations and depth D, with P parameters, each bounded by \\Theta? Is it possible to be extended to other function classes?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the sample complexity of training diffusion models. The bound derived in the paper is exponentially better than the previous results. The paper also discusses the difficulty of learning the score function in $L^2$.\n\nSTRENGTHS:\n1. The paper derives better sample complexity results for diffusion models. The bound in this work is exponentially better than the existing results.\n2. The paper discusses the difficulties of learning the score in $L^2$. Two examples are provided to show this challenge.\n\nWEAKNESSES:\n1. The major concern is the presentation of this paper. I understand the results are important. However, the authors should clearly position this work in the literature. I suggest the authors provide a table that compares the sample complexity to the ones in prior work. In particular, the authors should discuss the improvement in terms of both the TV bound and the Wasserstein results. \n2. I notice the authors use BMR20 as a benchmark in Section 1.1. However, there are many better sample complexity results on diffusion models since that work. I suggest the authors carefully state how this work differs from those results. Also, I suggest the authors emphasize the main technical contributions in the proof instead of stating all the steps one by one. In particular, which step and what technique lead to the exponential improvement?\n3. The importance of Section 4 is not clear. Recent work has explored the learning score function in $L^2$; see [1-5]. All the work listed shows learning the score in $L^2$ is possible, and some of them show the minimax optimal rate. Specifically, learning the score function for an arbitrary distribution can be hard. However, learning the denoised score function during the forward process can be much easier. Therefore, I suggest the authors clarify why the examples in Section 4 can be interesting.  \n\n[1] Chen, Minshuo, et al. \"Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Han, Yinbin, Meisam Razaviyayn, and Renyuan Xu. \"Neural network-based score estimation in diffusion models: Optimization and generalization.\" arXiv preprint arXiv:2401.15604 (2024).\n\n[3] Wibisono, Andre, Yihong Wu, and Kaylee Yingxi Yang. \"Optimal score estimation via empirical bayes smoothing.\" arXiv preprint arXiv:2402.07747 (2024).\n\n[4] Zhang, Kaihong, et al. \"Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions.\" arXiv preprint arXiv:2402.15602 (2024).\n\n[5] Oko, Kazusato, Shunta Akiyama, and Taiji Suzuki. \"Diffusion models are minimax optimal distribution estimators.\" International Conference on Machine Learning. PMLR, 2023.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors studied the sample complexity of training diffusion models. By using a sufficiently expressive neural network, the authors showed an exponential improvement in the dependence on Wasserstein error and network width, which is expressed as $\\tilde{O}(d^2PD\\log\\Theta \\log^3(1/\\gamma)/\\varepsilon^3)$. This bound has a better polynomial dependence on the dimension $d$ and $P$, a better exponential dependence on $\\Theta, \\gamma$ and $D$ but a worse polynomial dependence on $\\varepsilon$ as a cost.\n\nSTRENGTHS:\nThis paper is clearly written and the presentation is fairly good. The theorems and lemmas proposed in this paper are sound and solid.\n\nWEAKNESSES:\nThis contribution of this paper seems not to be enough for an accept of a top tier machine learning conference. As we know, the TV error or Wasserstein error of learning diffusion models mainly lies in the discretization error caused by solving backward ODE (or SDE) through Euler Maruyama method, as well as the score estimation error. The discretization error is dominated by the latter error according to Chen et al.'s work. In this paper, the authors mainly studied the score estimation error. My biggest concern is that inserting a better generalization bound of score estimation (with regard to polynomial dimensional dependence) is not novel enough. For more questions, please refer to the next section.\n\nQUESTIONS:\n1. For the assumption A2, I think you can directly put the result of Theorem C.3 into the main text since the function approximation is also important, and the statement like \"sufficiently small\" doesn't seem to be strict. \n2. Please provide some explanations on the novelty of the sample complexity theory, other than an insert of polynomial dimension learning error rate of expressive neural networks. \n3. The work focuses on DDPM. However, in the original work of Song et al, they used VE (variance exploding) for consistency training. Is it possible for the sample complexity bound in the paper adapted to the VE case?\n\nLIMITATIONS:\nThere is not potential societal impact of this work since it is a completely theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}