Title: Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations
Venue: NeurIPS 2023
Accepted: yes
Keywords: diffusion models; stochastic differential equations; score-based generative models; asymptotic analysis
Abstract: The diffusion model has shown remarkable success in computer vision, but it remains unclear whether the ODE-based probability flow or the SDE-based diffusion model is more superior and under what circumstances. Comparing the two is challenging due to dependencies on data distributions, score training, and other numerical issues. In this paper, we study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. We first introduce a pulse-shape error to perturb the score function and analyze error accumulation of sampling quality, followed by a thorough analysis for generalization to arbitrary error. Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient. However, when the perturbation occurs earlier, the SDE model outperforms the ODE model, and we demonstrate that the error of sample generation due to such a pulse-shape perturbation is exponentially suppressed as the diffusion term's magnitude increases to infinity. Numerical validation of this phenomenon is provided using Gaussian, Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10.

Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations

Abstract

The diffusion model has shown remarkable success in computer vision, but it remains unclear whether the ODE-based probability flow or the SDE-based diffusion model is more superior and under what circumstances. Comparing the two is challenging due to dependencies on data distributions, score training, and other numerical issues. In this paper, we study the problem mathematically for two limiting scenarios: the zero diffusion (ODE) case and the large diffusion case. We first introduce a pulse-shape error to perturb the score function and analyze error accumulation of sampling quality, followed by a thorough analysis for generalization to arbitrary error . Our findings indicate that when the perturbation occurs at the end of the generative process, the ODE model outperforms the SDE model with a large diffusion coefficient. However, when the perturbation occurs earlier, the SDE model outperforms the ODE model, and we demonstrate that the error of sample generation due to such a pulse-shape perturbation is exponentially suppressed as the diffusion term's magnitude increases to infinity. Numerical validation of this phenomenon is provided using Gaussian, Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10.

1 Introduction

Diffusion models have achieved remarkable success in various artificial intelligence context generation tasks, particularly in computer vision [13]. This technique is rapidly evolving with industrial-level products like DALL · E series. The diffusion model was first proposed and studied by Sohl-Dickstein et al. [26] in 2015. Later, Song and Ermon [29] proposed score matching with Langevin dynamics (SMLD) and Ho et al. [16] further explored the Denoising Diffusion Probabilistic Models (DDPM). Both formalisms can be interpreted as time-discretization of stochastic differential equations (SDEs) [30]. Since the publication of these seminal works, many techniques have been proposed to improve the efficiency and accuracy of diffusion models, such as DDIM [28], Analytic-DPM [4], gDDIM [36], EDM [19], and consistency model [31], among others.

The score-based diffusion model involves two steps [30, 17]. Firstly, one estimates the score function, which is the gradient of the logarithm of the probability density function, in the form of a neural network. This step uses trajectories of an Ornstein-Uhlenbeck (OU) process starting with given data

samples. This process of injecting noise into structured data is usually referred to as the inference process . Secondly, new samples are generated by simulating a time-reversed SDE, with a drift term depending on the learned score function from the first step. This is known as the generative process .

In general, there are two diffusion coefficients g in the inference process and h in the generative process; see § 2. Regardless of the choice of h and g , it is always possible to design an SDE in the generative process that matches the forward inference process in the weak sense, i.e., the probability density functions match for both processes. We highlight that this function h (unlike g ) does not only appear in the diffusion term, but also enters the drift term in the generative SDE. The choice of g is equivalent to time re-scaling (see Appx. B.2), while the choice of h is an important topic in practice. Two common choices of h are Probability Flow h = 0 [9, 32], which refers to as an ODE, and an SDE-based diffusion model with h = g [16, 29, 30]. When the score training is accurate, the choice of this function h does not affect the sample generation quality in the continuous-time setting.

In practice, numerical error is inevitable during training the score function. Recent theoretical works [10, 7] have shown that the sample generation quality are affected by three aspects: (1) the truncation of simulation time to a finite T ; (2) the inexact score function; (3) the time-discretization error. The first error is not significantly since the forward OU process converges to the equilibrium Gaussian measure exponentially fast in T . The third error can be reduced systematically by more efficient numerical schemes [21], such as exponential integrator proposed in [35, 36]. The inexact training of score function has a few important but subtle consequences. Recent works [10, 7] analyzed the convergence rate of diffusion models, provided that the score training error is sufficient small. However, once the score training is not accurate, the nice equivalence of the generated distribution free of the generative diffusion coefficient h no longer holds as in the idealized situation of exact score function. This raises a key question of our interest about how the choice of h can affect the sample quality in the face of the inexact score training error. Qualitatively, there are two distinctive cases: h = 0 or h is large. An important question to ask is: in the presence of non-negligible score training errors, which h will produce better sampling quality? Is it the probability flow ( h = 0 ) or the SDE? More quantitatively, what magnitude of h is optimal?

Related works The impact of h on the generative process seems not yet fully investigated in recent literature, as most experiments used the default choice of this parameter. However, some authors have reported related empirical observations. For example, Song et al. [30] empirically observed that the choice of h = g produces better sample generation quality than the ODE case ( h = 0 ) with real datasets. On the other hand, Denoising Diffusion Implicit Models (DDIM) in [28] includes both deterministic and stochastic samplers and points out that the probability flow ( h = 0 ) can produce better samples with improved numerical schemes for the generative process. [36] generalized the DDIM and tried to explain the advantages of a deterministic sampling scheme over the stochastic one for fast sampling. Moreover, Karras et. al., [19] had empirically searched for optimal coefficients which had shown to bring practical advantages. None of these empirical results delivered comprehensive investigations on the influence of the diffusion coefficient, and a consistent and affirmative answer to our question still awaits. Recently, there has been rapid progress in theoretical works on error analysis for diffusion models, as seen in [10, 7] and references therein. However, these analyses usually assume specific settings of h , such as h = g . Furthermore, it seems that directly analyzing upper bounds based on these error estimations cannot provide adequate information about choosing the optimal h ; see Appx. B.4. Albergo et al. proposed a unified framework known as stochastic interpolants and slightly discussed the optimal choice between the probability flow and diffusion models [1, Sec. 2.4]. It is interesting to see how our theoretical analysis below can generalize to their promising unified settings [1].

Our approach To investigate the effect of the diffusion coefficient h on sampling quality, we adopt the continuous-time framework, which precludes time discretization errors. We measure sample quality by the KL divergence between the data distribution p 0 and the distribution of the generative SDE at the terminal time T . Given the assumption that the score function carries numerical errors, we consider h as a controller and aim to minimize the KL divergence with respect to h . While the optimization problem is straightforward to set up, it is challenging to draw valuable theoretical insights in a general setting of approximate score functions. Therefore, we choose the asymptotic approach, assuming that the error from the training score is reasonably small with a magnitude of ϵ . Under this assumption, the leading-order term of the KL divergence takes the form error of sample generation in KL divergence = L ( h ) ϵ 2 + O ( ϵ 3 ) .

This ϵ 2 order is known in [7, 10], but the dependence of this Gateaux differential L ( h ) on h and other factors has yet to be understood at all. Our contribution is to analyze how L ( h ) behaves as h varies; in particular, by considering the constant h in two limiting situations: h = 0 and h ≫ 1 .

Main Contributions We summarize main contributions below:

We prove that when the error in score function approximation is a time-localized function only at the beginning of the inference step (i.e., at the end of the generative process), the ODE case ( h = 0 ) outperforms the SDE case ( h →∞ ); see Prop. 3.5. If this (time-localized) error occurs in the middle, then the SDE case has an exponentially smaller error than the ODE case ( h = 0 ), as h →∞ (see Prop. 3.4). See Appx. E.4 for reasons behind the time-localized choice.

For a general score training error, we prove that as h →∞ , the leading-order term L ( h ) above converges exponentially fast to a constant, which only depends on the distribution p 0 and the score training error at the end of the generative process; see Prop. 3.6. The conclusion about the optimal h depends on how the score training error is distributed over the time horizon [0 , T ] .

Numerically, we validate the above phenomenon for 1D Gaussian, 2D Gaussian mixture, and Swiss roll distribution, as well as realistic datasets like MNIST and CIFAR-10. Due to the tight connection between the distribution of score training error and h , our results may suggest backwardly modifying loss functions during training to adapt to a particular diffusion coefficient of interest. This is a topic of independent interest and we report some preliminary experiments in Appx. I to validate potential applications of our theoretical analysis. A comprehensive investigation will be left as future works.

Notation convention The time duration T > 0 is a fixed parameter. For any time-dependent function ( t, x ) ↦→ f t ( x ) , where x ∈ R d and t ∈ [0 , T ] , we denote f ← t ( x ) ≡ f T -t ( x ) . Sometimes we directly use the 'arrowed' variables f ← t ( x ) to highlight the direction of time is from reference noise to the data distribution (i.e., the generative direction) even without referring to f first. The notation f ≲ g means that f ≤ cg where c → 1 in a certain limit, i.e., lim sup f / g ≤ 1 ; f ∼ g means lim( f / g ) = 1 . The asymptotic parameter will be explained below explicitly. When two matrices A ⪯ B , it means B -A is positive semidefinite. I d is the d -dimensional identity matrix; I A means an indicator function of a set A ; Id is the identity operator. For a random variable X , law ( X ) means the distribution of X . Some important quantities are summarized in Appx. A.

2 Background

Score-based generative models Suppose we have a collection of data from an unknown distribution with density p 0 , we can inject noise into data via the following SDEs:

where the drift coefficient f ( · ) ( · ) : R d × R → R d is a time-dependent vector field, and the diffusion coefficient g ( · ) : R → R is a scalar-valued function (for simplicity). A widely used example is variance-preserving SDE (VP-SDE) with f t ( x ) = -1 / 2 g 2 t x and g > 0 is typically chosen as a non-decreasing function in literature [30]. Without loss of generality, we can assume g t = 1 since for any non-zero g , its effect is simply to re-scale the time; see Appx. B.2.

Denote the probability density of X t as p t and the score function is defined as ∇ log p t . One main innovation in diffusion models is to find a 'backward' SDE Y t such that Y t drives the state with distribution p T back to p 0 . We adopt the arrow of time in this backward direction now and write Y t as

where h ← t is an arbitrary real-valued function of time. The distribution of Y t is denoted as q t . Provided that the score function is available, we can select A ← such that q t is the same as p T -t , in particular, q T = p 0 . It is not hard to derive that we can ensure q t ≡ p T -t if we choose

A self-contained proof is provided in Appx. B.1. When h ← = g ← , it refers to the backward SDE used in [30]; when h ← = 0 , it refers to the probability flow therein. More general interpolation of diffusion and flow can be found in, e.g., [1].

Training of score functions The above score function ( t, x ) ↦→∇ log p ← t ( x ) needs to be trained from data. Denoising score matching [33] refers to the following score-matching loss (SML) function to train the score whose parameterized architecture is denoted as S :

where p t | 0 ( x t | x 0 ) is the transition probability of the state x 0 at time 0 towards the state x t at time t for the forward process (1). The function ω t ≥ 0 is a weight function. The default choice in many literature is that g t = √ β 0 +( β 1 -β 0 ) t , 0 < β 0 < β 1 are parameters, and one chooses the weight function as follows:

The quantity ϖ t has the meaning as the standard deviation of X t conditioned on a fixed X 0 in the forward process. See § 3.7 and Appx. I for more weight functions.

Source of errors There is usually intrinsic error due to an inexact score function. It is not negligible in many scenarios, e.g., there is only a finite amount of samples of p 0 available or only a small neural network architecture is achievable. However, it is reasonable to assume that this non-negligible error is reasonably small, and we decompose the trained score function S ← t into

̸

where ϵ is small, E ← t is assumed to be O (1) and the total error is ϵ E ← t . The generative process used in practice has to use law ( Y 0 ) = N (0 , I d ) instead since p T is intractable in (2). By choosing T large enough so that p T ≈ N (0 , I d ) , we can neglect this error due to the finite T . Besides, we also need some numerical schemes to simulate this generative SDE, which also leads into discretization errors. In summary , there are three sources of errors (1) p T = N (0 , I d ) : this is the source of errors in the initial distribution of the generative process; (2) E ← = 0 : error from imperfect score function from training; (3) numerical discretization of the generative process. The third error can be systemically eliminated by choosing a high-order scheme [21] or an extremely small time step. It has been observed that by choosing a more accurate numerical scheme, e.g., exponential integrator, one can reduce the computational costs [35, 36]. As for the first error, if one chooses the OU process for (1), p T converges to N (0 , I d ) exponentially fast and thus T = O (ln( δ )) where δ is the error between p 0 and the distribution of generated samples. Therefore, the choice of T is, in practice, not hard to manage. More details about these three error sources can be found in, e.g., [7, 10] or Appx. B.3.

̸

ScoreError w.r.t.Timet

SamplingError w.r.t.Diffusion Coefficient h

Givenafixedscoreerror(black),

byProp.3.5 orProp.3.6

westudytheeffectfrom

blueandgreenregions

by Prop.3.4

separately

erroris

"additive"

from different

regions

0

ho-2

ho

0

h

3 Asymptotic analysis of terminal errors

We use the KL divergence between the data distribution p 0 and the distribution of generated samples to quantify the performance of generative model, which depends on the error of score function ϵ E ← ,

the diffusion coefficient h ← , and data distribution p 0 . To extract the main feature, we first let ϵ → 0 and estimate

Whenever p 0 and E ← are obvious from the context, we simply write L ( h ← ) ≡ L ( h ← , E ← , p 0 ) . Next, we formulate the main problem setup and assumptions in § 3.1. The expression of L is shown in Prop. 3.2. Then we let h ← t ≡ h be independent of time, and study how the leading order function L depends on h in various settings of the error function E ← t . Firstly, we consider E ← t ( x ) = δ t -s E ( x ) as a time-localized function and two limiting scenarios: h ← = h where h = 0 (ODE case) and h →∞ (SDE case with large diffusion). When E ← t ( x ) = δ t -s E ( x ) is a time-localized function at the end of the generative process (i.e., s is close to T ), the ODE case will outperform the SDE case (see Prop. 3.5); otherwise, the SDE case has an exponentially smaller error than the ODE case as h →∞ (see Prop. 3.4). Secondly, by combing Prop. 3.4 and Prop. 3.5, the tail behavior of h ↦→ L ( h ) for a general E ← is described in Prop. 3.6. The reasons behind considering this pulse-shape E ← t will be discussed in Appx. E.4.

3.1 Set-up and assumptions

We fix the time T and consider the following SDE for the generative process in t ∈ [0 , T ] ,

which can be regarded as a perturbed equation (2) of Y t . Denote the distribution of ˜ Y t as ˜ q t . Note that ˜ q T depends on both h ← and ϵ E ← t (hidden inside S ← t ≡ ∇ log p ← t + ϵ E ← t ); however, when ϵ = 0 , by (3), ˜ q T ≡ q T ≡ p 0 for any h ← . We quantify the sample generation quality via

Due to the presence of ϵ E ← with non-zero ϵ , in general KL ( p 0 || ˜ q T ) > 0 . Assumption 3.1. Throughout this section, we assume that:

For the forward process, we assume f t ( x ) = -1 2 x , g t = 1 without loss of generality.

The data distribution has the density p 0 .

There exists c U ∈ R such that U 0 ( x ) -| x | 2 / 2 ≥ c U , for any x ∈ R d , where U 0 := -log p 0 .

Recall that a generic g t is equivalent to the time re-scaling (Appx. B.2). So this choice of g t = 1 refers to the generic choice in VP-SDE [30]. The second assumption is also mild; in practice, if p 0 is a delta distribution, a common practice is that one tries to learn the mollified version p σ ( x ) := ∫ R d 1 (2 πσ 2 ) d/ 2 exp ( -( x -y ) 2 / 2 σ 2 ) p 0 ( y )d y instead, as in the GAN [27] and early-stop techniques [7, 10]. The third assumption is not restrict, for example, U 0 ( x ) = | x | 2 and c U = 0 . As many realistic datasets are almost compactly supported, we expect that ρ 0 = e -U 0 decays faster than a Gaussian, namely, ρ 0 ( x ) ≤ ˜ Ce -| x | 2 / 2 for some ˜ C > 0 , which reduces to the third one.

3.2 Asymptotic expansion of the KL divergence with respect to ϵ

We introduce a time-dependent operator

which is the generator in the Fokker-Planck equation of q t for (2). Define an operator Φ ( h ← ) s,t as follows: given any function µ , define Φ ( h ← ) s,t ( µ ) to be the solution at time t of the following FokkerPlanck equation with r ∈ [ s, t ] : ∂ r θ r = L ( h ← ) r ( θ r ) , and with initial condition θ s = µ . We define Φ ( h ← ) s,t ( µ ) := θ t . Properties of this operator are collected in Appx. C.1.

We use the notation ˜ q ϵ to refer to ˜ q since we need to calculate the its derivative for the small parameter ϵ . The dependence of ˜ q on h ← is suppressed for short notations. We have the following asymptotic result with the proof given in Appx. C.3.

Proposition 3.2. Define v T := ∂ ϵ ˜ q ϵ T | ϵ =0 as the first-order perturbation of ˜ q ϵ T . We have

where

3.3 The role of h ← in the Fokker-Planck operator L ( h ← ) t

Let the potential U t := -log p t , and by the notation of time-reversal, U ← t ≡ U T -t . When h ← > 0 , we can rewrite

We now introduce a probability distribution induced by the potential V ← t :

By convection, we also have V t = V ← T -t and ρ t = ρ ← T -t . Note that V ← t depends on h ← and when h ← → ∞ , we have V ← , ∞ t = U ← t . The role of h ← in L ( h ← ) t now can be viewed as the time re-scaling and the effect of L ( h ← ) t at a local time t can be viewed as evolving the Fokker-Planck equation associated with the overdamped Langevin dynamics in the time-dependent potential V ← t for O ( h ← t 2 / 2 ) amount of time. When h ← → ∞ , L ( h ← ) t can be roughly viewed as constructing an 'almost quasi-static' thermodynamics [6] bridging the initial p T and the (quasi-)equilibrium p 0 = e -U ← T : for any distribution µ ← t (probably far away from ρ ← t ), within a short time period ∆ t slightly larger than O ( 1 /h ← t 2 ) , we have µ ← t +∆ t ≈ ρ ← t +∆ t , provided that s ↦→ µ ← s evolves according to L ( h ← ) s ; see Appx. C.2. This key finding will guide our analysis of the solution operator Φ ( h ← ) t,T .

3.4 Score function is perturbed by a pulse

From (10), we know that v T combines the averaged effect of Φ ( h ← ) t,T ( ∇· ( p ← t E ← t )) for various t . As a first result, we consider E ← t ( x ) = E ( x ) δ t -s for a fixed time instance s ∈ [0 , T ) , where δ t -s is the Dirac function. In this case, v T no longer involves time integration and we have

To proceed, we need to make additional assumptions:

Assumption 3.3.

For any t ∈ [0 , T ] , U t = -log p t is assumed to be strongly convex and the Hessian of U t is bounded by two positive numbers m t and M t as below

(2) For all t ∈ [0 , T ] , we choose h ← t = h as constant.

Introduce

Moreover, assume that

which characterizes the Hessian lower bound of V ← t (11). Note κ ← t ≈ m ← t when h ≫ 1 . We would like to explain the reason behind the above assumptions, in particular, their practical relevance. Part (1) Strong convexity: this is a common assumption for Langevin sampling analysis [11, 14]. As the role of L ( h ← ) t is essentially simulating a Langevin dynamics with time-dependent potential, it is reasonable to use this assumption as a starting point. Moreover, the algorithmic improvement in gDDIM [10] is highly inspired by a form with assuming the data distribution as a Gaussian; Fréchet inception distance (FID) [15], a widely used metric to evaluate the quality of generative model, essentially treats the data (in the feature space) as Gaussians. Therefore, we believe that this assumption can still capture some main features of realistic datasets. Part (2) m t ≥ 1 for any t ∈ [0 , T ] : The second assumption m t ≥ 1 means that p t is more localized (smaller variance) than the standard Gaussian (unit variance), which is compatible with Assumption 3.1 (3). It can also ensure that V t is strongly convex with positive Hessian lower bounds, i.e., κ ← t ≥ 1 , for any t ∈ [0 , T ] and h ∈ (0 , ∞ ) .

Proposition 3.4. Under Assumptions 3.1 and 3.3, suppose that E ← t ( x ) = E ( x ) δ t -s for some fixed s ∈ [0 , T ) . If h ≥ h lb := max { 1 / 2 , h 0 ( 1 / 2 ) , √ max { 0 , -c U ln(2) , sup t ∈ [ s,T ] C ← , (2) t } } , we have the upper bound of L ( h ) in (13) :

where C h = 1 2 ∫ ( ∇· ( p ← s E ) ) 2 / ρ ← s , C ← , (2) t = 20+30 M ← t 2 m ← t 2 ; see Appx. E.1 for details about C ← , (1) t and h 0 ( 1 / 2 ) ; c U ∈ R comes from Assumption 3.1. Moreover, lim h →∞ C h , lim h →∞ C ← , (1) t exist.

See Appx. E.1 for proofs. We remark that the above bound focuses on capturing the scaling with respect to h 2 but may not be tight for other parameters. It remains interesting to see how we can improve the above upper bound. The main conclusion is that: if the error function E ← t is a pulse at time t = s , then for a large h , L ( h ) will decay to zero exponentially fast with respect to h . For 1D Gaussian case, we can clearly observe such an exponential decay in Fig. 2a, where we pick E ← t = I t ≤ 0 . 95 T ∇ log p ← t . The intuition behind this exponential suppressed prefactor is that for large h , Φ ( h ← ) s,T can be viewed as an almost quasi-static thermodynamics dragging any positive measure towards ρ ← T ≈ p 0 , as mentioned above in § 3.3. As ν = ∇· ( p ← s E ) has measure zero, we can split it into positive and negative parts: ν = ν + -ν -( assume ∫ ν ± = 1 WLOG). Each term Φ ( h ← ) s,T ( ν + ) ≈ ρ ← T ≈ Φ ( h ← ) s,T ( ν -) , which explains that Φ ( h ← ) s,T ( ν ) ≈ 0 for large h .

0

1

2

3

4

h 2

10 -2

10 -1

10 0

L(h)

σ0 =0.4

σ0 =0.6

σ0 =0.8

(a)

E

t

=

I

t

0

.

95

T

∇

log

p

t

0

25

50

75

100

h 2

10 -2

10 -1

L(h)

σ0 =0.1

σ0 =0.2

σ0 =0.3

(b)

E

t

=

I

t

0

.

995

T

∇

log

p

t

0

25

50

75

h 2

10 0

L(h)

σ0 =0.2

σ0 =0.4

σ0 =0.6

(c)

E

t

=

∇

log

p

t

3.5 Score function is only perturbed near the end of the generative process

Proposition 3.5. Under Assumptions 3.1 and 3.3, suppose that E ← t ( x ) = I t ∈ [ T -a,T ] E ( x ) where a ≪ 1 . Then when a ≪ 1 and h ≫ 1 , asymptotically ,

with κ 0 = (1 + 1 / h 2 ) m 0 -1 / h 2 as in (16) . The upper bound of L ( h ) is tight asymptotically.

We remark that we made no assumption on how a h 2 scales . If h ≫ 1 , L ( h ) / L (0) ≲ 4 ( 1 -e -a h 2 κ 0 / 2 ) 2 /a 2 κ 2 0 . Case (I): If a h 2 ≫ 1 , then L ( h ) / L (0) ≲ 4 / a 2 κ 2 0 , which is large as a ≪ 1 . Case (II): If a h 2 ≪ 1 , then L ( h ) / L (0) ≲ h 4 , which is still large. In either case, L ( h ) / L (0) is expected to be large for a general E and the ODE model is preferred in this case. The intuition is that there is almost no time for the operator L ( h ← ) t to suppress the error E , so the prefactor 1 + ( h ← t ) 2 in v T (10) dominates (which is the key difference compared with Prop. 3.4). The proof is postponed to Appx. E.3. In Fig. 2b, we consider 1D Gaussian and only perturb the score function at the end of the generative process ( E ← t = I t ≥ 0 . 995 T ∇ log p ← t ); clearly, the SDE-based models have comparatively larger error.

3.6 General error in score function

We can generalize Prop. 3.4 and Prop. 3.5 to a general error function E ← , and observe that L ( h ) will converge to a constant exponentially fast as h →∞ .

Proposition 3.6. Under Assumptions 3.1 and 3.3, we consider a general error function ( t, x ) ↦→ E ← t ( x ) . Let γ = inf h ≥ h lb inf t ∈ [0 ,T ] κ ← t . For any α ∈ (0 , 1) and β ∈ (0 , 2) , when h ≫ 1 ,

where C is given in Appx. F .1 and T (only depending on p 0 and E ← T ) is upper bounded by

In the limit h →∞ , (1 -α 2 ) T ≲ L ( h ) ≲ (1 + α 2 ) T , where α ∈ (0 , 1) is arbitrary. Hence, the tail behavior is that L ( h ) converges to T exponentially fast as h →∞ . If we assume that E ← T = ∇ log p 0 , p 0 = N (0 , σ 2 0 I d ) in d -dimension, then the above upper bound is simply T ≲ d , which is independent of σ 0 (see Appx. F.2). For 1D Gaussian in Fig. 2c, we can indeed observe that lim h →∞ L ( h ) exists, and is bounded above by d = 1 ; see Appx. G for more types of error functions.

The above upper bound has an interesting tight connection to the generator of (overdamped) Langevin dynamics with drift -∇ U 0 ≡ ∇ p 0 . If we adopt constrained score models [23, 25], namely, parameterizing log p t instead of the score function ∇ log p t during training, the error E ← T = ∇ φ for some scalar-valued function φ . Then the above upper bound becomes

where L ∗ ( φ ) := △ φ -∇ U 0 · ∇ φ whose adjoint operator L ( µ ) = ∇· ( ∇ U 0 µ ) + △ µ is the FokkerPlanck generator of the following Langevin dynamics d X t = -∇ U 0 ( X t ) d t + √ 2 d W t where W t is the standard Brownian motion. We remark that this formula (19) is general for constrained score models [23, 25]; see also F.2 for elaborations. An interesting open question is whether and how we can take the above upper bound into consideration when designing the loss function.

3.7 An application: exploring the effect of training weight

The above theoretical discussions suggest that diffusion models with large diffusion coefficients are more negatively affected by score error near data's side, whereas the ODE model is more negatively affected by the score error near the noise end. This leads us to conjecture that if we can control the training (e.g., by optimizing the training weight ω t ), so that the score error distribution near the noise end is reduced and meanwhile the score-matching loss is not significantly impacted, then it will very likely improve the ODE models. We report preliminary numerical experiments to support this idea in Appx. I, whereas a comprehensive study will be left as future works.

4 Numerical experiments

We present experiments on 2D Gaussian mixture model, Swiss roll, CIFAR10 to support our theoretical results: when numerical discretization error is not dominating, the sampling quality increases as h increases, a reminiscent of Prop. 3.4. Experimental details are postponed to Appx. H, as well as more numerical results (e.g., results about 1D Gaussian mixture and MNIST). Results by adopting various weight functions, a technique arising from theoretical predictions, are postponed to Appx. I. Source codes are available at https://github.com/yucaoyc/OptimalDiffusion .

Example 1: 2D 4-mode Gaussian mixture. We verify the theoretical results on 2D Gaussian mixture with a specified score error. In Fig. 3, a clear trend is that a higher h produces generated distributions that better match the marginal densities of p 0 , and it is numerically verified by the purple line of Fig. 4a. In Fig. 4, with multiple settings of E ← t and ϵ , we observe a consistent phenomenon that as h increases, the KL divergence of true data and generated data converges exponentially fast, thus validating Prop. 3.6. It worths noticing that in all three settings of E ← t , by simply adopting a larger diffusion coefficient h in (3), we can obtain better generative models without any extra training costs.

-1

0

1

0

5

-1

0

1

0

5

h 2

=0.0 g 2

-1

0

1

0

2

4

-1

0

1

0

2

4

h 2

=1.0 g 2

-1

0

1

0

2

4

-1

0

1

0

2

4

h 2

=2.0 g 2

-1

0

1

0

2

4

-1

0

1

0

2

4

h 2

=3.0 g 2

-1

0

1

0

2

4

-1

0

1

0

2

4

h 2

=4.0 g 2

KL divergence

2

0

1

2

3

4

h 2

/g

2

10 -3

10 -2

10 -1

KL divergence

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(a)

E

t

=

∇

log

p

t

0

1

2

3

4

h 2

/g

2

10 -3

10 -2

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(b)

E

t

=

1+sin(2

πt/T

)

∇

log

p

t

0

1

2

3

4

h 2

/g

2

10 -2

KL divergence

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(c)

E

t

=

I

t<

0

.

95

T

∇

log

p

t

Example 2: Swiss roll. We consider Swiss roll, a more complex distribution without exact score function available. We train the score function with the denoising score-matching objective [33] (Appx. H). The first plot in Fig. 5a shows the difference between true data and generated data measured by Wasserstein distance, which decays to zero exponentially fast, verifying Prop. 3.6. In the second plot of Fig. 5a, we tested various h and time steps for the generative process. The ODE model ( h = 0 ) does not improve, as the number of time discretization steps increases, but near the continuous-time limit, all SDE cases ( h > 0 ) are better than the ODE model. It suggests that our conclusions here is limited to the continuous-time setting. The exploration of time discretization errors will be future works. The generated data results in Fig. 5b demonstrate that with increasing h , the sample quality is improved; see Appx. H.6 for more results.

0

1

2

3

4

h 2

/g

2

1.0 ×10 -2

1.5 ×10 -2

2.0 ×10 -2

10 2

10 3

10 4

Discretization steps

2 ×10 -2

4 ×10 -2

6 ×10 -2

h 2

=0g 2

h 2

=1g 2

h 2

=2g 2

h 2

=3g 2

h 2

=4g 2

(a) Improved model performance with larger

h

(b) Visualization results for selective

h

Example 3: CIFAR-10. When using a large amount of parameters for score matching in practice, we observe that SDEs appear to perform better than ODEs as discretization error descreases, a result similarly observed on Swiss roll. This implicates the practical applications on generating samples of better quality under a given (pre-trained) score-matching model.

5 Summary and outlook

In this work, we study the effect of the diffusion coefficient on the quality of overall sample generation in the generative process. Theoretically, we provide understandings of scenarios in which the ODEbased model and the SDE-based model is superior than the other; see Prop. 3.4 and Prop. 3.5. Numerically, these results are validated via toy examples as well as benchmark datasets.

There are many interesting directions for continuing works. (1) As we focus on the asymptotic case, a time-dependent h ← t with large magnitude (i.e., h ← t ≫ 1 for all t ) is essentially no different from a constant h ← t ≡ h with h ≫ 1 . Whether it is possible to use time-dependent t ↦→ h ← t to combine the advantages of ODE and large diffusion cases in dealing with different types of error of score functions in the non-asymptotic region is still an open question. (2) Can we design a practical criterion that directly learn the optimal magnitude of the noise-level function h ← t by looking at the score-training error distribution? Can we develop certain theoretical understanding of the empirical results in [19]? (3) How can we find a stable and accurate numerical scheme to deal with the fast diffusion case? (4) How can we generalize the above theoretical results by removing the convexity assumption, and including the low-dimensional feature of datasets into the theory [8, 5, 18]?

Concerning the broad impact, though we don't foresee any negative social impact of this work, the potential improvement of generative model might relate to creation of 'deep fakes'.

Acknowledgments and Disclosure of Funding

YC is sponsored by Shanghai Pujiang Program, and acknowledges the support from City University of Hong Kong during his visit. JC and YL are supported by the NSFC Major Research Plan - Interpretable and General-purpose Next-generation Artificial Intelligence (92370205). XZ is supported by General Research Funds from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. 11308121, 11318522) and the NSFC/RGC Joint Research Scheme [RGC Project No. N-CityU102/20 and NSFC Project No. 12061160462].

References

Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Eijnden. Stochastic Interpolants: A Unifying Framework for Flows and Diffusions, 2023. arXiv:2303.08797 [cond-mat].

D. Bakry and M. Émery. Diffusions hypercontractives. In Lecture Notes in Mathematics , pages 177-206. Springer Berlin Heidelberg, 1985. doi: 10.1007/bfb0075847.

Dominique Bakry, Ivan Gentil, and Michel Ledoux. Analysis and Geometry of Markov Diffusion Operators . Springer International Publishing, 2014. doi: 10.1007/978-3-319-00227-9.

Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations , 2022.

Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Transactions on Machine Learning Research , 2022.

Herbert B Callen. Thermodynamics and an Introduction to Thermostatistics . John Wiley & Sons, Nashville, TN, 2 edition, 1985.

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 4735-4763, 2023.

Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 4672-4712, 2023.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations , 2023.

Xiang Cheng and Peter Bartlett. Convergence of Langevin MCMC in KL-divergence. In Algorithmic Learning Theory , pages 186-211, 2018.

Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11462-11471, 2022. doi: 10.1109/ CVPR52688.2022.01118.

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-20, 2023. doi: 10.1109/TPAMI.2023.3261988.

Arnak S. Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic Langevin diffusions. Bernoulli , 26(3):1956-1988, 2020. doi: 10.3150/19-BEJ1178.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusionbased generative models and score matching. In Advances in Neural Information Processing Systems , 2021.

Ian T. Jolliffe and Jorge Cadima. Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 374(2065):20150202, 2016. doi: 10.1098/rsta.2015.0202.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , 2022.

Fima C Klebaner. Introduction to Stochastic Calculus with Applications . Imperial College Press, 1998. doi: 10.1142/p110.

Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations . Springer Berlin Heidelberg, 1992. doi: 10.1007/978-3-662-12616-5.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the Underlying Score Fokker-Planck Equation. In Proceedings of the 40 th International Conference on Machine Learning , 2023.

F. Otto and C. Villani. Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality. Journal of Functional Analysis , 173(2):361 - 400, 2000. doi: https://doi.org/10.1006/jfan.1999.3557.

Tim Salimans and Jonathan Ho. Should EBMs model the energy or the score? In Energy Based Models Workshop-ICLR 2021 , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 2015. PMLR.

Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amortised MAP inference for image super-resolution. In International Conference on Learning Representations , 2017.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 32211-32252. PMLR, 2023.

Esteban G. Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the loglikelihood. Communications in Mathematical Sciences , 8(1):217-233, 2010.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput. , 23(7):1661-1674, 2011. doi: 10.1162/NECO_a_00142.

LeCun Yann. MNIST, 1998. URL http://yann.lecun.com/exdb/mnist/ .

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2023.

Qinsheng Zhang, Molei Tao, and Yongxin Chen. gDDIM: Generalized denoising diffusion implicit models. In The Eleventh International Conference on Learning Representations , 2023.

Supplementary Material for 'Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations"

A Notation Convention.

B Discussion and proof for § 2

B.1 Proof of (3)

We re-state the conclusion in (3) in the following lemma:

Lemma B.1. For any function h ← t , if one chooses A ← as in (3) , then we have q t ( x ) = p T -t ( x ) for any t ∈ [0 , T ] and x ∈ R d .

Proof. We can easily write down the Fokker-Planck equation of (1)

Since we need to ensure q t = p T -t , we require

by noting that A ← t ( x ) is specified in (3). This equation is exactly the Fokker-Planck equation of (2).

B.2 The role of g t

The function t ↦→ g t as the diffusion coefficient in the forward Fokker-Planck equation (1), in fact, plays a role as time re-scaling as long as g t > 0 for any t . Indeed, if we have an SDE in the following

form (the VP-SDE in [30])

then by introducing τ : R + → R + via the ODE τ ′ ( t ) = ( g τ ( t ) ) -2 , τ (0) = 0 , we know that X t := X τ ( t ) satisfies the following SDE [20]:

where θ := τ -1 ( T ) means the inverse function of τ at time T . The Brownian motion W in X t is not the same Brownian motion in X t , i.e., the driven-noise in the last two equations are not the same technically; we slightly abuse the notation for the simplicity of notations. By Lem. B.1, this SDE has the backward process as follows:

where p t is the density function of X t by notation conventions in § 2. By Lem. B.1, so far we know that

Let f s := θ -( τ -1 )( T -s ) and Y s := Y f s , where s ∈ [0 , T ] . By straightforward computation, we know

To get the second line, we used the fact that

By chain rules, it is easy to compute that

Therefore, the above SDE for Y s has the form

This matches the form in Lem. B.1 by choosing

Hence, if we simply pick h = c as a constant function in (20), it has the same effect as choosing h ← s = cg ← s where c ∈ R + . The former (h = c ) is used in § 3 for simplicity in theoretical analysis, and the latter ( h ← s = cg ← s ) is used in numerical experiments in § 4 to match previous literature in practice (namely, a general g ). In conclusion, the above discussion justifies the consistency of notation and set-up between our theoretical analysis and numerical experiments.

B.3 Existing analysis of sample generation quality

In practice, one simulates the following SDE:

From the analysis by Chen et al. [7, Theorem 2.1] for the case h ← = g ← , one has

where ̂ q is the distribution of Z T after applying the exponential integrator to (21) and N is the number of time-discretization steps. The first term KL ( p T ||N (0 , I d ) ) ≤ ( M 2 + d ) e -T , where M 2 = E p 0 | x | 2 is the second moment of data distribution.

Therefore, to ensure that KL ( p 0 || ̂ q ) ≤ δ , it is sufficient to choose

The time-discretization error can be eliminated by choosing N →∞ . What is so far less clear in literature is the term O ( Tϵ 2 ) .

B.4 Existing error bounds appear to fail to characterize the optimal h ←

Note that the target dynamics q t in (2) and the approximated dynamics ˜ q t in (7) only differ in the drift term. Recall that q T = p 0 , and to estimate KL ( p 0 || ˜ q T ) ≡ KL ( q T || ˜ q T ) , we can simply estimate how the quantity KL ( q t || ˜ q t ) changes for t ∈ (0 , T ) . By [7, Lemma C.1], for any t ∈ R ,

where the Fisher information J ( p || q ) := ∫ d p ∣ ∣ ∣ ∇ log p q ∣ ∣ ∣ 2 . By Cauchy-Schwarz inequality,

where we chose c 2 = h ← t 2 + g ← t 2 2 h ← t 2 in the last line. This bound captures the scaling extremely well when h ← = g ← , which helps to establish the effectiveness of score-based diffusion method in [7]. However, this bound is not directly applicable for a general h ← , as it is clear that this bound fails to provide useful information when h ← ≈ 0 (namely, the probability flow), as well as the large diffusion case ( h ← →∞ ). From directly optimizing the upper bound, i.e., minimizing ( h ← t 2 + g ← t 2 ) 2 8 h ← t 2 , one ends up with the choice h ← = g ← , which has been used in many literature. We acknowledge that h ← = g ← is an effective choice; however, as we show in § 4, this is not really the optimal case in general, and the above argument cannot justify choosing the diffusion model over the probability flow.

C Discussion for § 3 and proof of Prop. 3.2

C.1 The operator Φ ( h ← ) s,t

Φ ( h ← ) ( · ) , ( · ) satisfies the following property, i.e., for any s, t, r ∈ R

Moreover, Φ ( h ← ) t,t = Id is an identity operator for any t .

For any s, t ∈ R ,

Proof. The structure in (24) is easy to imagine and is thus omitted herein. Next, we shall only prove the first result in (25) for illustration:

Lemma C.2. Suppose ( t, x ) ↦→ µ t ( x ) , ( t, x ) ↦→ θ t ( x ) are time-dependent functions. Suppose ∂ t µ t = L ( h ← ) t ( µ t ) + θ t with µ 0 = 0 , then

Proof. Note that

Therefore,

By applying the operator Φ ( h ← ) 0 ,T to both sides, we have

C.2 Time re-scaling of L ( h ← ) and connections to almost quasi-static Langevin process

By time re-scaling, it is immediate to obtain the following result:

Lemma C.3. Suppose h ← t = h for any t ∈ [0 , T ] for simplicity. For any probability distribution µ , the probability distribution Φ ( h ← ) s,t ( µ ) (with s < t ) is the final state of the following PDE on the time interval ˜ r ∈ [ 0 , ( t -s ) h 2 / 2 ] :

It corresponds to the following Langevin with time-dependent potential:

Proof. To obtain Φ ( h ← ) s,t ( µ ) , we simply solve the following PDE:

By change of variables ˜ r = ( r -s ) h 2 / 2 and ˜ µ ˜ r := µ r ≡ µ s + 2 ˜ r / h 2 , we immediately have (26), and the corresponding Langevin dynamics easily follows.

When h →∞ , the potential in the Langevin dynamics ˜ r ↦→ V ← s + 2 ˜ r / h 2 evolves extremely slowly. From (11), we also know that V ← s + 2 ˜ r / h 2 ≈ U ← s when ˜ r = 0 and is approximately U ← t when ˜ r = ( t -s ) h 2 2 . Therefore, the Langevin dynamics can be viewed as an almost quasi-static process [6] approximately transforming the state p ← s to the state p ← t over an extremely long time period though.

C.3 Proof of Prop. 3.2

Denote

Namely, we expand ˜ q ϵ t via the following

Recall that when ϵ = 0 , ˜ q 0 t ≡ q t ; in particular, ˜ q 0 T = q T Lem. B.1 = p 0 . The cost function can be easily expanded via Taylor's formula:

Since ∫ ˜ q ϵ T ≡ 1 , it is easy to know that ∫ v T = ∫ ζ T = 0 and thus

Next we need to study v T . Recall from (7) that the Fokker-Planck equation of ˜ q t is

By taking derivatives with respect to ϵ on both sides of this equation and then passing ϵ → 0 ,

By Lem. C.2,

D Preliminary results

We collect two lemmas which will be useful in proving Prop. 3.5 and Prop. 3.6 later. Lemma D.1.

The operator K ( f ) := △ f + ∇· ( ∇ V · f ) is a Hermitian/self-adjoint operator in the space ⟨· , ·⟩ L 2 ( ρ -1 ) where ρ ∝ e -V . Therefore, K has the eigen-decomposition in this weighted L 2 ( ρ -1 ) space.

Assume that ∇ 2 V ( x ) ≥ m I d for any x ∈ R d for some m> 0 . Then the operator -K is a positive operator on the space { f : ∫ f = 0 } with spectrum gap at least m .

Proof. For any f, g , note that

Therefore, K is a Hermitian operator. In the space { f : ∫ f = 0 } , we know

The validity of Poincaré inequality under the strong convexity assumption is well-known in literature; see, e.g., [2, 24] and [3, Corollary 4.8.2].

Lemma D.2. Suppose the operator K is defined as K ( f ) := △ f + ∇· ( ∇ V · ( f ) ) with ∇ 2 V ( x ) ≥ m I d for any x ∈ R d . Then for any c, a ∈ R + , and any function φ with ∫ φ = 0 ,

where ρ ∝ e -V , { ( λ k , ϕ k ) } ∞ k =1 are eigen pairs for the operator -K , and φ = ∑ ∞ k =1 α k ϕ k .

Remark. Note that as K is an operator, exp ( ct K ) := ∑ ∞ k =0 ( ct K ) k k ! is the operator exponential.

Proof. Let us first denote the eigenvalue decomposition of K as K ( ϕ k ) = -λ k ϕ k where λ k ≥ m for k ∈ N and ⟨ ϕ j , ϕ k ⟩ L 2 ( ρ -1 ) = δ j,k for any j, k ∈ N by Lem. D.1. Then we can decompose φ by φ = ∑ k α k ϕ k . It is not hard to verify that

Then

Hence,

E Proof of Prop. 3.4, Prop. 3.5, and a discussion on the pulse-shape error

E.1 Proof of Prop. 3.4

For convenience, we summarize some notations below; see also Appx. A.

Denote the global minimum of U ← t as X ← t and denote the global minimum of V ← t as Y ← t . When h →∞ , we know

Denote the normalizing constant Z V := ∫ e -V for an arbitrary potential V .

˜ · Recall that p t := e -U t and ρ t ∝ e -V t in (12).

Recall that the distribution of the exact dynamics Y t is q t ≡ p T -t ≡ p ← t and the distribution of the approximated dynamics ˜ Y t is q t .

Lemma E.1. Under Assumption 3.3, we have:

(i) Assumption 3.1 (3) is valid, i.e., the existence of c U .

(ii) For any t ∈ [0 , T ] , the probability distribution ρ t defined in (12) satisfies the Poincaré inequality with constant κ t (16) :

(iii) Z V 0 := ∫ R d e -V 0 is both upper and lower bounded: for any δ ∈ (0 , 1) , there exists h 0 ( δ ) > 0 such that whenever h ≥ h 0 ( δ ) ,

(iv) If we pick δ = 1 / 2 , for h ≥ max { h 0 (1 / 2) , √ max { 0 , -c U ln(2) } , the function ρ 0 /p 0 is uniformly bounded:

and

Proposition E.2. For any h ≥ 1 / 2 , we have

where

Z V ← t = ∫ e -V ← t , M ← t and m ← t are shown in Assumption 3.3, and X ← t and Y ← t are the global minimum points of functions x ↦→ U ← t ( x ) and x ↦→ V ← t ( x ) , respectively.

Remark. A similar result holds for any h > 0 ; we choose h ≥ 1 / 2 in order to simplify the above constants (30).

Note that the constant C ← , (2) t is independent of h . We remark that when h ≫ 1 , C ← , (1) t approximately behaves as follows:

Lemma E.3. When h →∞ , we have

We now proceed to finish the proof of Prop. 3.4. The detailed proofs of Lem. E.1, Prop. E.2, and Lem. E.3 are postponed to Appx. E.2.

Proof of Prop. 3.4. By Lem. E.1,

To simplify notations, let

After taking the time-derivative and using Lemma C.1 (in the first line below), and expressions (11) in the second line below, and integration by parts in the third line, the expression of ρ ← t (12) in the fourth line, we have

The major challenge is to estimate ∫ R d Λ ← t 2 ( ρ ← t ) 2 d d t ρ ← t . By Prop. E.2,

To verify the condition in Prop. E.2, we can readily confirm that

where the third equality in the last equation comes the fact that the Fokker-Planck solution operator Φ ( h ← ) s,t preserves the total mass. Note that we need h 2 -C ← , (2) t ≥ 0 in order to ensure the correct direction when applying the Poincaré inequality above, which explains the lower bound that h needs to satisfy in Prop. 3.4.

By Gröwnwall's inequality, for any t ≥ s ,

Finally, by (31), we have

where

Recall that C ← , (2) t (30) is independent of h . By Lem. E.3, we already discussed that lim h →∞ C ← , (1) t exists. When h →∞ , we know that ρ ← s → p ← s (11)(12), and thus,

This completes the proof of Prop. 3.4.

E.2 Proof of Lem. E.1, Prop. E.2, and Lem. E.3

Proof of Lem. E.1.

(i) From m 0 > 1 in Assumption 3.3, x ↦→ U 0 ( x ) -x 2 / 2 is strongly convex and thus is surely bounded from below by some c U ∈ R .

(ii) The second result is a classical result by Bakry-Emery criterion [2, 3], as V ← t (12) is strongly convex with Hessian lower bound κ ← t (16).

(iii) To prove the third one, note that

To prove the lower bound,

.

Recall from the beginning of this Appendix, X 0 is defined as the global minimum of x ↦→ U 0 ( x ) Then, U 0 ( x ) ≥ U 0 ( X 0 ) for any x and we know h ↦→ e -(1+ h -2 ) ( U 0 ( x ) -U 0 ( X 0 ) ) is monotone increasing. By monotone convergence theorem,

Thus, the limit of the right-hand side of (35) is 1 when h →∞ . Hence, the lower bound of Z V 0 follows immediately. Finally, when h ≥ h 0 ( δ ) ,

(iv) When we pick δ = 1 / 2 and choose h as specified, we can immediately obtain (29).

Before we prove Prop. E.2, we need the following three lemmas.

Lemma E.4. Under Assumption 3.3, we have

Proof. Since p t follows the Fokker-Planck equation ∂ t p t = ∇· ( 1 2 xp t ) + 1 2 △ p t , then U t = -log p t satisfies ∂ t U t ( x ) = -1 2 ( d -x · ∇ U t ( x ) -△ U t ( x ) + |∇ U t ( x ) | 2 ) . Then by Assumption 3.3,

Recall that X t is defined as the global minimum of U t and thus ∇ U t ( X t ) = 0 .

Lemma E.5. When h ≥ 1 2 ,

where

Proof. By direct calculation from the definitions of ρ ← t in (12) and V ← t in (11),

Hence, by Lem. E.4 and h ≥ 1 2 ,

Lemma E.6. Assume that ρ ∝ e -V and φ decays fast enough such that

as the radius R →∞ ( B 0 ( R ) is the ball centered at 0 with radius R ), e.g., φ ∈ C 2 0 ( R d ) . Then

Proof. The identity ∇· ( φ 2 ∇ ρ ) = ∇ φ 2 · ∇ ρ + φ 2 △ ρ and the trivial facts ∇ ρ = -ρ ∇ V , △ ρ = ( |∇ V | 2 -△ V ) ρ show that

Therefore,

Proof of Prop. E.2. The above three lemmas show that

To get the fifth line, we used the definition of Y ← t : Y ← t is the global minimum of V ← t so that ∇ V ← t ( Y ← t ) ≡ 0 . To get the second last line, we used the fact that

Proof of Lem. E.3. Note that ς ← t does not depend on h . When h → ∞ , as V ← t → U ← t (11), we know Z V ← t converges to Z U ← t ≡ 1 by the definition of p t := e -U t , we immediately know that

Moreover, Y t →X t as h →∞ , we have

E.3 Proof of Prop. 3.5

Case I: h = 0 . For this ODE case, (8) becomes

and the formula of leading order term L (0) is

Then

since we perturb the E ← t only when t ≈ T , that is, E ← t ( x ) = I t ∈ [ T -a,T ] E ( x ) for a small positive a . Then we know that

Recall that p 0 = p ← T ≈ p ← t when t ≈ T ; moreover, Φ (0) t,T ≈ Φ (0) T,T ≡ Id. Therefore, the leading order term of L (0) when a ≪ 1 is

Case II: h →∞ . For the SDE case,

where

Then when we further assume h ≫ 1 ,

To apply Lem. D.2 in the second line, we remark that K ← T ( µ ) = △ µ + ∇· ( ∇ V ← T µ ) , V ← T has Hessian lower bound κ ← T ≡ κ 0 (16).

E.4 Discussion on the pulse-shape error

Though the error of the score function is time-dependent, we can always divide the error function into a linear combination of step functions:

where χ is the indicator function and ϕ j is the function value of E ← t on the time interval [ t j , t j +1 ] . Without loss of generality, assume that the time-discretization is uniform, i.e., t j +1 -t j = ∆ t for any j . If we don't worry about ill-behaved functions, this decomposition can be made arbitrarily accurate by choosing a smaller time interval; it is not difficult to make this approximation mathematically rigorous. It is hard to assume that this error function at time e.g., t = 0 . 1 has some subtle connection

with its error at e.g., t = 0 . 8 in general; how the error function exactly looks like depend on a vast amount of hyper-parameters in training. Therefore, we might as well treat each ϕ j as 'mostly independent' in order to handle the worst case situation. If we consider how each ϕ j contributes to the final sample generation error (quantified via KL divergence), then we might as well study each one independently. We remark that this is simply a reasonable assumption in order to treat generic error types.

More specifically, recall that in Prop. 3.2, v T linearly depends on each ϕ j

and recall that the leading order term (10) is simply

Therefore, after plugging the decomposition of E ← t into L ( h ← ) , we have

We had chosen h ← t = h for all t ∈ [0 , T ] for simplicity. Either by Cauchy-Schwartz inequality, or by assumptions on the independence of each ϕ j as discussed above, the main feature is the following term

̸

Therefore, it makes sense to study how this quantity scales for each j , and for two asymptotic regions h = 0 and h → ∞ . For each fixed j , we can easily observe that the above quantity arises from choosing the error function as a pulse-shape error, i.e., ϕ k = 0 if k = j (with certain normalization rescaling).

F Proof of Prop. 3.6 and the discussion of the large diffusion limit

F.1 Proof of Prop. 3.6

Let us pick an a = h -β ≪ 1 . We split v T (10) into two parts:

Upper bound

By Cauchy-Schwartz inequality ( ( x + y ) 2 = x 2 + y 2 +2 xy ≤ (1 + α 2 ) x 2 +(1 + α -2 ) y 2 for any x, y and α > 0 ),

For the second term T 2 ,

which decays exponentially fast as h →∞ as long as h 2 ≫ a -1 ≫ 1 . To get the second line above, we used Cauchy-Schwarz inequality and Fubini's theorem. Recall the definition of γ ∈ R + in the statement of Prop. 3.6.

Overall, when a = h -β ≪ 1 ,

where

Please refer to Lem. E.3 for lim h →∞ C ← , (1) r .

Lower bound

The lower bound can be proved in the same way: by Cauchy-Schwarz inequality again,

The remaining task is to estimate T 1 .

Asymptotic limit of T 1

We only provide an asymptotic result below. As a ≪ 1 ,

where we used Γ ← t ≈ Γ ← T , when t ≈ T in the second line; we used L ( h ← ) t h 2 / 2 ≈ L ( h ← ) T h 2 / 2 when t ≈ T in the third line; L ( h ← ) T = h 2 / 2 K ← T (40) herein. In the last line, we used the fact that ρ 0 ∼ p 0 when h →∞ .

By Lem. D.2,

Therefore, in the limit,

where ( λ ( h ) k , ϕ ( h ) k ) are eigen pairs of K ← T and Γ ← T = ∑ ∞ k =1 α ( h ) k ϕ ( h ) k . When h →∞ , we know that the eigenvalues of K ← T , which depends on h and has the form

should converge to K ∞ , defined as

where ( λ ( ∞ ) k , ϕ ( ∞ ) k ) are eigen pairs of K ∞ and Γ ← T = ∑ ∞ k =1 α ( ∞ ) k ϕ ( ∞ ) k .

Upper bound of T 1

By the upper bound in Lem. D.2 (together with a = h -β and h ≫ 1 ) or by applying λ ( h ) k ≥ κ 0 directly to the asymptotic of T 1 , when h ≫ 1 ,

The term T in Prop. 3.6 is simply the limit of T 1 .

F.2 Remark on the large diffusion limit

Score function parameterization in constrained score models

We would like to remark on the parameterization of score function. It is common in literature to directly parameterize S t (4) via some neural network. However, in general, this practice can

rarely guarantee that S t (which is supposed to approximate ∇ log p t ) has the gradient form as well [23, 25]. If we instead parameterize log p t ( x ) ≈ N t ( x ; θ ) , where N · ( · ; θ ) is some (scalar-valued) neural network with parameter θ , we can simply use the following ansatz in training

Consequently, the error of score function estimate also has a gradient form:

This validates the discussion in § 3.6. Apart from the apparent benefit in preserving the gradient form, this ansatz can help us identify some interesting structure for the the upper bound of lim h →∞ L ( h ) in (18), discussed in § 3.6.

More discussion on the upper bound

Because the score error E ← T comes from approximating ∇ log p 0 , it is reasonable to consider the case φ = log p 0 , the upper bound (18), or equivalently (19), becomes

This quantity heuristically characterizes how difficult the probability distribution p 0 can be learn by score-based diffusion models with large diffusion coefficient.

It is of interest to further explore how to utilize the above upper bound to improve the training of score function, or provide theoretical understanding about what types of probability distributions are easy/difficult to learn via score-based SDEs. We shall leave an extensive study of these important and interesting questions for future work. Below, we shall provide a simple example.

Example: d -dimensional Gaussian

When p 0 = e -U 0 = N ( µ, σ 2 0 ) , the quantity m 0 (14) is m 0 = 1 σ 2 0 . Then this upper bound (18) is

This is compatible with the insight that it is more difficult to learn a high dimensional probability distribution.

G Numerical experiment: 1D Gaussian case

We demonstrate and validate main findings via a simple 1D Gaussian. Suppose f t ( x ) = -1 2 x , g t = 1 , and the exact data distribution is a Gaussian p 0 = N (0 , σ 2 0 ) . The error is quantified by KL ( p 0 || ˜ q T ) . We investigate this error as a function of the magnitude of h ← (which is chosen as a constant function h ← t = h for any t ∈ [0 , T ] ).

G.1 Explicit formulas for 1D Gaussian case

From solving the SDE for the forward process, we know that

The backward dynamics (7) on t ∈ [0 , T ] is

In this example, there is only one source of error, which is E ← , as we know explicitly the distribution p T and we can choose the time step small enough such that numerical discretization error is negligible. Due to the structure of the score function, it is reasonable to consider the following ansatz

This example has explicit formulas: ˜ Y T is a Gaussian distribution with E [ ˜ Y T ] = 0 and

where

The sample generation error quantified by KL divergence also has an explicit formula:

G.2 Experiment 1: Fix error magnitude ϵ and consider various error types

We choose T = 2 , h ← t = h for all t ∈ [0 , T ] , and the error function is chosen as

For these choices, we only perturb the true score function by a bounded prefactor. We can observe that the error KL ( p 0 || ˜ q T ) is a complicated function of h in general in Fig. 6. When we only perturb the score function during the initial period of the generative process (case 4), we can clearly observe that the sampling error decays exponentially fast with respect to h 2 , which numerically validates Prop. 3.4. When we only perturb the score function near the end of the generative process (case 5), increasing the diffusion coefficient h will actually increase the error, which is predicted by Prop. 3.5. For a general error (case 1, 2, 3), overall we can still expect that increasing diffusion coefficient h will generally suppress the error when σ 0 is small.

Table 3: Approximated value of L ( h ) when h 2 ≈ 20 . We can observe that in each column (referring to a specific error type) the value is almost independent of the σ 0 in particular when σ 0 < 1 .

G.3 Experiment 2: consider L ( h ) for various error types

We further numerically approximate L ( h ) ≡ L ( h , E ← , p 0 ) by linear regression and study how σ 0 (i.e., the data distribution p 0 ), error type E ← , and diffusion coefficient h affect the leading order term L ( h , E ← , p 0 ) . Recall that we use L ( h ) as a simplified notation when E ← and p 0 are clear from context; see § 3. As a remark, to approximate L ( h ) , we used the leading-order approximation that KL ( p 0 || ˜ q T ) = L ( h ) ϵ 2 + O ( ϵ 3 ) : we choose a few ϵ values and use linear regression to estimate L ( h ) .

In Fig. 7, we can clearly observe that L ( h ) converges to a constant extremely fast when h increases for cases σ 0 < 1 . The value of E ← T for the case 3 is only 1 / 2 of that for case 1 and case 2. By Prop. 3.6, we know that lim h →∞ L ( h ) for the case 3 should be approximated 1 / 4 of that for cases 1 and 2. This is also (approximately) numerically observed in Table 3.

σ0 =1.5

σ0 =3.0

0

20

-3.6

-3.0

σ0 =0.2

case 1

0

20

-4.0

-3.5

-3.0

case 2

0

20

-4.5

-4.0

-3.5

case 3

0

2

-6

-4

case 4

0

20

-4.8

-4.4

-4.0

case 5

0

20

-4.0

-3.5

-3.0

σ0 =0.4

0

20

-4.0

-3.6

-3.2

0

20

-4.4

-4.0

-3.6

0

5

-6

-4

0

20

-5.6

-4.8

-4.0

0

20

-4.0

-3.6

-3.2

σ0 =0.6

0

20

-3.6

-3.2

0

20

-4.4

-4.0

0

5

-5

-4

0

20

-6

-5

0

20

-3.9

-3.6

-3.3

σ0 =0.8

0

20

-3.9

-3.6

-3.3

0

20

-4.8

-4.4

-4.0

0

5

-4.8

-4.0

0

20

-7

-6

-5

0

20

-3.90

-3.75

0

20

-3.90

-3.75

0

20

-4.8

-4.4

0

5

-4.25

-4.00

-3.75

0

20

-7.2

-6.4

-5.6

0

20

-4.0

-3.9

-3.8

σ0 =2.0

0

20

-4.0

-3.9

-3.8

0

20

-5.2

-4.8

-4.4

0

5

-4.1

-4.0

-3.9

10

20

-7.0

-6.5

-6.0

0

20

-4.50

-4.25

-4.00

0

20

-4.50

-4.25

-4.00

0

20

-5.1

-4.8

0

5

-4.50

-4.25

-4.00

5

10

15

-7.2

-7.0

-6.8

H More details about numerical experiments in § 4

In this section, we discuss datasets, network architectures, evaluation metrics, numerical schemes (exponential integrator), and the default weight in denoising score matching.

H.1 Datasets

1D 2-mode Gaussian mixture : p 0 ( x ) = ∑ 2 i =1 0 . 5 N ( x ; ( -1 . 0) i , 0 . 01 ) .

2D 4-mode Gaussian mixture : p 0 ( x ) = ∑ 2 i,j =1 0 . 25 N ( x ; (( -1 . 0) i , ( -1 . 0) j ) , 0 . 05 2 I 2 ) .

Swiss roll : Swiss roll generates samples by ( x, y ) = ( t sin( t ) , t cos( t ) ) with t drawn from the uniform distribution U ( 3 π 2 , 9 π 2 ) .

0

10

-0.6

0.0

0.6

σ0 =0.2

case 1

0

10

-0.5

0.0

0.5

case 2

0

10

-1.0

-0.5

0.0

case 3

0

10

-0.5

0.0

0.5

σ0 =0.4

0

10

-0.4

0.0

0.4

0

10

-1.2

-0.8

-0.4

0

10

-0.4

0.0

0.4

σ0 =0.6

0

10

-0.3

0.0

0.3

0

10

-1.2

-0.8

-0.4

0

10

-0.6

-0.3

0.0

σ0 =0.8

0

10

-0.50

-0.25

0.00

0

10

-1.2

-0.8

0

10

-0.60

-0.45

-0.30

σ0 =1.5

0

10

-0.5

-0.4

-0.3

0

10

-1.6

-1.2

-0.8

0

10

-0.6

-0.5

-0.4

σ0 =2.0

0

10

-0.60

-0.45

0

10

-1.6

-1.2

0

10

-1.00

-0.75

-0.50

σ0 =3.0

0

10

-1.00

-0.75

-0.50

0

10

-1.8

-1.5

-1.2

MNIST : MNIST [34] contains 60,000 28 × 28 gray-scale images with hand-written digits.

CIFAR-10 : CIFAR-10 [22] contains 60,000 32 × 32 RGB images with ten categories.

H.2 Network architectures and other parameters

For experiments on 1D/2D Gaussian mixtures, the exact scores can be obtained analytically if we use VP-SDE. We set T = 4 and g t = 1 for t ∈ [0 , T ] . For the time discretization when solving the reverse SDE with Euler-Maruyama method, we apply 40,000 steps and 80,000 steps for 1D and 2D Gaussian mixtures, respectively.

For experiments on Swiss roll, we apply a three-layer neural network for score matching, where the width of each layer is set as 50 , 50 , and 2 and we apply ReLU as the nonlinear activation for two hidden layers. We set T = 1 and g t = √ 0 . 1(1 -t ) + 20 t for t ∈ [0 , T ] . The learning rate is set as 0.01 and decays by 0 . 5 every 8,000 steps. The batch size is set as 400. We train the neural network for 20,000 steps. For the time discretization when solving the reverse SDE with Euler-Maruyama method, we apply 20,000 steps.

For experiments on MNIST, we apply the net architecture in [17] for score matching, where we use two resolution blocks in U-net and set the multipliers of channels to be one and two. We set T = 1 . 4 and g t = √ 0 . 1(1 -t ) + 20 t for t ∈ [0 , T ] . The number of iteration is 20 , 000 , the batch size is set as 64 . We solve the reverse SDE with exponential integrator; see also Appx. H.4.

For experiments on CIFAR-10, we apply the DDPM++ cont. in [30] as the net architecture, and use their pretrained checkpoint in for score estimation. We use the same setting as [30], i.e., T = 1 and g t = √ 0 . 1(1 -t ) + 20 t for t ∈ [0 , T ] . For the time discretization when sampling with EulerMaruyama method, we apply 100, 200, 500, 1000, 2000, 3000 and 4000 steps.

H.3 Evaluation metrics

For experiments of 1D/2D Gaussian mixtures and Swiss roll, we apply approximated divergences for evaluating the performances. Specifically, we discretize the space into 100 bins in each dimension, then obtain the empirical densities of 10,000 true samples and 10,000 generated samples, and use Jensen-Shannon divergence, Kullback-Leibler divergence and Wasserstein distance between both empirical densities as the metrics for evaluation.

H.4 Exponential integrator

We shall explain the exponential integrator for (2) with t ∈ [0 , T ] , i.e., the following equation

Then for any time t ∈ [ t k , t k +1 ] , and given ˆ Y t k , we approximate the above dynamics by

This dynamics is a linear SDE and we can solve it exactly

Therefore,

If we pick

then after straightforward calculations,

where δ k := t k +1 -t k and γ k := exp ( δ k ( 2 β 0 +(2 t k -2 T + δ k )( β 0 -β 1 ) ) 4 ) .

H.5 Default training weight

Given the fixed X 0 , we can explicitly solve (1) (with the choice f t ( x ) = -g 2 t 2 x ):

where G t = exp ( ∫ t 0 g 2 s 2 d s ) . The standard deviation of the Brownian motion term above is

When g t = √ β 0 +( β 1 -β 0 ) t as used in many literature [30, 17], we can explicitly solve for ϖ t :

In literatures, to balance the noise over time in training the score function, a common practice is to use the default weight ω t = ϖ 2 t (5) in the (denoising) score-matching loss function (4).

H.6 Additional numerical results for experiments in § 4

We present more numerical results on 1D Gaussian mixture, Swiss roll, and MNIST to further verify theoretical results.

1D Gaussian mixture

We evaluate the performances of generative models under different values of h for 1D Gaussian mixture, and present the visualization and numerical results in Fig. 8 and Fig. 9, respectively. In Fig. 8, a clear trend shows that with increasing h , the empirical density of generated samples better matches the true density function. This trend is more quantitatively captured in Fig. 9, from which we clearly observe that the distance between the empirical and the true density function decreases to the numerical threshold exponentially fast. Numerical threshold means the error of various distances when ϵ = 0 ; due to the space discretization when computing various distances, the numerical values of various distances are not exactly zero even when we use the exact score function. However, increasing h can help us to almost reach this limit and this phenomenon is theoretically described in Prop. 3.4.

-2

-1

0

1

2

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=0.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=1.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=2.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=3.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=4.0 g 2

(a)

ϵ

=

-

0

.

2

.

-1

0

1

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=0.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=1.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=2.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=3.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=4.0 g 2

(b)

ϵ

=

-

0

.

1

.

-1.5

-1.0

-0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=0.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=1.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=2.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=3.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=4.0 g 2

(c)

ϵ

= 0

.

0

.

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

2.5

h 2

=0.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

h 2

=1.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

h 2

=2.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=3.0 g 2

-1.5

-1.0

-0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=4.0 g 2

(d)

ϵ

= 0

.

1

.

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

h 2

=0.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

h 2

=1.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

h 2

=2.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.0

0.5

1.0

1.5

2.0

h 2

=3.0 g 2

-1.0

-0.5

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

h 2

=4.0 g 2

(e)

ϵ

= 0

.

2

.

KL divergence

2

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

h 2

/g

2

10 -3

10 -2

10 -1

KL divergence

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(a)

E

t

=

∇

log

p

t

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

h 2

/g

2

10 -3

10 -2

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(b)

E

t

=

1+sin(2

πt/T

)

∇

log

p

t

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

h 2

/g

2

10 -3

10 -2

KL divergence

ε = - 0.2

ε = - 0.1

ε = 0.0

ε = 0.1

ε = 0.2

(c)

E

t

=

I

t<

0

.

95

T

∇

log

p

t

Swiss roll

In Fig. 10, we provide additional figures to discuss the effect of time-discretization. When the numerical error is negligible, we can observe that the generative process with a larger h can provide a clearer picture of Swiss roll, as shown in Fig. 10c. However, when the discretization error cannot be ignored, the conclusion may be reversed. Therefore, it is necessary to design and employ more accurate numerical methods for models with large h in order to fully benefit from diffusion models with a large diffusion coefficient.

In Fig. 11, we display JS and KL divergences between the true density p 0 and the generated samples. Since the data distribution of Swiss roll is highly localized (data is concentrated on a curve embed in 2D), accurately computing the KL divergence poses a significant numerical challenge. That is why we use Wasserstein distance instead in Fig. 5a. Based on more robust symmetric metrics (i.e., the JS and Wasserstein distance herein), we can observe that a larger h can indeed diminish the error in sample generation.

(a) discretization steps = 100

(b) discretization steps = 1,000

h2 = 0.0 g2

h2 = 1.0 g2

h2 = 2.0 g2

h2 = 3.0 g2

h2 = 4.0 g2

O

h2 = 0.0 g2

h2 = 1.0 g2

h2 = 2.0 g²

h2 = 3.0 g2

h2 = 4.0 g2

h2 = 0.0 g2

h2 = 1.0 g2

h2 = 2.0 g²

h2 = 3.0 g2

h2 = 4.0 g2

(c) discretization steps = 10,000

0

1

2

3

4

h 2

/g

2

4.5 ×10 -1

5.0 ×10 -1

JS divergence

0

1

2

3

4

h 2

/g

2

6.0 ×10 -2

8.0 ×10 -2

KL divergence

MNIST

We conduct further experiments to explore the effect of h ← and the time discretization steps. It is important to note that if one has an extremely well trained score function, then the effect of h ← is indeed negligible, as shown in (3). To somewhat magnify the score training error for MNIST (but with a reasonable score function), we increase the time T and use a smaller architecture with fewer parameters; the reason of the occasional failure to generate clear MNIST images in later figures comes from this deliberate experimental design .

For an existing pre-trained score function with non-negligible error, we notice that increasing h ← can almost ubiquitously improve the quality of sample generation; see Fig. 12 (as well as Fig. 15 and Fig. 16 under different training setup). This improvement is supported by our theoretical results, particularly Prop. 3.4. Notably, even when choosing h ← = 0 (ODE) and h ← = g ← (the default diffusion model in many studies) occasionally fail to generate an image, simply by increasing the magnitude of h ← (possibly at the cost of more computational resources), we have a larger chance of generating an image with reasonable quality; for instance, see the last row in Fig. 12 (particularly see the last row in Fig. 15). The ODE-based model sometimes fails to generate hand-writing digits even when the step number is 10 3 , whereas the diffusion model (with large h ← ) does not encounter this issue. This conclusion might be reversed when the time step size is large, which is similarly observed in the Swiss roll.

The computational cost of SDE-based models with large diffusion ( h ← > g ← ) is relatively high due to the necessity of a larger number of time steps: a larger h ← has the similar effect as running Langevin for a larger time horizon as discussed in § 3.3 and a longer time simulation is expected to be more expensive and also its accuracy largely relies on a well-chosen time step. However, this can be offset by the ability to use a lightweight architecture that possibly speeds up the generative process. A detailed comparison of various diffusion-based models with the same computational budget constraint is challenging and is slightly beyond the scope of this work, and we will leave this task to future work.

weight type=default, trial=1

step=30

h 2

/ g 2 =0.0

h 2

/ g 2 =0.5

h 2

/ g 2 =1.0

h 2

/ g 2 =1.5

h 2

/ g 2 =2.0

step=60

step=100

step=200

step=1000

4

8

3

37

6

9

3

9

E

S

9

2

B

9

9

2

4

9

y

3

2

6

3

D

0

9

9

2

8

9

E

B

B

2

2

80

2

6

6

691

2&8

8233381

9203aa

39309484

I Numerical experiments for adopting different weight in training

We have two reasons to explore the effects of various weight functions in training.

The first reason is that theoretically, any positive scalar-valued functions ω t on (0 , T ) is a valid candidate. This prompts the question of whether such a default weight function (5) is optimal in designing the loss function. We fully acknowledge that the default choice adopted in most literatures is a very effective one. However, the mathematical reason behind it is still not satisfactory in our opinion. This motivates us to ask whether the default choice is really the optimal one, at least in certain circumstances.

The second (and actually the primary reason) comes from our theoretical predictions discussed in § 3.7. As there is a tight connection between the optimal reverse-time generative process and the time-distribution of the score error (see Prop. 3.4, Prop. 3.5 and Prop. 3.6), if we are willing to train or re-train the score function and are interested in using the ODE-based model (for fast sample generation), it appears that we should focus more on the noise's end comparatively. To achieve this goal, namely, to control the distribution of score error, we adopt different weight schemes in the loss function only for training: default weight in (5), a data-driven case (more weight in the data side) and a noise-driven case (more weight in the noise side):

There is no theoretical reasons behind the noise-driven and data-driven choices in the last equation. We merely experiment with two reasonable choices and at the same time they are expected to help us control the time distribution of the score error.

After the initial camera-ready version of this work, we notice that such a weight design (in particular the noise-driven weight) has also been studied in [12]. Their experiments seem to provide further evidence about the validity of our arguments. However, we would like to emphasize that the approaches and perspectives towards proposing such a weight design are different: in [12], their conjecture is based on an insightful perception argument, whereas we take a mathematical approach and come up with the design based on the above error accumulation analysis.

I.1 Our guess

Based on our theoretical results, we guess that if a different weight function can really achieve our expected goal (namely, control the time distribution of the score function), the noise-driven case should be more suitable for ODE models, and the data-driven weight should give us the worst performance for ODE models.

We remark that this conjecture is surely not universal , and its validity remains to be fully validated by more benchmark experiments. Nevertheless, our numerical experiments below suggest its potential usefulness and it prompts an interesting question to explore and design the optimal score-matching loss function, which is rarely studied in literature. In what follows, we report numerical experiments for MNIST and CIFAR-10.

I.2 MNIST

We used different weights to train the score function and then visualize their generated samples in Fig. 13. We can clearly observe that the score function trained by the noise-driven weight produces comparatively better samples in ODE models. We plot the denoising score-matching loss for score functions obtained from training with various weights in Fig. 14:

to make the comparison more straightforward, we visualize the time-distribution of relative score-matching loss rather than the absolute value, namely, we demonstrate:

and S ( i ) refers to the score function obtained from training using weight i and we use test datasets to approximately represent p 0 ;

to ensure robustness, we independently conduct two trials with different neural network initializations: For each trial, the initial neural network is the same and the only difference in training is to adopt different weights in the loss function.

In Fig. 14, we can clearly observe that adopting different weights indeed help us to control how the score error is distributed over time as we expect (e.g., noise-driven weight helps us reduce the error near t ≈ T and has an opposite effect near t ≈ 0 ); as shown in Fig. 13, their numerical performances in terms of sample generation also match our guess above.

We further visualize how the diffusion coefficient h and time step size affect the sample generation quality for score functions obtained by data-driven weight (see Fig. 15) and noise-driven weight (see Fig. 16). The conclusion is the same as in the default weight case. This further validates our theoretical results in § 3.

weight type = data

weight type = default

weight type = noise

(a) Trial number

1

weight type = data

weight type = default

weight type = noise

BJ8430

4

S

2

Z

7

R

？

8

5.83947

0

87

6

八

8

5

90

?

b

3

7

S

7

b

4

S

8

S

6

C

8

S

S

a

b

8

子

3

8

4

Y

L 9

9

1

Y

T

人

9

9

94

8

6

3

q

2

2

8

3

5

3

B

B

3

9

9

D

7

9

6915

3

5

6

5

5

h

2

9

5

C

A

U

6

5

9

C

D

a

8

6

9

3

5

B

S

3

B

S

r

3

9

9

9

k

7

9

6

3

8

9？

96720

了

q

2

D

3

2

Y

K

5

y

5

7

8

6

?

B

T

S

3

7

2

9

C

6

h

9

9

9

S

4

Cl

b

S

9

A

3

A

2

3

3

3

6

Q

2

9

3

Q

M

(b) Trial number

2

Relative score-matching loss

t/T

0.0

0.2

0.4

0.6

0.8

1.0

0.6

0.8

1.0

1.2

1.4

1.6

data, trial 1

noise, trial 1

data, trial 2

noise, trial 2

reference

weight type=data, trial=1

step=30

h 2

/ g 2 =0.0

h 2

/ g 2 =0.5

h 2

/ g 2 =1.0

h 2

/ g 2 =1.5

h 2

/ g 2 =2.0

step=60

step=100

step=200

step=1000

3

1.87

weight type=noise, trial=1

4

step=30

h 2

/ g 2 =0.0

h 2

/ g 2 =0.5

h 2

/ g 2 =1.0

h 2

/ g 2 =1.5

h 2

/ g 2 =2.0

step=60

step=100

step=200

step=1000

8

S

6

3

3

S

3

8

8

S

8

9

7

Y

9

5

2

5

2

T

4

8

3

G

9

B

S

59G

6

3

乡

38388

8

5

8

8

6

71

7

2

D

9

5

7

B

9

3

3

H

了

2

8

S

7

S

b

4

?

2

D

3

d

2

6

9

3

3

3

8

3

2

3

D

S

U

50G

6

8b

183166

Z

3

9

9

9

8

9

3

S

3

5

Q

5

8

9

4

G

4

4

D

4

4

3

7

b

女

4

Q

3

Q.

6

6

6

9

8

S

S

3

Y

3

B

a

9

6

9

6

9

4

a

3

3

6

94

C

?

9

6

4

力

D

于

Y.

T

人

9

C

4

C

5

4

e

5

7

e

=

5

4

8

7

7

7

08

3

9

6

373

6

2

0095

6

7

0

60Q

S

6

2

002

59G

63-155

x2083±3387

8233a239M

9333a8961

39589

I.3 CIFAR-10

We carry out a similar experiment for CIFAR-10 to test the effect of training weights ω t : we used the same initialization (referred to as the trial number below) and all other hyper-parameters, except that we employ different weights in score-matching loss (4). The same architecture is used as in [17] for CIFAR-10 and the detailed hyper-parameters can be found in source codes.

In Fig. 17, we observe that overall over the whole training period, the noise-driven weight leads into a score function estimate no worsen than that by the default weight: due to stochastic fluctuations and other uncertainties (in particular if we adopt the mixed precision training), there is no guarantee that the noise-driven one is always better, but the overall tendency is still observable and clear. In Fig. 18, the noise-driven one actually has a slightly larger score-matching loss (SML) than the default one (which could be possibly explained by how we measure the SML in Fig. 18). What is interesting is that score functions trained by the noise-driven weight and the data-driven weight have a similar SML, which both decay at the similar pace; however, the FID values for the score function estimate by noise-driven weight are much smaller than that by the data-driven weight. This apparent gap clearly explains that apart from the total score-matching loss (which does matter), the time distribution of the score error plays an important role in determining the final sampling error, echoing our theoretical results in § 3.

For instance, in Fig. 17, if we consider the first experiment (i.e., trial=0) with float16 mixed-precision training (i.e., mixed=True), we notice that relatively near 80k and 120k training iterations, the performance of noise-driven one is much better than the default one, which is consistent with Fig. 19 that the relative loss near t ≈ T is more minimised for iterations 80k and 120k, compared with other iteration stages. Moreover, for the same experiment in Fig. 17, the data-driven one has a much worsen FID value at iteration 200k, which is compatible with the increasing relative error near the noise's end (i.e., t ≈ T ) in the last row of Fig. 19. For the remaining three experimental setup (either different initialization or training precision), we notice a similar consistency between how the time-distribution of the SML behaves and how FID values change . This relation, so far, still cannot be used as a rigorous quantitative indicator to predict one based on the other quantity, but qualitatively, the above explained relationship does appear to be numerically valid and theoretically sound.

In summary , if we have two score functions S 1 and S 2 from training:

If the SML for S 1 is much larger than the SML for S 2 , then we can probably confidently expect that S 2 is a more accurate estimate.

However, when the total SML (4) for both are close, then the time-distribution of the scorematching loss together with which generative dynamics is chosen will play a significant role in determining the final sample generation quality, which is probably largely overlooked in current literature as far as we know. A full investigation and in particular whether it is possible to adapt this observation to achieve the state-of-art models will be left to the next stage of research.

40k

80k

120k

160k

200k

10 2

2×10 1

3×10 1

4×10 1

6×10 1

FID

trial=0, mixed=True

40k

80k

120k

160k

200k

10 2

2×10 1

3×10 1

4×10 1

6×10 1

FID

trial=0, mixed=False

40k

80k

120k

160k

200k

3×10 1

4×10 1

6×10 1

FID

trial=1, mixed=True

40k

80k

120k

160k

200k

2×10 1

3×10 1

4×10 1

6×10 1

FID

trial=1, mixed=False

data

noise

default

40k

80k

120k

160k

200k

74

75

76

77

SML

trial=0, mixed=True

40k

80k

120k

160k

200k

74

75

76

77

SML

trial=0, mixed=False

40k

80k

120k

160k

200k

74

75

76

77

SML

trial=1, mixed=True

40k

80k

120k

160k

200k

74

75

76

77

SML

trial=1, mixed=False

data

noise

default

trial=0, mixed=True

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 40k

Relative loss

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

Loss

noise

data

default

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 80k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 120k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 160k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

t/T

0.5

1.0

1.5

2.0

train step = 200k

0.00

0.25

0.50

0.75

1.00

t/T

10 0

10 1

10 2

trial=0, mixed=False

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 40k

Relative loss

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

Loss

noise

data

default

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 80k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 120k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 160k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

t/T

0.5

1.0

1.5

2.0

train step = 200k

0.00

0.25

0.50

0.75

1.00

t/T

10 0

10 1

10 2

trial=1, mixed=True

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 40k

Relative loss

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

Loss

noise

data

default

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 80k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 120k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 160k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

t/T

0.5

1.0

1.5

2.0

train step = 200k

0.00

0.25

0.50

0.75

1.00

t/T

10 0

10 1

10 2

trial=1, mixed=False

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 40k

Relative loss

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

Loss

noise

data

default

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 80k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 120k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

0.5

1.0

1.5

2.0

train step = 160k

0.00

0.25

0.50

0.75

1.00

10 0

10 1

10 2

0.00

0.25

0.50

0.75

1.00

t/T

0.5

1.0

1.5

2.0

train step = 200k

0.00

0.25

0.50

0.75

1.00

t/T

10 0

10 1

10 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies diffusion models, which comprise a class of generative models based on stochastic differential equations. In diffusion models, data is first transformed into Gaussian noise via a stochastic differential equation (usually an Ornstein-Uhlenbeck process), and then a backward process, which turns noise into synthetic data, is learned via the score function. In most works, the diffusion coefficient of the backward process is either chosen to be zero, or to be a constant matching the diffusion term of the forward. The first case is an ODE implementation, while the second case is an SDE corresponding to the exact time reversal of the forward process. However, any function of time can be chosen for the diffusion coefficient of the backward process, and in this work the authors explore the consequences of different choices of diffusion coefficient. In particular, they ask how the optimal choice of diffusion coefficient depends on the error induced by the estimated score function.\n\nThe main results of the paper are as follows. Let $h$ denote the diffusion coefficient of the backward process. Assume that the score function estimator has the for $s_t(x)= \\nabla \\log p_t(x) + \\epsilon E_t(x)$, where the first term is the exact score function and the function $E_t(x)$ is a bounded error function. Let $\\hat{q}_T$ denote the law of the backward process at time $T$ with $s_t(x)$ in the place of the true score function. Then under certain assumptions on $p_0$, it holds that $$ KL(p_0 \\parallel \\hat{q}_T = L(h) \\epsilon^2 + O(\\epsilon^3),$$ where the leading order term $L(h)$ depends on $h$, $p_0$, $E_t(x)$ and $T$. The leading order term $L(h)$ is computed explicitly, which allows the authors to analyze several specific cases when $h$ is constant. In particular, they show that if the error $E_t(x)$ is time-localized (i.e., only nonzero at $t=s$ for some $s > 0$) in the middle of the time interval, the leading order term $L(h)$ decays exponentially fast to zero as $h \\rightarrow \\infty$. This suggests that the SDE implementation can significantly outperform the ODE implementation in some cases. They also show that for certain forms of the error function, the opposite is true: the ODE implementation has better KL divergence error than the SDE implementation.\n\n\n\n--------------\nAfter rebuttal: Thank the authors for the clarifications and comments, which addressed my earlier concerns. I will keep my rating unchanged.\n\nSTRENGTHS:\nOverall, the paper presents a novel and interesting question with some practical implications. The question of choosing the diffusion coefficient in the backward SDE seems largely unstudied because, as the authors mention, the ODE and reverse-time SDE approaches are standard conventions. The fact that they show that different choices of the diffusion coefficient can lead to vastly different performances suggests that the problem is worth studying further. \n\nThe authors provide a nice explicit characterization of the leading order term in the KL divergence error, the derivation of which requires a lot of work. Namely, they show that as $h \\rightarrow \\infty$, the term $L(h)$ converges exponentially fast to a constant $\\Tau$ which is upper bounded by $$ \\Tau \\lesssim \\frac{1}{2m_0^2} \\int_{\\mathbb{R}^d} \\frac{(\\nabla \\cdot (p_0 E_t))^2}{p_0} dx.$$ Here, $m_0$ is the second moment of $p_0$. This suggests the quantity on the right-hand side above plays a significant role in the error analysis of diffusion models, and thus it could be insightful to understand it for different classes of probability distributions.\n\nThe authors also leave a lot of room for interesting future work. Most interestingly, the authors pose the question of how to infer the optimal choice of $h$ from data, which could lead to a lot of subsequent research.\n\nWEAKNESSES:\nOne weakness of the paper is that the upper bound on the constant $\\Tau$ is not further explained through discussion or examples. It seems like the upper bound plays a big role in bounding the KL divergence error for diffusion models. It would be very interesting to discuss explicit examples of distributions $p_0$ where the upper bound on $\\Tau$ can be explicitly computed or controlled, other than Gaussian distributions. This could be mentioned as a direction for future work. It would also be helpful to describe intuitively what the bound represents about a distribution (what does a large/small $\\Tau$ say about a distribution?).\n\nIn addition, the numerical experiment on the 2D 4-mode Gaussian mixture could benefit from additional explanation. It doesn’t seem like a machine learning problem, since you already know the score of the distribution from which you are trying to sample. Perhaps the point is simply to illustrate the convergence of $L(h)$ to a constant as $h \\rightarrow \\infty$, in which case the purpose of the experiment should be stated more clearly.\n\nQUESTIONS:\n1.\tCan you explain in what setting, if any, one would ever encounter a score estimator where the error term was localized in time? Or was the purpose of this choice of error term simply to illustrate that different choices of diffusion coefficient can lead to very different performance? It seems like a very strong assumption that would never be realized in practice.\n2.\tIt would be good to state in the main result any dependence on the sub-Gaussian constant $c_U$. As it is stated in the paper, one often works with a mollified version of the empirical distribution, which leads to a tradeoff between bias from smoothing and faster convergence for learning smoother distributions. Thus, rather than treat the constant $c_U$ as hidden, it could be beneficial to discuss how it balances with other terms.\n3.\tIn this paper, only constant choices of the diffusion term $h$ are discussed. Is this just for simplicity? Could it be of interest to consider certain parameterizations of nonconstant functions for $h$? If so, this could be mentioned for future work.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors try to understand the impact of noise in the reverse process of diffusion models in the presence of an approximate score network. In particular, they look at how the Kullback-Leibler divergence between the true data distribution and the denoised distribution evolves w.r.t. the diffusion term $h$ of the backward (generative) process, with an asymptotic expansion of this KL divergence and studying the first non-zero term.\nThey then show that if the score has a pulse (delta) error early on in the generative sampling, higher values of $h$ are able to recover the density trajectory better, whilst with and error in the late stage of the denoising process it's the opposite.\nThey empirically back these results on synthetic experiments (With heuristic errors) and on the swiss-roll and MNIST dataset with trained score network.\n\nSTRENGTHS:\n- I believe this paper tackle a really important question, which is the role of noise in the generative process. In particular with the success of flow matching which shows that continuous normalising flows, which are (deterministic) ODE based generative models, can be trained akin to denoising score matching models.\n- The analysis seems sounds and rigorous, and the assumptions reasonable.\n- The take home message (if I understood correctly) is that apart if the score error is concentrated at the end, the higher the level of noise the better (with infinite computational budget to actually discretise the backward process). \n- They link they theoretical findings with empirical results, both on simple settings which is nice to unambiguously be able to conclude, and more realistic settings when the score network is trained.\n\nWEAKNESSES:\n- No major weakness, but some of the interpretation of the theoretical results could perhaps be enhanced (cf following questions).\n- The fact that higher values of $h$ requires more discretisation steps is something that should be further stressed I believe.\n\nQUESTIONS:\n- Eq 5: missing the score? Or this is the process of the error?\n- Section 3.3: typo in title 'Placn'\n- Equation 9: This does not assuming $h \\gg 0$ right? Worth moving it above in Section 3.2 to avoid any ambiguity perhaps?\n- line 194: 'By convection [...]' what does this mean? Doesn't this simply come from the definition of the time-reversal?\n- line 196-200: So when $h \\rightarrow \\infty $ not only $V_t \\rightarrow U_t$ but because of the prefactor $h_t^2/2$ in Eq 9 (left) the distribution $\\mu_t$ converges extremely fast to to $\\mu_0$? Not sure to know what '“almost quasi-static” thermodynamics' refers to. Likely worth expanding a bit more on this in the main paper?\n- Can this be seen trough the perspective of Langevin dynamics as a corrector, which would 'project' $\\rho_t$ back to $p_t$ at $t>s$?\n- line 233: '$L(h)$ will decay to zero exponentially fast' with $h$?\n- Section 3.5: Here the setting in that the error accumulates over time in contract with the 'pulse' of Section 3.4. The results of Section 3.5 are that with error at the end of the generative process the smaller $h$ the better, yet if the pulse is also near the end, wouldn't large $h$ help (as the larger $h$ the faster it would correct this er§ror?).\n- MNIST: This section is not really self-contained as the weighting functions are in the appendix. What is the motivation for this cubic 'noise' weighting?\n- appendix: 'The computational cost of SDE-based models with large diffusion (h > g) is relatively high due to the necessity of a larger number of time steps.' What is the reason for this? That is quite key and worthy of being further developed (in the main text).\n\nLIMITATIONS:\n- Figure 4: It shows that indeed higher values of $h^2/g^2$ can lead to smaller error yet it also requires a lot more step. At $10^3$ steps, which is quite a reasonable number of steps already, the probability flow ODE ($h=0$) perhaps best and the curves starts overlapping around ~2000-3000 steps. So although the theoretical conclusion is that larger values of $h^2/g^2$ is better, for practical implementations there is a trade-off between larger number of discretisation steps (with larger values of $h$) and larger score network architecture.\n- The authors look at different types of errors on the score network, both theoretically with pulse-shape error to simplify the analysis, but also experimentally with simple errors (weighting the true score for some time range) which makes it easier to interpret. It would be interesting to look at the empirical error of trained score network (e.g. on image dataset), depending on the weighting $w(t)$ in the loss, but also the scheduling $g(t)$, and in particular its (average) evolution through the denoising time. Is the error spread uniformly (wrt time)? Writing the score as $\\epsilon / \\sigma(t)$ (with $\\sigma(t)$ the marginal variance at time $t$) couldn't one argue that with uniform error in the estimation of noise $epsilon$, the error in the score will blow up near $t=0$ thus the setting where the error is large close to the end of the generative process is the most likely setting (and therefore $h=0$ should work best)? Perhaps this is related with the fact that practitioners stop the denoising process at time $t=\\eta$ with $\\eta$ a small hyperparameter (e.g. \\eta=e-3)?\n- Would suggest increasing the fontsize of Figures (e.g. Fig 4)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on reverse diffusion process in the presence of non-negligible error in the score function, and estimate KL divergence between the data distribution and the distribution generated by reverse process. They analyze how the KL divergence varies with the diffusion coefficient and demonstrate that a large diffusion coefficient is beneficial when the error of score function is concentrated near beginning of generation process, while a small diffusion coefficient is beneficial in the opposite case. The findings are validated through numerical experiments on Gaussian mixture model and MNIST data set.\n\nSTRENGTHS:\nThe paper introduces a novel analysis of how error accumulation occurs in diffusion models when the error in the score function is not negligible. This is in contrast to conventional analysis that bound error between target and generated distributions when the error in the score function is small.\nThe analysis is only employed in exploring the diffusion coefficient in reverse generative process. However, the analysis has potential implications for various applications where the correction or modification of score function is important. For instance, more theoretical treatments could be possible on image editing(classifier-free-guidance), fair generation(discriminator-guidance), or so on.\n\nWEAKNESSES:\nIn Line 178, relation between the assumption and low-dimensionality of data distribution require further explanation to enhance understanding.\n\nQUESTIONS:\nRegarding the scaling hyperparameter in the loss function of conventional diffusion model, which aims to balance the error at different time of the generative process, can the practical choice of the scaling be associated with the theoretical results presented in this work?\n\nLIMITATIONS:\nThe authors acknowledge limitation of their work. One additional limitation is lack of large-scale experiments to further validate the findings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper provides a theoretical analysis of the estimation error of SDE and ODE methods along with some numerical experiments.

STRENGTHS:
By perturbing the score function, the authors study how the estimation error changes in ODE and SDE methods.

WEAKNESSES:
It is commonly known that the sample generation error consists of three parts: discretization error, estimation error, and initialization error, see e.g. [1, 2]. It is also commonly known that usually the ODE works better with fewer NFE and SDE works better with more NFE due to the interplay between discretization error and estimation error. The estimation error might not even dominates the total error. I don't think only studying the continuous time model provides very useful insight into how to choose those two methods optimally. There is a significant lack of connection between this theory and what happens in reality. Please comment on this.

[1] Chen et al., Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions


[2] Lee et al., Convergence of score-based generative modeling for general data distributions

QUESTIONS:
see weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-13

comments: SUMMARY:
The paper explores the difference between ODE-based probability flow and SDE-based diffusion models when score training errors are present. Specifically, they investigate how setting the generative diffusion coefficient h impacts sample quality.

STRENGTHS:
- The question the authors are trying to answer seems to be a good research question. 
- It's nice to see experiments validating some of their assumptions (Fig 1.).

WEAKNESSES:
- As is, the paper is very difficult to follow. 
- Many equations are presented, but few of them are explained. 
- With so many variables presented in the paper, it is difficult to keep track of all of them, making the paper challenging to comprehend.

QUESTIONS:
N/A

LIMITATIONS:
- The authors briefly address potential negative societal impacts of their work. 
- I did not see any mention of limitations of their analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-25