Title: Leveraging Drift to Improve Sample Complexity of Variance Exploding Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Variance exploding diffusion models

Convergence guarantee

Manifold hypothesis
Abstract: Variance exploding (VE) based diffusion models, an important class of diffusion models, have shown state-of-the-art (SOTA) performance. However, only a few theoretical works analyze VE-based models, and those works suffer from a worse forward convergence rate $1/\text{poly}(T)$ than the $\exp{(-T)}$ of variance preserving (VP) based models, where $T$ is the forward diffusion time and the rate measures the distance between forward marginal distribution $q_T$ and pure Gaussian noise. The slow rate is due to the Brownian Motion without a drift term. In this work, we design a new drifted VESDE forward process, which allows a faster $\exp{(-T)}$ forward convergence rate. With this process, we achieve the first efficient polynomial sample complexity for a series of VE-based models with reverse SDE under the manifold hypothesis. Furthermore, unlike previous works, we allow the diffusion coefficient to be unbounded instead of a constant, which is closer to the SOTA models. Besides the reverse SDE, the other common reverse process is the probability flow ODE  (PFODE) process, which is deterministic and enjoys faster sample speed. To deepen the understanding of VE-based models, we consider a more general setting considering reverse SDE and PFODE simultaneously, propose a unified tangent-based analysis framework, and prove the first quantitative convergence guarantee for SOTA VE-based models with reverse PFODE.
We also show that the drifted VESDE can balance different error terms and improve generated samples without training through synthetic and real-world experiments.

Leveraging Drift to Improve Sample Complexity of Variance Exploding Diffusion Models

Abstract

Variance exploding (VE) based diffusion models, an important class of diffusion models, have shown state-of-the-art (SOTA) performance. However, only a few theoretical works analyze VE-based models, and those works suffer from a worse forward convergence rate 1 / poly ( T ) than the exp( -T ) of variance preserving (VP) based models, where T is the forward diffusion time and the rate measures the distance between forward marginal distribution q T and pure Gaussian noise. The slow rate is due to the Brownian Motion without a drift term. In this work, we design a new drifted VESDE forward process, which allows a faster exp( -T ) forward convergence rate. With this process, we achieve the first efficient polynomial sample complexity for a series of VE-based models with reverse SDE under the manifold hypothesis. Furthermore, unlike previous works, we allow the diffusion coefficient to be unbounded instead of a constant, which is closer to the SOTA models. Besides the reverse SDE, the other common reverse process is the probability flow ODE (PFODE) process, which is deterministic and enjoys faster sample speed. To deepen the understanding of VE-based models, we consider a more general setting considering reverse SDE and PFODE simultaneously, propose a unified tangent-based analysis framework, and prove the first quantitative convergence guarantee for SOTA VE-based models with reverse PFODE. We also show that the drifted VESDE can balance different error terms and improve generated samples without training through synthetic and real-world experiments.

1 Introduction

Recently, diffusion modeling has shown impressive performance in different areas [Ho et al., 2022, Rombach et al., 2022, Esser et al., 2024, Li et al., 2024]. Diffusion models consist of two processes: the forward and reverse process. The forward process gradually converts data q 0 to Gaussian noise, which can be described by an intermediate marginal distribution sequence { q t } t ∈ [0 ,T ] . The reverse process sequentially predicts noise and removes it from data to generate samples.

There are two common forward processes: (1) Variance preserving (VP) SDE and (2) variance exploding (VE) SDE. The VPSDE corresponds to an Ornstein-Uhlenbeck process, and the stationary distribution is N (0 , I ) . The VESDE has an exploding variance in the forward process. In earlier

times, VP-based models [Ho et al., 2020, Lu et al., 2022] provide an important boost for developing diffusion models. Recently, VE-based models have shown the ability to generate data distribution supported on low-dimensional manifolds [Song and Ermon, 2019, 2020]. Since image and text datasets typically exhibit a low-dimensional manifold nature [Pope et al., 2021, Tang and Yang, 2024], VE-based models have achieved great performance in image generation, one-step generation, and reinforcement learning [Teng et al., 2023, Song et al., 2023, Ding and Jin, 2023]. Furthermore, Karras et al. [2022] unify VP and VESDE and prove that the ODE solution trajectory of a specific VESDE is linear and directly towards the data manifold, which makes the denoise process easy.

After determining a forward SDE, diffusion models reverse it and generate samples by running the corresponding reverse process. Since the reverse drift term contains the gradient of forward logarithmic density ∇ log q t (a.k.a. score function), we estimate it by using the score matching technique [Vincent, 2011]. After that, diffusion models discretize the continuous reverse process and run this discrete process starting from pure Gaussian. There are two widely used reverse processes: reverse SDE [Ho et al., 2020] and probability flow ODE (PFODE) [Song et al., 2020a]. The reverse SDE usually generates higher quality samples [Kim et al., 2022]. The reverse PFODE always has a faster generation speed and is useful in other aspects such as calculating likelihoods [Song et al., 2020b] or obtaining one-step generation models [Song et al., 2023]. Hence, these processes are both critical, and providing the guarantee for VE-based models with these processes is necessary.

Recently, many works analyze the convergence guarantee of the VP-based diffusion models under the reverse SDE setting and prove that the VP-based models can sample from the target data distribution with polynomial complexity [Chen et al., 2023c,a,b, Lee et al., 2023, Benton et al., 2023]. As the first step of this work, we also analyze VE-based models under the reverse SDE setting. Different from the VP-based models, only a few works consider VE-based models and all of them suffer from slow 1 / Poly ( T ) forward convergence rate [Lee et al., 2022, Gao et al., 2023, Gao and Zhu, 2024], which is worse than exp( -T ) one for VPSDE. A slow forward convergence rate makes a large distance between q T and pure Gaussian noise, which leads to a large reverse beginning error. From the theoretical perspective, this error introduces hardness to balance three error sources, as shown in Section 5. From the empirical perspective, Lin et al. [2024] show that this error introduces a data information leakage problem, which leads to bad performance. To deal with this problem, De Bortoli et al. [2021] introduce a small drift term to obtain a exp( -√ T ) reverse beginning error. However, they introduce an additional exp( T ) in the discretization error term. Furthermore, their results do not allow unbounded β t , which is the key point of the optimal solution trajectory and used by the SOTA models [Karras et al., 2022, Song et al., 2023]. Therefore, the following question remains open:

Is it possible to design a VESDE with a faster forward convergence rate than 1 / poly ( T ) and achieve the polynomial sample complexity when the diffusion coefficient is unbounded?

In this work, for the first time, we propose a new drifted VESDE forward process, which enjoys a faster forward convergence rate and allows unbounded coefficients. We first show that the drifted VESDE has similar trends but performs better than the original SOTA VESDE on synthetic data (Section 7). After that, we analyze the sample complexity of drifted VESDE under the realistic manifold hypothesis. The manifold hypothesis means the data q 0 is supported on a lower dimensional compact set M , and much empirical evidence shows that image and text dataset satisfy this hypothesis [Fefferman et al., 2016, Pope et al., 2021, Tang and Yang, 2024]. Furthermore, as shown in Section 2, the manifold hypothesis is more realistic than previous data assumptions since it allows the blow-up phenomenon of the score function at the end of the reverse process, which matches the empirical observation [Kim et al., 2021]. Under the manifold hypothesis, we prove that the drifted VESDE with a suitable larger β t balances the reverse beginning, discretization, and approximated score errors and achieves the first efficient polynomial sample complexity for VE-based models with reverse SDE.

To better understand VE-based models, we analyze reverse SDE and PFODE simultaneously after obtaining polynomial complexity for reverse SDE. Despite the great performance, a few theoretical works consider reverse PFODE [Chen et al., 2023d,b, Gao and Zhu, 2024], and these works either focus on VPSDE or have strong assumptions. Hence, we propose the tangent-based framework for VE-based models and achieve the first quantitative convergence for the SOTA VE-based models with reverse PFODE. In conclusion, we accomplish the following results under the manifold hypothesis:

We propose a new drifted VESDE forward process, which allows exp( -T ) forward convergence guarantee with suitable β t . With this process, we achieve the first polynomial sample complexity for a series of VE-based models under the reverse SDE setting.

When considering the general setting, we propose the tangent-based unified framework and analyze reverse SDE and PFODE simultaneously. Under this framework, we prove the first quantitative guarantee for SOTA VE-based models with reverse PFODE.

We show that the drifted VESDE balances different error terms and improves generated samples without training via synthetic and real-world experiments.

2 Related Work

Before providing current results, we first discuss different assumptions about the data distribution from strong to weak. The strongest assumption is the log-concave distribution. While the log-Sobelev inequality (LSI) assumption is slightly weaker, it does not allow the presence of substantial nonconvexity, which is far away from the multi-modal distribution. Recently, some works assume the score function is L -Lipschitz to allow the multi-modal distribution. However, this assumption can not explain the blow-up phenomenon of score [Kim et al., 2021]. The last assumption is the manifold hypothesis, which is supported by much empirical evidence and allows the blow-up score.

Analyses for VP-based models. For the reverse SDE, Lee et al. [2022] achieve the first polynomial complexity with strong LSI assumption. Chen et al. [2023c] remove the LSI assumption, assume the Lipschitz score and achieve polynomial complexity. Bortoli [2022] is the first work to focus on the sample complexity of diffusion models under the manifold hypothesis, and it is the most relevant work to our unified framework. However, as discussed in Section 6.1, the original tangent-based lemma can not deal with reverse PFODE even in VPSDE. We carefully control the tangent process to avoid additional exp( T ) by using the exploding variance property of VESDE. Recently, Chen et al. [2023a] and Benton et al. [2023] also remove the Lipschitz score assumption, and Benton et al. [2023] achieve optimal dependence on d . More recently, Conforti et al. [2023] use bounded Fisher information assumption and replace d with a Fisher information term.

For the PFODE, Chen et al. [2023d] propose the first quantitative result with exponential dependence on the Lipschitz constant. Chen et al. [2023b] achieve polynomial complexity by introducing a corrector component to inject suitable noise. More recently, Li et al. [2023] remove the additional corrector. However, their results rely heavily on the very specific β t , which goes to 0 as T → + ∞ . Since VE-based models have an unbounded β t , this method is not suitable for our models.

Analyses for VE-based models. When considering VESDE, most works focus on constant β t and reverse SDE. De Bortoli et al. [2021] provide the first convergence guarantee with exponential dependence on T . Lee et al. [2022] analyze a constant diffusion coefficient VESDE and achieve polynomial sample complexity under the LSI assumption. When considering the reverse PFODE, Chen et al. [2023d] only consider the discretization error and provide a quantitative convergence guarantee. However, their results introduce additional exp( T ) compared to ours (Section 6.1). Recently, Gao et al. [2023] and Gao and Zhu [2024] provide the polynomial results for a series of VESDE with reverse SDE and reverse PFODE under the log-concave assumption, respectively.

3 The Drifted Variance Exploding (VE) SDE

Diffusion models usually consist of a forward process and a reverse process. The forward process gradually injects noise to convert the data distribution to pure noise. To generate samples, diffusion models reverse the forward process and run the corresponding reverse process.

This section first recalls two previous forward processes: VPSDE and VESDE. Recently, the VEbased models achieve great performance in application [Karras et al., 2022, Song et al., 2023]. However, unlike the widely analyzed VP-based models [Benton et al., 2023, Chen et al., 2023b], the VE-based models suffer from challenges in obtaining an efficient sample complexity due to the slow forward convergence rate. To address this limitation, we introduce a new drifted VESDE forward process, which has a faster forward convergence rate, balances different error terms and achieves the first efficient polynomial sample complexity (see Section 5). Finally, we introduce how to reverse this new forward process and obtain an implementable algorithm.

3.1 The VP and VESDE of Diffusion Models

We first introduce the general form of the forward process and then recall two common forward processes, VPSDE and VESDE, adopted in previous works [Ho et al., 2020, Karras et al., 2022]. Let

q 0 be the data distribution. Given X 0 ∈ R d , the forward process is defined by

where ( B t ) t ≥ 0 is the standard Brownian motion in R d , f ( X t , t ) is a drift coefficient and g ( t ) is a diffusion coefficient. Let q t be the density function of X t at time t . With a suitable choice of drift and diffusion terms (e.g. Section 3.1 and 3.2), the forward process gradually converts the data distribution into Gaussian noise. More specifically, the conditional distribution X t | X 0 is exactly N ( m t X 0 , σ 2 t I ) given X 0 , where m t is determined by the drift term and σ 2 t is determined by the diffusion term.

The VPSDE forward process. Let { β t } t ≥ 0 be a non-decreasing sequence with bounded range [1 / ¯ β, ¯ β ] . The VPSDE has the following formula:

In this case, m t = exp( -∫ t 0 β s d s ) and σ 2 t = 1 -m 2 t . Note that m T ≤ exp( -T/ ¯ β ) ,which indicates a fast forward convergence rate TV( q T |N (0 , I )) ≤ exp( -T/ ¯ β ) [Chen et al., 2023c].

The VESDE forward process. The VESDE forward process is defined without a drift term:

Two common choices for σ 2 t are t and t 2 , with the latter achieving SOTA performance [Karras et al., 2022, Teng et al., 2023]. However, VESDE only has a slow polynomial-decay forward convergence rate (Theorem 4.2), which introduces hardness to obtain an efficient sample complexity (see Section 5). This motivates us to design an improved VESDE process with a fast forward convergence rate.

3.2 The Drifted VESDE Forward Process

Note that the forward convergence rate of the general process is upper bounded by m T /σ 2 T (Theorem 4.2). In practical applications [Ho et al., 2020, Karras et al., 2022, Song et al., 2023], the variance of the forward process σ 2 t at time T , which is determined by the diffusion term, does not exceed T 2 . This indicates the contribution of σ 2 T to the forward convergence rate is only 1 / Poly ( T ) . Hence, the exponential-decay forward convergence rate of VPSDE comes from the drift term, which introduces an exponential-decay m t ≤ exp( -T/ ¯ β ) . Due to the absence of the drift term in VESDE, the data information, such as expectation E [ q 0 ] and covariance Cov [ q 0 ] , does not decay and m t ≡ 1 , which is a key to an only polynomial-decay forward convergence rate m T /σ 2 T ≤ 1 / Poly ( T ) . With the drift term, the VPSDE gradually removes the data information from q t during the process, which makes q t quickly converge to pure Gaussian noise. Inspired by this elimination effect of the drift term, we propose a drifted VESDE forward process:

where τ ∈ [ T, T 2 ] is the coefficient used to balance the drift and diffusion term 2 , and { β t } t ≥ 0 is a positive non-decreasing sequence. In this case,

We show that the drifted VESDE is not only an effective representation of the existing VESDE but also extends beyond it (see Section 7 and Appendix A.1). We also prove that this process with suitable β t has a exp( -T ) forward convergence rate and enjoys an efficient polynomial sample complexity.

3.3 The Reverse Process of the Drifted VESDE

To generate samples from Gaussian noise, a diffusion model reverses the forward process. Let q τ t be the density function of the drifted VESDE forward process at time t and ( Y t ) t ∈ [0 ,T ] = ( X T -t ) t ∈ [0 ,T ] . As shown in Cattiaux et al. [2021], the reverse process of drifted VESDE has the following form 3 :

The parameter η ∈ [0 , 1] is used to determine the type of reverse processes. There are two common reverse processes in the application: reverse probability flow ODE (PFODE) ( η = 0 ) [Song et al., 2020b, 2023] and reverse SDE ( η = 1 ) [Ho et al., 2020].

To generate distribution q 0 through running the above reverse process, diffusion models need the true score function ∇ log q τ T -t ( Y t ) and the accurate reverse beginning distribution q τ T . However, ∇ log q τ T -t ( Y t ) and q τ T contain the data information and usually can not be exactly calculated. For the score function, diffusion models approximate it using a score network s ( T -t, · ) by minimizing the score matching objective function [Vincent, 2011]. For the initial distribution of the reverse process, since q τ T should be close to a pure Gaussian, we choose q τ ∞ = N (0 , σ 2 T I ) as an approximation. Then, the continuous reverse process ( ̂ Y t ) t ∈ [0 ,T ] , incorporating s ( T -t, · ) and q τ ∞ , is defined as:

Since diffusion models can not run a continuous process due to the nonlinear score function, these models usually discretize the above continuous process and freeze the approximated score at the beginning of each interval. Let { γ k } k ∈ [ K ] be the stepsize and t k +1 = ∑ k j =0 γ j . As shown in Kim et al. [2021], ∇ log q τ T -t ( Y t ) goes to + ∞ at the end of the reverse process. To mitigate this issue, they use the early stopping technique t K = T -δ , and we also employ this technique in this work. With the stepsize, we choose the exponential integrator discretization scheme [Zhang and Chen, 2022] to discretize the above process, which runs the following process:

As shown in Karras et al. [2022], the choice of β t significantly affects the performance of models, and we need to determine β t before running the reverse process. The state-of-the-art diffusion models adopt β t = t , which increases rapidly and has an unbounded range. However, current theoretical works assume β t to be a constant [Chen et al., 2023c] or confined to a bounded interval [1 / ¯ β, ¯ β ] [Bortoli, 2022] to match the setting of VPSDE. To align more closely with practical applications of VE-based models, we allow an unbounded β t in this work. Furthermore, we make a detailed assumption on β t when considering different reverse processes.

Assumption 3.1. Let { β t } t ≥ 0 be a positive, non-decreasing sequence. For any τ ∈ [ T, T 2 ] , there exists constants ¯ β and C , such that for any t ∈ [0 , T ] : (1) for η = 0 , then 1 / ¯ β ≤ β t ≤ max { ¯ β, t } and ∫ T 0 β t /τ d t ≤ C ; (2) for η = 1 , then 1 / ¯ β ≤ β t ≤ max { ¯ β, t 2 } .

As shown in Chen et al. [2023b], due to the absence of the stochasticity, the small errors for quickly accumulate and are magnified. Hence, we assume a conservative β t for the reverse PFODE, whose growth rate is at most t , to avoid an additional exp( T ) in the convergence guarantee (see Section 6.1). We note that this choice of β t is satisfied in practical applications [Song et al., 2020b, Karras et al., 2022]. For the reverse SDE setting, we assume the growth rate of β t can depend on τ instead of at most linear. For example, when τ = T 2 , we can choose β t = t 2 , which has the same order as τ . As shown in Theorem 4.2, the drifted VESDE with aggressive β t = t 2 has an exponential-decay forward convergence rate, which leads to the first efficient polynomial complexity for VE-based models.

d d × d

Notations. For x ∈ R and A ∈ R , we denote by ∥ x ∥ and ∥ A ∥ the Euclidean norm for vector and the spectral norm for matrix. We denote by ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k the maximum stepsize for k ∈ [0 , K -1] . We denote by q 0 P T the distribution of X T , Q q τ ∞ t K the distribution of Y t K , R q τ ∞ K the distribution of ˜ Y t K and Q q 0 P T t K the distribution which does reverse process starting from q τ T (Eq. 5). We denote by W 1 and W 2 the Wasserstein distance of order one and two, respectively.

4 The Faster Forward Convergence Rate for the Drifted VESDE

This section shows that the drifted VESDE has a fast forward convergence rate. Since q τ T contains the data information, we first introduce the manifold hypothesis before controlling TV( q τ T , q τ ∞ ) .

Assumption 4.1. q 0 is supported on a compact set M and 0 ∈ M .

We denote R the diameter of the manifold by R = sup {∥ x -y ∥ : x, y ∈ M} and assume R > 1 . As shown in Section 1, the manifold hypothesis is supported by much empirical evidence [Bengio

et al., 2013, Fefferman et al., 2016, Pope et al., 2021] and allows the blow-up phenomenon of the score. Recently, Tang and Yang [2024] show that diffusion models can adapt to the intrinsic manifold structure. With Assumption 4.1, we obtain the forward process guarantee for the drifted VESDE.

Theorem 4.2. Assume Assumption 4.1 and 3.1. Let q τ ∞ = N (0 , σ 2 T I ) . With m T , σ T defined in Equation (4) , we have TV( q τ T , q τ ∞ ) ≤ √ m T ¯ D/σ T , where ¯ D = d | c | + E [ q 0 ] + R and c is the eigenvalue of Cov [ q 0 ] with the largest absolute value.

∫ T

Recall that m T = exp( -0 β t /τ d t ) , the previous VESDE [Song et al., 2020b, Karras et al., 2022, Lee et al., 2022] chooses a conservative β t satisfies ∫ T 0 β t /τ d t ≤ C . However, with an aggressive β t , the drifted VESDE will have a faster convergence rate. To illustrate the accelerated forward process, we use τ = T 2 as an example and discuss different β t = t α 1 , α 1 ∈ [1 , 2] . Due to the definition of σ T , σ T ≈ T , and the forward convergence rate mainly depends on √ m T . When α 1 = 1 is conservative, m T is a constant, and the convergence rate is 1 /T . When α 1 = 1 + ln(2 r ln( T )) / ln( T ) is slightly aggressive, the convergence rate is 1 /T r +1 for r > 0 . When α 1 ≥ 1 + ln( T -ln( T )) / ln( T ) is aggressive, the convergence rate is faster than exp( -T ) . In our analysis, whether β t can be aggressive depends on the reverse process (see Section 6). When choosing an aggressive β t , the drifted VESDE achieves an improved sample complexity compared with pure VESDE (see Section 5).

5 The Polynomial Complexity for a Series of VESDE with Reverse SDE

In this section, we first pay attention to the reverse SDE ( η = 1 ) to show the power of the drifted VESDE. More specifically, we show that our general drifted VESDE form covers the current models (VP and VESDE). After that, we show that drifted VESDE can go beyond the current models and achieve an improved complexity with an aggressive β t . Since the objective function minimizes the L 2 distance between the ground truth and the approximated score, we assume that the approximated score is L 2 -accuracy, which is exactly the same with Chen et al. [2023c] and Benton et al. [2023].

With this assumption, we provide a universe convergence guarantee for τ ∈ [1 , T 2 ] and β t ∈ [1 , t 2 ] and discuss the sample complexity of VP and VE-based models in detail.

Theorem 5.2. Assume Assumption 3.1, 4.1, 5.1. Let ¯ D defined in Theorem 4.2, ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k , τ = T 2 and β t ∈ [1 , t 2 ] . Then, we have that

To guarantee the above convergence guarantee smaller than ˜ O ( ϵ TV ) , each component of the result needs to be smaller than ϵ TV . As shown in Remark 5.3, it is difficult for pure VESDE to balance the approximated score and the first two error terms to achieve an efficient sample complexity. Hence, we discuss how to balance the reverse beginning and discretization error. More specifically, we require ¯ D √ m T /σ T ≤ ϵ TV and ¯ γ K ≤ σ 8 δ ϵ 2 TV / ( R 4 dβ T τT ) , where the first inequality determines the order of T and the second inequality determines the stepsize ¯ γ K . After that, with sample complexity K = T/ ¯ γ K , we have that TV( R q τ ∞ K , q δ ) ≤ ˜ O ( ϵ TV ) . The last step is to guarantee q δ and q 0 is close enough W 2 2 ( q 0 , q δ ) ≤ ϵ 2 W 2 , which requires σ 2 δ ≤ ϵ 2 W 2 / ( d + R √ d ) .

Following the above process, this general convergence guarantee leads to the polynomial sample complexity for VP and VE-based models. When β t = 1 and τ = 1 , the drifted VESDE becomes VPSDE and achieve the complexity ˜ O (1 / ( ϵ 8 W 2 ϵ 2 TV )) , which achieve exactly the same order compared with Chen et al. [2023c]. When β t = 1 and τ = T , our formula is similar but slightly better (Figure 2 and 3) to pure VESDE ( σ 2 t = t ) and achieves O (1 /ϵ 8 W 2 ϵ 8 TV ) result. For β t = t and τ = T 2 , the general formula is similar to SOTA pure VESDE ( σ 2 t = t 2 ) and achieves the first polynomial results O (1 /ϵ 8 W 2 ϵ 7 TV ) for this model under the manifold hypothesis 4 .

Although we achieve the first polynomial sample complexity for VE-based models under the manifold hypothesis, it is clear that the results of the VE-based models are significantly worse than the result

of VP-based models since the slow 1 / Poly ( T ) forward convergence guarantee. More specifically, the forward convergence rate of VPSDE is exp( -T ) , which means T has the order log(1 /ϵ TV ) and can be ignore. When considering the pure VESDE with σ 2 t = t 2 , the forward convergence rate is ¯ D/T , which indicates T ≥ ¯ D/ϵ TV is a polynomial term and can not be ignored. Hence, the results K = R 4 dT 5 / ( σ 8 δ ϵ 2 TV ) of pure VESDE ( σ 2 t = t 2 ) is heavily influenced by T . When considering pure VESDE with σ 2 t = t , it suffers from a slower forward convergence guarantee ¯ D/ √ T , which indicates T ≥ ¯ D 2 /ϵ 2 TV and K = R 4 dT 3 / ( σ 8 δ ϵ 2 TV ) . This is the first work to explain why pure VESDE ( σ 2 t = t 2 ) performs better than pure VESDE ( σ 2 t = t 2 ) from a theoretical perspective.

Remark 5.3 . In this part, we explain the reason why the pure VESDE fails to balance the reverse beginning and the approximated score. We use the pure VESDE with σ 2 t = t 2 as an example. Under this setting, the guarantee has the form 1 /T + √ ¯ γ K T 4 /δ 4 + ϵ score √ T 2 , which requires T ≥ 1 /ϵ TV . Then, ϵ score √ T 2 is larger than ϵ score / √ ϵ 2 TV . Hence, it is hard to achieve non-asymptotic results.

Drifted VESDE with an aggressive β t balances different error terms. This part shows that our drifted VESDE with a suitable aggressive β t can balance the above three error terms. More specifically, we show that introducing aggressive β t only slightly affects the discretization error and significantly benefits in balancing reverse beginning and approximated errors. As a result, we obtain an efficient polynomial complexity for a series of VE-based models with unbounded β t .

Corollary 5.4. Following the setting of Theorem 5.2. When considering τ = T 2 , β t = t 2 , by choosing δ ≤ ϵ 2 / 3 W 2 ( d + R √ d ) 1 / 3 , T ≥ 2 ln( ¯ D/ϵ TV ) , ¯ γ K ≤ δ 12 ϵ 2 TV ln 5 ( ¯ D/ϵ TV ) / ( R 4 d ) and assuming ϵ score ≤ ˜ O ( ϵ TV ) , R q τ ∞ K is ( ϵ TV + ϵ score ) close to q δ , which is ϵ W 2 close to q 0 , with sample complexity

Defined by R q τ ∞ K,R 0 the output R q τ ∞ K projected onto B(0 , R 0 ) for R 0 = ˜ Θ( R ) . Then, we achieve pure W 2 guarantee W 2 ( R q τ ∞ K,R 0 , q 0 ) ≤ ϵ W 2 with sample complexity ˜ O ( dR 8 ( d + R √ d ) 4 ϵ 12 W 2 ) .

Since our drifted VESDE with τ = T 2 and β t = t 2 has a fast forward convergence rate exp( -T ) /T 2 , T becomes a logarithmic term and does not influence the discretization term, which is the source of the improved sample complexity. Furthermore, the requirement of ϵ score has the same order with ϵ TV , which indicates the drifted VESDE balances the reverse beginning and approximated score error. In fact, we only require the forward convergence rate of drifted VESDE is exp( -T ) , which indicates a series of VE-based models can achieve this sample complexity. We use τ = T 2 as an example. When considering the β t = t α 1 , we require 2 ≥ α 1 ≥ 1 + ln( T -ln( T )) / ln( T ) to enjoy exp( -T ) forward convergence rate and achieve K ≤ ˜ O ( 1 / ( ϵ 8 W 2 ϵ 2 TV ) ) sample complexity. For β t = t and τ = T , we also obtain complexity K ≤ ˜ O ( 1 / ( ϵ 8 W 2 ϵ 2 TV )) (Appendix C.1).

Remark 5.5 . Recently, Lee et al. [2022] and Gao et al. [2023] consider the sample complexity of VESDE with reverse SDE under strong assumption. Lee et al. [2022] consider VESDE with σ 2 t = t and achieve ˜ O ( L 2 /ϵ 4 TV ) result under the LSI assumption. Under the manifold hypothesis, the result is ˜ O (1 /ϵ 8 W 2 ϵ 4 TV ) , which is worse than Corollary 5.4. Gao et al. [2023] achieve pure W 2 guarantee ˜ O (1 /ϵ 2 . 5 W 2 ) under the log-concave distribution, which is even stronger than LSI assumption and ignore the influence of δ . Hence, they ignore an additional 1 / Poly ( W 2 ) (Detail in Appendix A.2).

6 The Tangent-based Analysis Framework

To deepen the understanding of VE-based models instead of the specific reverse process, we introduce the unified framework for VESDE with reverse SDE and PFODE. Similar to previous PFODE work [Chen et al., 2023d], we assume an accurate score and consider the other errors.

Theorem 6.1. Assume Assumption 3.1 and 4.1, δ ≤ 1 / 32 and γ k sup v ∈ [ T -t k +1 ,T -t k ] β v /σ 2 v ≤ 1 / 28 for ∀ k ∈ { 0 , ..., K -1 } . Let γ K = δ . Then, for ∀ τ ∈ [ T, T 2 ] :

(1) If η = 1 (the reverse SDE), choosing β t = t 2 , W 1 ( R q τ ∞ K , q 0 ) is bounded by

where κ 1 ( τ ) = T 2 (1 /τ + ¯ β/δ 3 ) and C 1 ( τ ) is linear in τ 2 .

(2) If η = 0 (PFODE), choosing a conservative β t (Assumption 3.1), W 1 ( R q τ ∞ K , q 0 ) is bounded by

where κ 2 ( τ ) = T ( 1 /τ + ¯ β/δ 2 ) and C 2 ( τ ) is linear in τ 2 .

Theorem 6.1 proves the first quantitative guarantee for VE-based models with reverse PFODE using the unified tangent-based framework. Correspondingly, the Girsanov-based method [Chen et al., 2023c,a] can not deal with reverse PFODE since the reverse process diffusion term is not well-defined. Recently, Chen et al. [2023d] employ the Restoration-Degradation framework to analyze VESDE with reverse PFODE, which also has exponential dependence on R and δ . Furthermore, their results have exponential dependence on β t ( g max in Chen et al. [2023d]), which corresponds to τ . However, our dependence on τ appears in the polynomial term. Hence, our framework is a suitable unified framework. Furthermore, we emphasize that our tangent-based unified framework is not a simple extension of Bortoli [2022]. We carefully control the tangent process according to the variance exploding property of VESDE to avoid exp( T ) term when considering PFODE (Section 6.1).

Theorem 6.1 has exponential dependence on R and δ , which is introduced by the tangent process. Similar to Bortoli [2022], if we assume the Hessian ∥ ∥ ∇ 2 log q t ( x t ) ∥ ∥ ≤ Γ /σ 2 t , we obtain a better control on the tangent process and replace the exponential dependence on δ by a polynomial dependence on δ and exponential dependence on Γ when considering reverse PFODE.

Corollary 6.2. Assume Assumption 3.1, 4.1 and ∥ ∥ ∇ 2 log q t ( x t ) ∥ ∥ ≤ Γ /σ 2 t . Let η = 0 (reverse PFODE), δ ∈ (0 , 1 / 32) , τ = T 2 , β t = t and κ 2 ( τ ) , C 2 ( τ ) defined in Theorem 6.1, we have

Though the additional assumption is strong, many special cases, such as hypercube M = [ -1 / 2 , 1 / 2] p satisfy this assumption. We emphasize that our analysis also holds for VESDE ( σ 2 t = t 2 ) with reverse PFODE, which means our results can explain the SOTA model in Karras et al. [2022].

6.1 The Discussion on the Unified Framework

In this section, we introduce the unified tangent-based framework for reverse SDE and PFODE and discuss key steps to achieve the quantitative guarantee for PFODE. Firstly, we decompose the goal W 1 ( R q τ ∞ K , q 0 ) into three terms: W 1 ( R q τ ∞ K , Q q τ ∞ t K ) + W 1 ( Q q τ ∞ t K , Q q 0 P T t K ) + W 1 ( Q q 0 P T t K , q 0 ) .

These terms correspond to the discretization scheme, reverse beginning distribution, and the early stopping parameter δ . We focus on most difficult discretization term and first recall the stochastic flow of the reverse process for any x ∈ R d and s, t ∈ [0 , T ] with t ≥ s :

The key of the discretization error is to bound tangent process ∥ ∥ ∇ Y x s,t K ∥ ∥ . For this term, we consider the reverse SDE and PFODE simultaneously and propose a general version of Bortoli [2022].

Lemma 6.3. Assume Assumption 3.1 and 4.1. For ∀ s ∈ [0 , t K ] and x ∈ R d , we have

(a) Original Figure

𝑇 = 100

)

(c) Drifted VESDE (

𝑇 = 100

)

swissroll

Swissroll,VESDE,Euler_Maruyama_sampler

swissroll,Drift_VESDE,Euler_Maruyama_sampler

We emphasize that the general bound for the tangent process is the key to achieving the guarantee for VESDE with the reverse ODE. Recall that in the original lemma for the tangent processes, since τ is independent of T and β t is bounded in a small interval [1 / ¯ β, ¯ β ] , ∫ t K 0 β T -u /τ d u = Θ( T ) , which means there is an additional exp( T ) when considering VPSDE with revere PFODE. However, our tangent-based lemma makes use of the variance exploding property of VESDE to guarantee that ∫ T 0 β t /τ d t ≤ C with a conservative β t = t when considering reverse PFODE. When η = 1 , we choose aggressive β t = t 2 since the choice of β t does not affect the bound of the tangent process.

For the early stopping term, it corresponds to δ and is smaller than 2( R/τ + √ d ) √ δ . Since we can not use the data processing inequality in Wasserstein distance, the reverse beginning terms consists of the bound of tangent process term and the forward process term:

One notable future work is introducing the short regularization technique [Chen et al., 2023b] and suitable corrector to remove the above exponential dependence.

7 Experiments

In this section, we show the power of the drifted VESDE forward process through experiments. Section 7.1 shows that aggressive one achieves good balance in different error terms. After that, we consider the approximated score and show that the conservative one can improve the quality of the generated distribution without training in the synethetic and real-world setting.

7.1 The Aggressive Drifted VESDE Balances Errors

In this section, we do experiments on 2-D Gaussian to show that the aggressive drifted VESDE balances different errors. Since the ground truth score of the Gaussian can be directly calculated, we use the accurate score function to discuss the balance between the other two error terms clearly. We show how to use approximated score in Section 7.2.

As shown in Figure 2, the process with aggressive β t = t 2 achieves the best and second performance in EI and EM discretization, which supports our theoretical result (Corollary 5.4). The third best process is conservative β t = t with the small drift term. The reason is that though it can not achieve a exp( -T ) forward guarantee, it also has a constant decay on prior information, which slightly reduces the effect of the reverse beginning error (Section 3.1). The worst process is pure VESDE since it is hard to balance different error sources. Our experimental results also show that EI is better than EM.

EM,βt=t²,T=T2

6

EM,βt=t,T =T2

KL Divergence

5

EM,pureVESDE,o²=t2

El,βt=t²,T=T2

4

El,βt=t, T=T2

El, pure VESDE, o² = t2

3

2

0.0050.0100.0150.0200.0250.0300.0350.040

Stepsize

7.2 The Conservative Drifted VESDE Benefits from VESDE without Training

As shown in Figure 2, the red and orange lines have similar trends. Hence, for conservative drifted VESDE, which satisfies (2) of Assumption 3.1, we can directly use the models trained by pure VESDE to improve the quality of generated distribution. We confirm our intuition by training the model with pure VESDE with σ 2 t = t and directly use the models to conservative drifted VESDE with β t = 1 and τ = T . From the experimental results (Figure 1), it is clear that pure VESDE has a low density on the Swiss roll except for the center one, which indicates pure VESDE can not deal with large dataset variance, as we discuss in Section 4. For conservative drift VESDE, as we discuss in the above section, it can reduce the influence of the dataset information. Figure 1 (c) supports our augmentation and shows that the density of the generated distribution is more uniform compared to pure VESDE, which means that the drift VESDE can deal with large dataset mean and variance.

Beyond the synthetic data, we show that our conservative drifted VESDE can improve the generated images of pure VESDE without training on the real-world CelebA256 dataset. From the qualitative perspective, as shown in Figure 3, the images generated by our drifted VESDE have more detail (such as hair and beard details). On the contrary, since pure VESDE can not deal with large variance, the images generated by pure VESDE appear blurry and unrealistic in these details. From the quantitative results, we use aesthetic score [Schuhmann et al., 2022] and Inception Score to measure the quality of generated images. Our drifted VESDE achieves aesthetic score 5 . 813 , and IS 4 . 174 , which is better than the results of baseline pure VESDE (aesthetic score 5 . 807 and IS 4 . 082 ). There are more examples on CelebA256 and more experiments on Swiss roll and 1D-GMM to explore different sampling methods (RK45, PFODE) and different T . We refer to Appendix G for more details.

8 Conclusion

In this work, we analyze the VE-based models under the manifold hypothesis. Firstly, we propose a new forward drifted VESDE process, which enjoys a faster forward convergence rate. Then, we show that with an aggressive β t , the new process balances different errors and achieve the first efficient polynomial sample complexity for a series of VE-based models with reverse SDE.

After achieving the above results, we go beyond the reverse SDE and propose the tangent-based unified framework, which considers reverse SDE and PFODE at the same time. Under this framework, we make use of the variance exploding property of VESDE and achieve the first quantitative convergence guarantee for SOTA VE-based models with reverse PFODE. Finally, we show the power of the new drifted forward process through synthetic and real-world experiments.

Future Work and Limitation. This work proposes the first unified framework for VE-based models with an accurate score. After that, we plan to consider the approximated score error and provide a polynomial complexity for the VE-based models with reverse PFODE under the manifold hypothesis.

Broader Impact. Our work focuses on the convergence guarantee of the SOTA diffusion models and deepens the understanding of diffusion models. Therefore, this work can be viewed as a fundamental step in improving the quality of diffusion models and the societal impact is similar to general generative models [Mirsky and Lee, 2021].

Acknowledgments and Disclosure of Funding

The author Bo Jiang is supported by National Natural Science Foundation of China (62072302).

References

Vladimir M Alekseev. An estimate for the perturbations of the solutions of ordinary differential equations. Vestn. Mosk. Univ. Ser. I. Math. Mekh , 2:28-36, 1961.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Linear convergence bounds for diffusion models via stochastic localization. arXiv preprint arXiv:2308.03686 , 2023.

Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. Trans. Mach. Learn. Res. , 2022, 2022. URL https://openreview.net/forum?id=MhK5aXo3gB .

Patrick Cattiaux, Giovanni Conforti, Ivan Gentil, and Christian Léonard. Time reversal of diffusion processes under a finite entropy condition. arXiv preprint arXiv:2104.07708 , 2021.

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: Userfriendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning , pages 4735-4763. PMLR, 2023a.

Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. arXiv preprint arXiv:2305.11798 , 2023b.

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023c. URL https://openreview.net/pdf?id=zyLVMgsZ0U_ .

Sitan Chen, Giannis Daras, and Alex Dimakis. Restoration-degradation beyond linear diffusions: A nonasymptotic analysis for ddim-type samplers. In International Conference on Machine Learning , pages 4462-4484. PMLR, 2023d.

Giovanni Conforti, Alain Durmus, and Marta Gentiloni Silveri. Score diffusion models without early stopping: finite fisher information is all you need. arXiv preprint arXiv:2308.12240 , 2023.

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems , 34: 17695-17709, 2021.

Pierre Del Moral and Sumeetpal Sidhu Singh. Backward itô-ventzell and stochastic interpolation formulae. Stochastic Processes and their Applications , 154:197-250, 2022.

Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984 , 2023.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society , 29(4):983-1049, 2016.

Xuefeng Gao and Lingjiong Zhu. Convergence analysis for general probability flow odes of diffusion models in wasserstein distances. arXiv preprint arXiv:2401.17958 , 2024.

Xuefeng Gao, Hoang M Nguyen, and Lingjiong Zhu. Wasserstein convergence guarantees for a general class of score-based generative models. arXiv preprint arXiv:2311.11003 , 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35:26565-26577, 2022.

Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and Il-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2106.05527 , 2021.

Dongjun Kim, Yeongmin Kim, Wanmo Kang, and Il-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. arXiv preprint arXiv:2211.17091 , 2022.

Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Fpdiffusion: Improving score-based diffusion models by enforcing the underlying score fokker-planck equation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 18365-18398. PMLR, 2023.

Jean-François Le Gall. Brownian motion, martingales, and stochastic calculus . Springer, 2016.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems , 35:22870-22882, 2022.

Holden Lee, Jianfeng Lu, and Yixin Tan. Convergence of score-based generative modeling for general data distributions. In International Conference on Algorithmic Learning Theory , pages 946-985. PMLR, 2023.

Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards faster non-asymptotic convergence for diffusion-based generative models. arXiv preprint arXiv:2306.09251 , 2023.

Haoran Li, Haolin Shi, Wenli Zhang, Wenjun Wu, Yong Liao, Lin Wang, Lik-hang Lee, and Pengyuan Zhou. Dreamscene: 3d gaussian-based text-to-3d scene generation via formation pattern sampling. arXiv preprint arXiv:2404.03575 , 2024.

Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5404-5411, 2024.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022.

Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey. ACM Computing Surveys (CSUR) , 54(1):1-41, 2021.

Jakiw Pidstrigach. Score-based generative models detect manifolds. arXiv preprint arXiv:2206.01018 , 2022.

Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension of images and its impact on learning. arXiv preprint arXiv:2104.08894 , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35: 25278-25294, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020a.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems , 33:12438-12448, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020b.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 32211-32252. PMLR, 2023.

Rong Tang and Yun Yang. Adaptivity of diffusion models to manifold structures. In International Conference on Artificial Intelligence and Statistics , pages 1648-1656. PMLR, 2024.

Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhuoyi Yang, and Jie Tang. Relay diffusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.03350 , 2023.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7): 1661-1674, 2011.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 , 2022.

Appendix

A More Discussion on Drifted VESDE and Current Works

A.1 The Drifted VESDE is Representative Enough.

In this work, we consider τ ∈ [ T, T 2 ] and show that this choice is enough to represent the current VESDE. More specifically, since VESDE with σ 2 t = t has q T = N ( E [ q 0 ] , Cov [ q 0 ] + T I ) and drifted VESDE with τ = T 2 and β t ≡ 1 / 2 has q τ T = N (exp ( -1 2 T ) E [ q 0 ] , exp( -1 T ) Cov [ q 0 ] + (1 -exp( -1 T )) T 2 I ) , these two setting is almost identical when T → + ∞ . The second choice of VESDE σ 2 t = t 2 , which achieves the state-of-the-art performance [Karras et al., 2022], is almost identical to τ = T 2 , β t = t . The simulation experiments also show that VESDE and drifted VESDE with specific β t and τ have similar performance (Figure 2).

A.2 The Detailed Calculation of Previous work

The results of Lee et al. [2022] Lee et al. [2022] consider VESDE ( σ 2 t = t ) with reverse SDE under the LSI assumption with parameter C LS. The LSI assumption does not allow the presence of substantial non-convexity and is far away from the multi-modal real-world distribution. Furthermore, they use unrealistic assumption ϵ score ≤ 1 / ( C LS + T ) to avoid the effect of the approximated score, which is stronger than Assumption 5.1. Under the above strong assumption, Lee et al. [2022] achieve the polynomial sample complexity ˜ O ( L 2 d ( d | c | + R ) 2 /ϵ 4 TV ) . Under the manifold hypothesis, by Lemma E.2, we know that

Then, the result is ˜ O ( R 4 d 5 ( d | c | + R ) 2 /ϵ 8 W 2 ϵ 4 TV ) , which is worse than Corollary 5.4.

The results of Gao et al. [2023]. Gao et al. [2023] analyze a series of VESDE with reverse SDE and achieve 1 /ϵ 2 . 5 W 2 sample compelxity for VESDE with σ 2 t = t 2 in 2-Wasserstein distance. However, they assume the data distribution is log-concave, which is even stronger than LSI assumption. Furthermore, under this assumption, ∇ log q T -t ( · ) do not blow-up at the end of the reverse process, which do not match the empirical phenomenon and ignore the influence of early stopping parameters. To transfer their results to our the results under the manifold hypothesis, we need to consider the influence of δ , which would introduce an additional 1 / Poly ( W 2 ) term.

B The Proof for the Faster Forward Process

Lemma B.1. The minimization problem min ¯ m t ,V t KL( q τ t | N ( ¯ m t , V t )) is minimized by ¯ m t = m t E [ q 0 ] and V t = m 2 t Cov[ q 0 ] + σ 2 t I , where m t and σ t defined in Equation (4) .

Proof . For simplicity, we denote the mean and covariance of q 0 by a and C ′ . We also define the optimize variable n t = N ( ¯ m t , C t ) . We can directly compute the KL divergence KL( q t | n t ) :

For the last term, we directly compute

where the second inequality follows that X t = m t X 0 + σ t Z . It is clear that the optimal solution of ¯ m t is m t a . In the next step, we focus on the optimization problem for V t :

Since the above optimization is a convex optimization problem, we use the method similar to Pidstrigach [2022], we obtain that the optimal solution of V t is m 2 t C ′ + σ 2 t I . ■

Lemma B.2. Let ¯ m t and V t be the optimal mean and covariance operator from Lemma B.1. Then

where c i are the eigenvalues of Cov[ q 0 ] , and c is the eigenvalue with the largest absolute value.

Proof . For t ≥ 0 , we directly calculate the KL divergence for this term:

where c i are the eigenvalues of Cov[ q 0 ] . Now, we only need to calculate H ( q t ) :

Then, we know that

we also know that

Finally, put these terms together, we have:

where c i are the eigenvalues of Cov[ q 0 ] . Then by choosing the largest absolute value eigenvalue largest absolute value, we can use the Taylor expansion to obtain the first results of this lemma. For the second result of this lemma, we directly compute the KL divergence between N ( ¯ m t , V t ) and N (0 , σ 2 t ) to obtain the final results. ■

Theorem 4.2. Assume Assumption 4.1 and 3.1. Let q τ ∞ = N (0 , σ 2 T I ) . With m T , σ T defined in Equation (4) , we have TV( q τ T , q τ ∞ ) ≤ √ m T ¯ D/σ T , where ¯ D = d | c | + E [ q 0 ] + R and c is the eigenvalue of Cov [ q 0 ] with the largest absolute value.

Proof . We know that

By directly using the Pinsker's inequality and Lemma B.2, we complete the proof. ■

C The Proof of the Polynomial Complexity for Reverse SDE

In this section, we prove Corollary 5.4. First, we recall the Girsanov's Theorem [Le Gall, 2016] used in Chen et al. [2023c]:

Lemma C.1 (Girsanov's theorem) . Let P T and Q T be two probability measures on path space C ( [0 , T ]; R d ) . Suppose that under P T , the process ( X t ) t ∈ [0 ,T ] follows

where ˜ B is a P T -Brownian motion, and under Q T , the process ( X t ) t ∈ [0 ,T ] follows

where B is a Q T -Brownian motion. We assume that for each t > 0 , α t is a d × d symmetric positive definite matrix. Then, provided that Novikov's condition holds,

we have that

If the Novikov's condition is satisfied, we apply the Girsanov theorem by choosing P T = R q τ T K , Q T = Q q τ T t K , ˜ b t = β T -t { 1 τ ˜ Y t +2 s ( T -t k , ˜ Y t ) } (for t ∈ [ t k , t k +1 ] ), b t = β T -t { 1 τ Y t + ( 1 + η 2 ) ∇ log q T -t ( Y t ) } , and α t = √ 2 β T -t I d .

Then, similar to Chen et al. [2023c], we have the following lemma.

Lemma C.2. Assuming that R q τ T K and Q q τ T t K satisfy Novikov's condition, it holds that

Before using the Girsanov's Theorem, we need to check the Novikov's condition. We use almost the same proof process compare to Chen et al. [2023c]. The key proof of the Novikov's condition is Lemma 19 of Chen et al. [2023c]. Hence, we give a complete proof of Lemma 19 in Chen et al. [2023c] under our drifted VESDE. Before the proof, we first introduce a smooth cuttoff function for truncating the drift terms.

Lemma C.3 (lemma 17 of Chen et al. [2023c]) . For any ¯ R > 0 , there is a smooth function ϕ R : R d → [0 , 1] satisfying: 1. ϕ ¯ R ( x ) = 1 for all ∥ x ∥ ≤ ¯ R , 2. ϕ ¯ R ( x ) = 0 for all ∥ x ∥ ≥ 2 ¯ R , 3. ϕ ¯ R is O (1 / ¯ R ) -Lipschitz.

Note that ¯ R is not R in Assumption 4.1 and will goes to + ∞ in the proof of Chen et al. [2023c]. Similar to Chen et al. [2023c], we also introduce a L ∞ and a modified process with truncation argument for t ∈ [ t k , t k +1 ] . Define the bad set

where ε score, ∞ > 0 is a parameter to be chosen later. We define the L ∞ -accurate score estimate to be

We note that ∥ s ∞ t -∇ ln q t ∥ ≤ ε score , ∞ .

The modified process with truncation argument for t ∈ [ t k , t k +1 ] is

where Y ∞ 0 = ˜ Y ∞ 0 is obtained by sampling Y ∞ 0 ∼ q τ T and setting ˜ Y ∞ 0 = X T if ∥ X T ∥ ≤ ¯ R and setting ˜ Y ∞ 0 = 0 otherwise. Then, we ara ready to prove the following lemma.

Lemma C.4 (Modified key lemma for Novikov's condition) .

Proof . We note that due to the manifold hypothesis (Assumption 4.1), if ¯ R ≥ √ dT 2 + R 2 , then the marginal distribution of Y ∞ T -t k is exactly the same compared to X t k . We also recall that ¯ R → + ∞ in Chen et al. [2023c] (Theorem 21 of Chen et al. [2023c]). Hence, we can use Lemma E.1 to prove that

and

Then, the left hand of this lemma is at most exp ( 2 T ( A ′ 2 + B ′ 2 )) < ∞ as claimed.

■

After obtaining the above inequality, the remaining proof for Novikov's condition are exactly compared to Chen et al. [2023c].

Since we assume the accurate score function in this work, this lemma need to control

As we shown in Lemma E.1, we know that with the early stopping parameter δ , ∥∇ ln q T -t ∗ ( x ∗ ) ∥ is controlled. By using Assumption 4.1, we know that 1 β T -t ∗ ≤ ¯ β . Finally, with similar process to Chen et al. [2023c], we can proof that the Novikov's condition is satisfied. The following lemma show the discretization error for our drifted VESDE with reverse SDE.

Lemma C.5 (Discretization) . Suppose that Assumption 4.1 and Assumption 5.1 holds. Let ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k , γ K = δ . If τ ∈ [1 , T 2 ] and β t ∈ [1 , t 2 ] , then with Q q τ T t K and R q τ T K defined in Lemma C.2,

Proof . First, we control the discretization error in an interval t ∈ [ t k , t k +1 ] :

where L = max t ∈ [0 ,T -δ ] ∥ ∥ ∇ 2 log q T -t ( Y t ) ∥ ∥ ≤ ( 1 + R 2 ) /σ 4 δ and the third inequality follows Lemma E.5. Then, we know that

After obtaining the general discretization error for our drifted VESDE, we focus on two special cases. For τ = T 2 and β t = t 2 , we have that

For τ = T and β t = t , we know that

■

Combined with the reversing beginning error controlled by Theorem 4.2, we can obtain the convergence guarantee for the general drifted VESDE with reverse SDE.

Theorem 5.2. Assume Assumption 3.1, 4.1, 5.1. Let ¯ D defined in Theorem 4.2, ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k , τ = T 2 and β t ∈ [1 , t 2 ] . Then, we have that

Proof . By the data processing inequality, we know that

Combined with Theorem 4.2 and Lemma C.5, we achieve the final result.

C.1 The Sample Complexity for Drifted VESDE

As shown in Theorem 5.2, the general convergence guarantee is

In this section, we provide the sample complexity for different β t and τ .

The results of τ = 1 and β t = 1 . When considering β t = 1 and τ = 1 , the drifted VESDE becomes VPSDE and m T = exp( -T ) , which indicates T is a logarithmic term and the dominated term of the convergence guarantee is the discretization term ˜ O ( R 2 √ d ¯ γ K /σ 4 δ ) . To make this term smaller than ϵ TV , we require ¯ γ K ≤ σ 8 δ ϵ 2 TV / ( R 4 d ) . To make sure that W 2 2 ( q 0 , q δ ) ≤ ϵ 2 W 2 , we require

σ 2 δ ≤ ϵ 2 W 2 d + R √ d . Then, we achieve the final sample complexity

We note that this results is exactly the same with Chen et al. [2023c], which means our drifted VESDE covers VPSDE setting.

C.1.1 The results of τ = T 2 with different β t

In this part, we analyze the influence of β t under setting τ = T 2 and show the power of our drifted VESDE.

The aggressive β t ( τ = T 2 ). When considering aggressive β t = t α 1 where 2 ≥ α 1 ≥ 1 + ln( T -ln( T )) / ln( T ) , √ m T /σ T ≥ exp( -T/ 2) , which means T is a logarithmic term and can be ignored. After that, the analysis process is exactly the same with the above VPSDE setting, and we achieve the sample complexity

Defined by R q τ ∞ K,R 0 the output R q τ ∞ K projected onto B(0 , R 0 ) for R 0 = ˜ Θ( R ) . Following exactly the same proof process of Chen et al. [2023c] (Corollary 5), we have that

The conservative β t = t . In this case, the first term is ¯ D/T . To make this term smaller than ϵ TV , we require T ≥ ¯ D/ϵ TV . For the stepsize, we require ¯ γ K ≤ σ 8 δ ϵ 2 TV / ( R 4 dT 4 ) , which means the sample complexity is

with sample complexity

■

The most conservative β t = 1 . In this case, m T = exp( -1 /T ) and σ 2 t = τ (1 -m 2 t ) = T 2 (1 -exp( -2 /T ) . When T is large enough, m T = Θ(1) and σ 2 T = T , which indicates the first term is ¯ D/ √ T . To make this term smaller than ϵ TV , we require T ≥ ¯ D 2 /ϵ 2 TV . For the second term, we require ¯ γ K ≤ σ 8 δ ϵ 2 TV / ( R 4 dT 3 ) Then, the final complexity is

C.1.2 The results of τ = T with different β t

At the remaining part, we show the sample complexity of setting τ = T with different β t .

The results for setting τ = T and β t = t . For this setting, as shown in Lemma C.5, the discretization error is

Furthermore, we choose an aggressive β t , which indicates T is a logarithmic term. Then, by choosing σ 2 δ ≤ ϵ 2 W 2 ( d + R √ d ) and ¯ γ K ≤ σ 8 δ ϵ 2 TV ln 3 ( ¯ D/ϵ TV ) / ( R 4 d ) , we obtain the sample complexity

The results for setting τ = T and β t = 1 . In this setting, the reverse beginning error is bounded by ¯ D/ √ T , which indicates T ≥ ¯ D 2 /ϵ 2 TV . For the discretization term, we require ¯ γ K ≤ σ 8 δ ϵ 2 TV / ( R 4 dT 2 ) . Then, the sample complexity is bounde by

D The Proof of the Convergence Guarantee in the Unified Framework

In this work, we introduce an indicator i ∈ { 1 , 2 } for σ T -t K to represent different β t . We use τ = T 2 as an example. When β t = t 2 is aggressive, we choose i = 1 , η = 1 and σ -2 ( i = 1) ≤ 1 + ¯ β 3

When β t = t is conservative, we choose i = 2 , η ∈ [0 , 1) and σ T -t K ( i = 2) ≤ 1 τ + δ 2 . In the proof process of Lemma 6.3, Lemma D.1, Lemma D.2 and Lemma D.3, we ignore the indicator i since this lemma does not involve the specific value of σ 2 T -t K ( i ) . Before the proof of this section, we first recall the stochastic flow of the reverse process for any x ∈ R d and s, t ∈ [0 , T ] with t ≥ s :

T -t K τ δ . -2 ¯ β

and the interpolation of its discretization for any k ∈ { 0 , ..., K } and t ∈ [ s k , t k +1 ) :

where s k = max( s, t k ) . To deal with the discretization error, we use the approximation technique used in Bortoli [2022]. Hence, we introduce the tangent process:

Then, we discuss the interpolation formula, which is used to control the discretization error. Proposition 1. For s, t ∈ [0 , T ) with s < t , any k ∈ { 0 , ..., K } and ( ω v ) v ∈ [ s,T ] , we define that

where s k = max( s, t k ) and u ∈ [ s k , t k +1 ) . Then, for any x ∈ R d , we have that

where for any u ∈ [0 , T ) , there exists a k ∈ { 0 , ..., K } satisfies u ∈ [ s k , t k +1 ) .

For reverse SDE, the augmentation is similar to Bortoli [2022] (Appendix E). When η = 0 , the stochastic extension of the Alekseev-Gröbner formula [Del Moral and Singh, 2022] degenerates into the original version [Alekseev, 1961]. After that, we control the tangent process.

Lemma 6.3. Assume Assumption 3.1 and 4.1. For ∀ s ∈ [0 , t K ] and x ∈ R d , we have

Proof . Using the definition of the tangent process and Lemma E.1, we have

Using Lemma F.1, we have

Note that ∇ Y s,s = I , we get

When we assume ∥ ∥ ∇ log q 2 t ( x t ) ∥ ∥ ≤ Γ /σ 2 t , we know that

Using Lemma F.1, we have

Then we have

Thus we complete our proof.

■

After bounding the gradient of the tangent process, the remaining term is ∥ ∆ b ∥ :

where b ( a ) = b and b ( d ) = ¯ b . Moreover,

We then control ∥ ∆ ( a,b ) b ∥ , ∥ ∆ ( b,c ) b ∥ , ∥ ∆ ( c,d ) b ∥ separately. In this section, ∥ ∆ ( c,d ) b ∥ = 0 since we assume that the accurate score function is achieved. For ∥ ∥ ∆ ( a,b ) b u ( ω ) ∥ ∥ , we have the following lemma.

Lemma D.1. For s, u ∈ [0 , T ) such that u ≥ s, u ∈ [ s k , t k +1 ) and ω = ( ω v ) v ∈ [ s,T ] we have

Proof . Without loss of generality, we assume s ≤ t k . Then

For ∥ ∥ ∆ ( b,c ) b u ( ω ) ∥ ∥ , we have the following lemma.

Lemma D.2. For s, u ∈ [0 , T ) such that u ≥ s, u ∈ [ s k , t k +1 ) and ω = ( ω v ) v ∈ [ s,T ] we have

Proof . Without loss of generality, we assume s ≤ t k . In this case s k = t k , Then

Then the proof is complete.

■

We need to control the reverse process when dealing with ∆ b . The following lemma shows an upper bound for the reverse Y k .

Lemma D.3. Assume Assumption 3.1 ,Assumption 4.1, and there exists δ > 0 such that γ k β T -t k σ 2 T -t k ≤ δ ≤ 1 / 28 for any k ∈ { 0 , · · · , K } , then we have

where

and µ is an arbitrary positive number which makes A > 0 . In particular, if δ ≤ 1 / 28 , then

Proof . Recall the discretization of the backward process (the explicit form of Equation (6))

where { Z k } k ∈ K are independent Gaussian random variables. It is clear that γ 1 ,k ≤ γ 2 ,k ≤ 2 γ 1 ,k , and using Lemma E.1 we have

where the first equality follows that we assume the accurate score function. For any µ > 0 . Again using Lemma E.1, we have

Combining the results above, we have

If we denote δ k = γ 1 ,k /σ 2 T -t k and notice the fact that m t ∈ [0 , 1] , σ 2 t ∈ [0 , τ ] , η ∈ [0 , 1] , then we have

We also have that

where the last inequality follows that γ k = exp( -T ) , β T -t k γ k ≤ 1 / 2 for small enough stepsize, and e ω -1 ≤ 2 ω for any ω ∈ [0 , 1 / 2] . We get δ k ≤ 2 γ k β T -t k /σ 2 T -t k ≤ 2 δ . Thus

Hence, we have

We denote A = 2 -14 δ -4(1 + δ ) µR and B = 16 R 2 δ +4(1 + δ ) R µ +4 τd , then

Notice that E [ ∥ Y 0 ∥ 2 ] = dτ and if E [ ∥ Y k ∥ 2 ] ≥ B/A it is decreasing, if E [ ∥ Y k ∥ 2 ] ≤ B/A we have E [ ∥ Y k +1 ∥ 2 ] ≤ B/A + δB . so

Notice that when δ ≤ 1 / 28 ,if we choose µ = 1 / (4(1 + δ ) R ) , A ≥ 1 / 2 , and

Then, the proof is complete.

The following lemma shows a discretization error in the k -the interval.

Lemma D.4. Assume Assumption 3.1,Assumption 4.1 and γ k β T -t k /σ 2 T -t k ≤ 1 / 28 for any k ∈ { 0 , · · · , K -1 } . Then for any k , t ∈ [ t k , t k +1 ] and i ∈ { 1 , 2 } , we have that

where L i ( τ ) = ¯ γ K κ i ( τ )( 64 σ 2 T -t K ( i ) + 8 τ ) U 0 ( τ ) + 64 R 2 ¯ γ K κ i ( τ ) σ 2 T -t K ( i ) + 4 d , ¯ γ K , κ i ( τ ) is defined in Lemma D.5 and U 0 ( τ ) is defined in Lemma D.3.

Proof . Recall the discretized backward process

where Z is a standard Gaussian random variable. By directly calculating, we have that

By Lemma E.1 and accurate score function assumption,

So we have that

By e 2 w -1 ≤ 1 + 4 w for any w ∈ [0 , 1 / 2] and γ k sup v ∈ [ T -t k +1 ,T -t k ] β v /σ 2 v ≤ 1 / 28 for any k ∈ { 0 , ..., K -1 } , we have

■

for ρ = 1 , 2 . And using Lemma D.3 and Lemma F.2 we have

We denote L i ( τ ) = ¯ γ K κ i ( τ )( 64 σ 2 T -t K ( i ) + 8 τ ) U 0 ( τ ) + 64 R 2 ¯ γ K κ i ( τ ) σ 2 T -t K ( i ) + 4 d for i ∈ { 1 , 2 } and the proof is complete. ■

Lemma D.5. Assume Assumption 3.1 and Assumption 4.1, γ k sup v ∈ [ T -t k +1 ,T -t k ] β v /σ 2 v ≤ 1 / 28 for any k ∈ { 0 , ..., K -1 } . Let ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k , κ i ( τ ) = max { ¯ β, T 2 T -1+ i } σ -2 T -t K ( i ) , and

for i ∈ { 1 , 2 } . Then, for any s, u ∈ [0 , t K ] with u ≥ s and i ∈ { 1 , 2 } , we have

where ¯ Y s,s ∼ N(0 , I ) .

Proof . Combining Lemma D.1, Lemma D.2 and the exact score function, we get

For any u ∈ [ T -t K , T ] , using Lemma F.3 we have β u /σ 2 u ( i ) ≤ κ i ( τ ) . Hence,

Combining this with Lemma D.3 and Lemma D.4,

We denote C i ( τ ) = 2(2 + R 2 )( R + U 1 / 2 ( τ )) + 2 L 1 / 2 ( τ ) τ 3 / 2 (1 + R 2 ) , for i ∈ { 1 , 2 } , then we have

0 i

■

Lemma D.6. Assume Assumption 3.1 and Assumption 4.1, γ k sup v ∈ [ T -t k +1 ,T -t k ] β v /σ 2 v ≤ 1 / 28 for any k ∈ { 0 , ..., K -1 } . Let ¯ γ K = argmax k ∈{ 0 ,...,K -1 } γ k , γ K = δ , and δ ≤ 1 / 32 . Then

where C i ( τ ) , κ i ( τ ) for i ∈ { 1 , 2 } are the same terms to Theorem 6.1.

Proof . By Proposition 1 we have

Then by definition of Wasserstein distance, we have

(1) If

η

= 1

(the reverse SDE), choosing

β

t

=

t

,

W

1

R

■

Theorem 6.1. Assume Assumption 3.1 and 4.1, δ ≤ 1 / 32 and γ k sup v ∈ [ T -t k +1 ,T -t k ] β v /σ 2 v ≤ 1 / 28 for ∀ k ∈ { 0 , ..., K -1 } . Let γ K = δ . Then, for ∀ τ ∈ [ T, T 2 ] :

2

(

q

τ

∞

K

, q

0

is bounded by

where κ 1 ( τ ) = T 2 (1 /τ + ¯ β/δ 3 ) and C 1 ( τ ) is linear in τ 2 .

(2) If η = 0 (PFODE), choosing a conservative β t (Assumption 3.1), W 1 ( R q τ ∞ K , q 0 ) is bounded by

where κ 2 ( τ ) = T ( 1 /τ + ¯ β/δ 2 ) and C 2 ( τ ) is linear in τ 2 .

Proof . To obtain the convergence guarantee, we need to control three error terms:

For the second term, we define ( Y x 0 ,t ) t ∈ [0 ,T ] and ( Y y 0 ,t ) t ∈ [0 ,T ] be the reverse processes with initial condition x and y . Then we have

where z λ = λx +(1 -λ ) y . In this work, we choose x ∼ q τ ∞ and y ∼ q 0 P T . Combined with the above inequality, Theorem 4.2 and Lemma 6.3, we know that:

)

For the last term, we use exactly the same process with Bortoli [2022] with bounded σ 2 T -t K :

where the second inequality follows that σ + τm T -t K = τ .

2 T -t K ■

In the end of the section, we provide the proof of Corollary 6.2.

Corollary D.7. Assume Assumption 3.1, 4.1 and ∥ ∥ ∇ 2 log q t ( x t ) ∥ ∥ ≤ Γ /σ 2 t . Let η = 0 (reverse PFODE), δ ∈ (0 , 1 / 32) , τ = T 2 , β t = t and κ 2 ( τ ) , C 2 ( τ ) defined in Theorem 6.1, we have

Proof . The proof of this corollary is almost identical to the proof of Theorem 6.1. We just need to replace the first bound for the tangent process in Lemma 6.3 by the second bound. ■

E Lemmas for the Logarithmic Density

In this section, we introduce auxiliary lemmas to control the gradient and Hessian of the logarithmic density under the manifold hypothesis. Lemma E.1, Lemma E.2 and Lemma E.3 come from Lemma C.1, Lemma C.2, and Lemma C.5 of Bortoli [2022]. Since these lemmas do not involve the relationship between m t and σ t , we can directly use the results from Bortoli [2022]. Following Bortoli [2022], we also define a empirical version of q 0 with N datapoints, i.e. q N 0 = (1 /N ) ∑ N k =1 X k , with { X k } N k =1 ∼ q ⊗ N 0 . We denote by ( q N t ) t> 0 such that for any t > 0 the density w.r.t. the Lebesgue measure of the distribution of X N t , and when N → + ∞ , q N t = q t .

Lemma E.1. Assume Assumption 4.1. Then for any t ∈ (0 , T ] and x t ∈ R d we have that

Lemma E.2. Assume Assumption 4.1. Then for any t ∈ (0 , T ] , x t ∈ R d and M ∈ M d ( R d )

In addition, we have

In addition, we have

The following lemma shows that the derivatives up to the fourth order are uniformly bounded since τ ∈ [ T, T 2 ] . Thus we can use the stochastic extension of the Alekseev-Gröbner formula [Del Moral and Singh, 2022].

Lemma E.3. Assume Assumption 4.1. Then, there exists ¯ C ≥ 0 such that for any t ∈ (0 , T ] we have

The following lemma shows that ∥ ∂ t ∇ log q t ( x t ) ∥ is bounded. The proof before using the relationship between σ t and m t is identical compared to Lemma C.3 in Bortoli [2022]. For the sake of completeness, we also give the proof process of this part.

Lemma E.4. Assume Assumption 4.1. Then for any t ∈ (0 , T ] and x t ∈ R d we have

Proof . Let N ∈ N and t ∈ (0 , T ] . We denote for any x ∈ R d , q N t ( x ) = ¯ q N t ( x ) / ( 2 πσ 2 t ) d/ 2 with

Next we denote f k t ≜ log e k t . Then we have

Therefore we have

In what follows, we provide upper bounds for | ∂ t f k t -∂ t f j t | , ∥∇ f k t -∇ f j t ∥ and ∂ t ∇ f k t . First we notice that ∇ f k t ( x ) = -( x -m t X k ) /σ 2 t , and using m t ≤ 1 we get

and

Notice the fact that ∂ t σ 2 t = -2 τm t ∂ t m t = 2 β t m 2 t and ∂ t m t = -β t τ m t , combined with the above equality, we know that

where the last equality holds that τm 2 t + σ 2 t = τ . The rest of the proof is identical to the Lemma C.3 in Bortoli [2022].

So using m t ≤ 1 we have

Now we compute ∇ ∂ t f k t ( x ) for any x ∈ R d

So we can bound the norm of it by

Combining results above we get for any x ∈ R d

Note that

and the proof is complete.

In the following lemma, similar to Chen et al. [2023c], we obtain a better control on the time discretization error instead of controlling ∥ ∂ t ∇ log q t ( x t ) ∥ for ∀ x t ∈ R d .

Lemma E.5. Assume Assumption 4.1 and X t satisfies the forward process Equation (3) . Define L = max t ∈ [0 ,T -δ ] ∥ ∥ ∇ 2 log q T -t ( Y t ) ∥ ∥ ≤ ( 1 + R 2 ) /σ 4 δ , then we have that

Proof . Due to the property of the forward process, we know that if S : R d → R d is the mapping S ( x ) := exp( -( t -t k )) x , then q T -t k = S # q T -t ∗ normal ( 0 , τ ( 1 -exp( -2 ∫ t k +1 t k β s /τ d s ) )) Similar to Chen et al. [2023c], we define α = exp [ ∫ t k +1 t k β s τ d s ] = 1 + O (¯ γ K ) and σ 2 = τ ( 1 -exp( -2 ∫ t k +1 t k β s /τ d s ) ) = O ( τ ¯ γ K ) . Then we can use Lemma C.12 of Lee et al. [2022] to obtain

The last inequality follows Lemma F.4 and the fact that

F Auxiliary Lemmas

Lemma F.1. For any s, t ∈ [0 , T ] we have

■

Proof . We directly compute

Similarly

(2) if i = 2 , then

where the first inequality follows from for any a ≥ 0 , exp[ -a ] ≥ 1 -a ; the second inequlity follows from Assumption 3.1 and δ ≤ 1 .

where the first inequality follows from for any a ≥ 0 , 1 / (1+exp[ -a ]) ≤ 1+1 /a , the second inequality follows from Assumption 3.1. It is easy to check that σ -2 u ( i ) ≤ σ -2 T -t K ( i ) , ∀ u ∈ [ T -t K , T ] .

■

Using the bound on σ -2 T -t K ( i ) immediately yields the following control of β u /σ 2 u ( i ) .

Lemma F.3. Assume Assumption 3.1. Then, we have for any u ∈ [ T -t K , T ] : (1) if i = 1 , then

■

Lemma F.2. Assume Assumption 3.1. For i ∈ { 1 , 2 } , we have σ 2 T -t K ( i ) ≤ 2 δ and σ -2 u ( i ) ≤ σ -2 T -t K ( i ) ≤ 1 τ + ¯ β δ 4 -i , ∀ u ∈ [ T -t K , T ] .

Proof .

Generally speaking, T ≥ ¯ β ≥ 1 . Hence, We can further simplify the above inequality by removing max .

In the rest of this section, we provide the useful lemma to achieve polynomial sample complexity for VE-based models with reverse SDE. As shown in Lemma E.1, we also need to control E [ ∥ X t ∥ 2 ] in the forward process. The following lemmas shows that this term is bounded by the R 2 and exploding variance.

Lemma F.4. Suppose that Assumption 4.1hold. Let ( X t ) t ∈ [0 ,T ] denote the forward process Equation (3) . Then, for all t ≥ 0 ,

Proof . As shown in Equation (4),

Lemma F.5 (movement bound for VESDE) . Let ( X t ) t ∈ [0 ,T ] denote the forward process Equation (3) . For 0 ≤ s < t with δ := t -s , if δ ≤ 1 , then

Proof .

Similar to Chen et al. [2023c], we can also show that if we do forward process for time δ , q δ will be close to q 0 in W 2 distance.

Lemma F.6. Suppose Assumption 4.1 holds. Let ϵ W 2 > 0 . If β 2 t = t 2 and τ = T 2 , we choose the early stopping parameter δ ≤ ϵ 2 / 3 W 2 ( d + R √ d ) 1 / 3 . If β t = t and τ = T , we choose δ ≤ ϵ W 2 ( d + R √ d ) 1 / 2

. If consider pure VESDE (SMLD) (Equation (2) ) with σ 2 t = t , we choose δ ≤ ϵ 2 W 2 d . Then we have W 2 ( q δ , q 0 ) ≤ ϵ W 2 .

Proof . For the forward process Equation (3), we know that X t := m t X 0 + σ t Z , where Z ∼ normal(0 , I d ) is independent of X 0 and m t ≤ 1 . Hence, for δ ≲ 1 ,

For β t = t 2 and τ = T 2 , we have that

Hence, we can take δ ≤ ϵ 2 / 3 W 2 ( d + R √ d ) 1 / 3 . For β t = t and τ = T , we have that

Hence, we can take δ ≤ ϵ W 2 ( d + R √ d ) 1 / 2 . For pure VESDE (Equation (2)) with σ t = t , we have

■

G Additional Synthetic Experiments

In this section, we do synthetic experiments to show the power of our new forward process with small drift term in different setting.

G.1 The Synthetic experiments with accurate score function

In this section, we do numerical experiments on 2 -dimension Gaussian distribution to show the power of our new VESDE forward process in balancing different error sources.

Experiment Setting. We set the mean of target distribution E [ q 0 ] = [6 , 8] , the covariance matrix and the reverse beginning distribution is

N (0 , T 2 I ) . We choose uniform stepsize γ k = h, ∀ k ∈ [ K ] where h ∈ { 0 . 005 , 0 . 01 , 0 . 02 , 0 . 04 } . For score functions, we directly calculate the ground truth score function instead of learning it by the score matching objective. We calculate the KL divergence between the generation distribution and target distribution q 0 as the experiments.

The implementable algorithm. We choose three different VESDE forward processes in the experiments: (1) aggressive β t = t 2 with τ = T 2 ; (2) conservative β t = t with τ = T 2 and (3) VESDE without drift term Equation (2) with σ 2 t = t 2 . After determining the forward process, we run the reverse SDE with the above γ k , k ∈ [ K ] . For the discretization scheme, we choose two common method: exponential integrator (EI) [Zhang and Chen, 2022] and Euler-Maruyama (EM) discretization [Ho et al., 2020].

Observations. The experimental results are shown in Figure 2. We note that the red line (EI, VESDE without drift, σ 2 t = t 2 ) and orange line (conservative drift VESDE, β t = t and τ = T 2 ) has a similar trend. Furthermore, the conservative drift VESDE has better performance compared to pure VESDE without drift term. Hence, our new forward process is representative enough to represent current VESDE, as discussed in Section 3.1.

The experimental results also support our theoretical results and show the power of the new forward process in balancing different error terms. As shown in Figure 2, the process with aggressive β t = t 2 with small drift term achieves the best and second performance in EI and EM discretization since it can balance the reverse beginning and discretization. The third best process is conservative β t = t with the small drift term. The reason is that though it can not achieve a exp( -T ) forward process guarantee, it also has a constant decay on prior information, as shown in Section 3.1. This decay slightly reduces the effect of the reverse beginning error. The worse process is VESDE without drift term since it is hard to balance different error sources. Our experimental results also show that EI discretization is better than EM discretization.

G.2 The Synthetic experiments with approximated score function

In this section, instead of using an accurate score function, we train an approximated score function on the pure VESDE (Equation (2)) without drift term on two synthetic datasets: multiple Swiss rolls and 1-D GMM. Then, for the drift VESDE, we do not train the approximated score corresponding to Equation (3); we directly use the approximated score learned by pure VESDE and show that the drift VESDE can improve the generated distribution without the training process.

Datasets. The 1-D GMM distribution contains three modes:

For multiple Swiss rolls, we use a similar code compared to Listing 2 of Lai et al. [2023], except Line 6. We change Line 6. to data /=10. to obtain a larger variance dataset. Each dataset contains 50000 datapoints.

The implementable algorithm. In this subsection, we choose two forward processes: (1) conservative β t = 1 with τ = T ; (2) pure VESDE without drift term (Equation (2)) with σ 2 t = t . To match

our analysis, we choose two sampling methods for the reverse process: Euler-Maruyama method for reverse SDE and RK45 ODE solver for the reverse PFODE method.

Wenote that although aggressive setting β t = t and τ = T has shown its power in theory (Lemma C.2) and the experiments with accurate score (Figure 2), other sampling issues may arise in practice. We leave the experimental exploration for drift VESDE with aggressive β t as a future work.

The training detail. For each dataset, we train a score function with pure VESDE (Equation (2), σ 2 t = t ). We train for 200 epochs with batch size 200 and learning rate 10 -4 . For both training and inference, the start time is δ = 10 -5 . For the conservative VESDE, we directly adapt the checkpoint learned by the pure VESDE since the conservative drift VESDE has a similar trend compared to pure VESDE, as shown in Figure 2. The above experiments are runned over 5 random seed and we present the average over these seeds in Table 1.

The above experiments are conduct on a GeForce RTX 4090. It takes 25 minutes to train a score function of pure VESDE.

Observation. We do experiments with T = 100 and lager T = 625 and these two choice show similar phenomenon. In this paragraph, we first use T = 100 as an example to discuss the results. As shown in Table 1, the conservative drift VESDE has smaller KL divergence compared to pure VESDE under all sampling methods and datasets. From Figure 1 and Figure 4, it is clear that pure VESDE has low density on the Swiss roll except the center one, which means that though pure VESDE can deal with small E [ q 0 ] , it is hard to deal with large dataset variance Cov [ q 0 ] , as we discuss in Section 4. For conservative drift VESDE ( β t = 1 and τ = T ), as we discuss in Section 3.1, there is a constant decay on the prior information E [ q 0 ] and Cov [ q 0 ] , which is helpful in deal with large dataset mean and variance. The experimental results support our augmentation. Figure 1 (c), Figure 4 (c) and Figure 5 (c) show that the density of the generated distribution is more uniform compared to pure VESDE, which means that the drift VESDE can deal with large dataset mean and variance.

We also do experiments with larger T = 625 . As we discuss in Section 4, larger T will reduce the influence of the prior data information and have greater generated distribution, as shown in Figure 4 (c) and Figure 4 (e). The experiments of 1D-GMM (Figure 5) show a similar phenomenon compared to the multi Swiss rolls.

G.3 The Real-World Experiments on CelebA 256

After achieving great performance under the synthetic data, we show that our conservative drifted VESDE can improve the results of pure VESDE without training.

Setting. In this experiment, we adapt well-known VESDE implementation [Song et al., 2020b] and do experiments on CelebA datasets (size: 256 ∗ 256 ∗ 3 ). More specifically, we use ve / celebahq_256_ncsnpp_continuous checkpoints provided by [Song et al., 2020b] and modify the sampling process strictly according to our drifted VESDE. To do a fair comparsion, we fix the random seed and use the reverse PFODE process. Then, we generate 10000 face images to calculate the metrics. We note that when using this checkpoint and pure VESDE pipeline provided by [Song et al., 2020b], the models would generate almost pure noise with a certain probability. Hence, we use an aesthetic predictor [Schuhmann et al., 2022] (aesthetic score ≥ 5 . 5 ) to filter the generated images to ensure that the images are clear faces.

(a) Original Figure

𝑇 = 100

)

(c) Drifted VESDE (

𝑇 = 100

)

𝑇 = 625

)

(e) Drifted VESDE (

𝑇 = 625

)

Swissroll

swissroll, VESDE,ode_sampler

swissroll, VESDE,ode_sampler

swissroll,Drift_ VESDE,ode_sampler

(a) Original Figure

𝑇 = 625

)

(c) Drifted VESDE (

𝑇 = 625

)

Mixture of Gaussian

MOG,VESDE,ode_sampler

0.30

0.35

0.30

0.30

0.25

0.25

0.25

0.20

Asuaa

0.20

Asuaa

0.20

AuIsu

0.15

8

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

-10

8

-6

=2

Value

0

2

0.00

-10.0

7.5

-5.0

2.5

Value

0.0

5.0

7.5

0.00

-10

Value

10

Discussion. From the qualitative perspective, as shown in Figure 3 (Figure 6 and 7), the images generated by our drifted VESDE have more detail (such as hair and beard details). On the contrary, since pure VESDE can not deal with large variance, the images generated by pure VESDE appear blurry and unrealistic in these details. From the quantitative results, our drifted VESDE achieves aesthetic score 5.813, and IS 4.174, which is better than the results of baseline pure VESDE (aesthetic score 5.807 and IS: 4.082). In conclusion, the real-world experiments show the potential of our drifted VESDE.

We note that the goal of these experiments is to show that our conservative drifted VESDE is plugand-play without training instead of achieving a SOTA performance. Hence, we focus on the relative improvement compared to the baseline [Song et al., 2020b]. There are two interesting empirical future works. For the conservative drifted VESDE, we will do experiments on the SOTA pure VESDE models [Karras et al., 2022] and improve their results without training. For the aggressive drifted VESDE, since this process makes a larger modification compared with the conservative one, we need to train a new score function instead of directly using a pre-train one to achieve better results.

(b)DriftedVESDE

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a new drifted VESDE forward process and achieve the first polynomial complexity for reverse SDE (Corollary 5.4). For reverse PFODE, we propose the firs t quantitative convergence guarantee for VESDE (SOTA) (Theorem 6.1). We also do synthetic experiments to support our results (Section 7).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss future work and limitation at Section 8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions, Theorem, Corollary, and proof sketch have been clearly stated in the main content. The detailed proof appears in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

We has shown all experiments detail including dataset and training detail in Appendix G. Furthermore, we discuss why these experiments results support our theoretical results in Section 7.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: As a theoretical work, we only do simple synthetic experiments to support our results. All detail and the used checkpoint are shown in Appendix G.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We has shown all experiments detail including dataset and training detail in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results of Figure 2 and Table 1 are calculated over 5 random seed.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have shown the compute works and computation time in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have checked the code of ethics and make sure that our work satisfies the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our work at the end of main paper. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDiffusion models are powerful tools in generative modeling.  As the paper pointed out, very few theoretical works in the literature\nconsider variance exploding diffusion models. Among those works, forward convergence rate $1/\\text{poly}(T)$ is achieved compared to $\\exp(-T)$ from the variance preserving models.  This paper proposes a drifted variance exploding diffusion model that allows a faster $\\exp(-T)$ forward convergence rate.  With this process, the polynomial sample complexity for a series of variance exploding models is achieved under the manifold hypothesis.  In addition to the reverse SDE, probability flow ODE is a popular alternative in the literature. The paper considers a more general setting and proves a convergence guarantee with probability flow ODE.\n\nSTRENGTHS:\n(1) The paper proposes a so-called drifted variance exploding forward process and to the best of my knowledge this is new. \n\n(2) When the data is supported on a compact set, the paper manages to derives convergence guarantees in total variation distance, and 1-Wasserstein distance.\n\n(3) The analysis seems to be rigorous.\n\nWEAKNESSES:\n(1) The model seems to be an interplay between the VE-SDE model and VP-SDE model  in the literature in the sense that when $\\tau=1$, it recovers the VP-SDE model and then $\\tau=\\infty$, it recovers the VE-SDE model. To better compare the model with the VP-SDE and VE-SDE models in the literature, I am wondering whether it is better to assume $\\tau=[1,\\infty]$ and for a fixed $\\tau$, to see what is the convergence guarantees and how it depends on $\\tau$? If you do that, will you find the optimal range of $\\tau$ to be $[T,T^{2}]$ which is what proposed in the paper? My guess is that for given $\\beta_{t}$, the optimal choice of $\\tau$ should depend on $\\beta_{t}$ instead of being in the range $[T,T^{2}]$.\n\n(2) Since the model proposed is a new model, it would be even more convincing if the paper can include some experiments beyond synthetic experiments.\n\nQUESTIONS:\n(1) You studied both total variation distance and 1-Wasserstein distance. Since your data is supported on a compact domain, would it be possible to study more general $p$-Wasserstein distance guarantees?\n\n(2) After Assumption 3.1. you wrote that the choice $\\tau\\in[T,T^{2}]$ is used to guarantee the exploding variance of the forward process. This is not that accurate. I understand when $\\tau$ is large, the mean-reverting effect of the forward SDE gets very weak, and the variance tends to grow. But that does not explain why you need $\\tau\\leq T^{2}$. I think you should also add some discussions on why you need $\\tau\\leq T^{2}$.\n\nLIMITATIONS:\nLimitations have been discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the Variance Exploding diffusion model under the manifold hypothesis. By a slight modification to the VESDE process, the authors propose a method whose convergence guarantees are better than prior best known rates in this regime.\n\nSTRENGTHS:\nThe rate obtained is state-of-the-art for Variance Exploding models, which are notorious for being difficult to analyze.\n\nAlthough the paper borrows parts of its analytic framework from the work of Bortoli et al. (2022), the adaptation of the analysis in this context appears to be non-trivial.\n\nThe paper is clearly written and the exposition of the proof is overall quite clear.\n\nWEAKNESSES:\nThe experiments in the main text and appendix are not particularly thorough. This is acceptable since the primary claimed contribution by this article is theoretical, but some kind of practical benchmark might also be beneficial.\n\nThe dimension and diameter dependence are quite severe. Is there any sense of how close these parameters are to their optimal values?\n\nQUESTIONS:\nThe final rates are only given in total variation. Is it possible to improve the metric to KL or to state W2 bounds with similar complexities? What is the analytical challenge otherwise?\n\nTheorems 6.1 and Corollary 6.2 could have their exposition simplified in the main text; the intuition provided in the subsequent section is much clearer.\n\nMinor:\n\nThe font used for \"KL\" is not consistent in the equations in the appendix.\n\n97: about data -> about the data\n\n102: score function -> the score function\n\n107: with strong LSI -> with a strong LSI\n\n108: assume the Lipschitz score -> assume the score is Lipschitz\n\n108: first work focus -> first work to focus\n\n138: introduces -> introduce\n\n187: \"reversing\" does not make sense here. Perhaps \"initial distribution of the reverse process\" is meant\n\n216: \"does reverse\" -> \"which reverse the process\"\n\n345: \"support\" -> \"supports\"\n\n364: \"support our\", \"show that\" -> \"supports our\", \"shows that\"\n\nLIMITATIONS:\nThe authors have already addressed all major limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on variance exploding (VE) based diffusion models and proposes a drifted VESDE forward process with an unbounded diffusion coefficient. This choice of coefficients allows an exponential-decay forward convergence rate, and the authors establish the first polynomial sample complexity for VE-based models with reverse PFODE. Moreover, the authors propose a tangent-based unified analysis framework with reverse SDE and PFODE and prove the first quantitative guarantee for SOTA VE-based models with reverse PFODE.\n\nSTRENGTHS:\nIn terms of originality, this paper proposes a new variance exploding (VE) based diffusion model and establishes the corresponding convergence guarantees. The theoretical results are solid. Moreover, this paper is well-organized and clearly written.\n\nWEAKNESSES:\n1. The convergence guarantees for VE-based models with reverse PFODE are relatively weak. For example, in Assumption 3.1, the choice of $\\beta_t$ is more conservative for reverse PFODE; In Theorem 6.2 part (2), the last term is $\\bar{D}/\\tau$ instead of $\\bar{D} e^{-T/2} / \\sqrt{\\tau}$ in part (1). Does this mean that the $e^{-T}$ forward convergence rate can only achieved by the reverse SDE?\n\n2. The numerical results only include synthetic experiments.\n\nQUESTIONS:\n1. In Assumption 3.1, why for reverse SDE and PFODE, the choices of $\\beta_t$ are different? Could the authors provide some intuitive explanation?\n\n2. In Theorem 5.2, the sample complexity has the same dependence on $\\epsilon_{W_2}$ and $\\epsilon_{TV}$ as that in Chen et al. (2023c). However, the dependence on $d$ is worse. Is it a consequence of a more aggressive $\\beta_t$? \n\n3. Could the theoretical results reflect the superiority of VE-based diffusion models over VP-based diffusion models?\n\nLIMITATIONS:\nThe authors discuss future work and limitations in Section 8.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an analysis of the convergence of diffusion models under the manifold hypothesis in a similar setting as [1]. The main contribution is the extension of the analysis to the case of VESDE (Variance Exploding SDE) contrary to [1] which is limited to VPSDE (Variance Preserving). The rates obtained by the authors are better than the ones obtained in [1] (although in a different context). They also extend their analysis to ODE samplers which are notably more difficult to deal with than SDE samplers from a theoretical point of view. The decomposition of the error is the same as in [1] but with a more careful analysis of the tangent process (see Section 6 \"The Tangent-based Analysis Framework\"). Experiments in toy settings are presented. \n\n[1] De Bortoli -- Convergence of denoising diffusion models under the manifold hypothesis\n\nSTRENGTHS:\n* These results are the first results obtained for the convergence of diffusion models in the VESDE setting under the manifold hypothesis. \n\n* The analysis of the tangent process represents an improvement over the results of [1]. This is an interesting development in itself.\n\n* The introduction of the drifted VESDE (Equation 5) is interesting and represents a good avenue for future studies of the VESDE process. \n\n\n[1] De Bortoli -- Convergence of denoising diffusion models under the manifold hypothesis\n\nWEAKNESSES:\n* Experiments are only toyish. I actually don't think they benefit the paper. This is mostly theoretical work and I'm struggling to understand what point is made here. If this is to illustrate the validity of the samplers this is already well established. If the point of the experiment is to illustrate the benefit of drifted VESDE then I would have appreciated a more challenging setting (like CIFAR10 or Imagenet in image processing or larger models like Stable Diffusion). This does not require pretraining a large model since this is a modification of the sampler.\n\n* There is actually not a lot of discussion on how drifted VESDE relate to VESDE. Can one obtain convergence results for  VESDE (classical) based on drifted VESDE?\n\n* As of now, the analysis is limited to drifted VESDE. It would be interesting to analysis if the improved results can transfer to the VPSDE framework and improve on the results of [2].\n\n* The paper is easy to follow and clearly presented. \n\n[1] Karras et  al. -- Elucidating the Design Space of Diffusion-Based Generative Models\n\n[2] De Bortoli -- Convergence of denoising diffusion models under the manifold hypothesis\n\nQUESTIONS:\n* l.34 \"Furthermore, Karras et al. [2022] unify two processes and show that the optimal parameters of the general formula correspond to VESDE.\" --> Not clear what the authors are referring to here.\n\n* l.48 \"leads to a large reverse beginning error\" --> I disagree as one could argue that most of the error in diffusion models arises from the approximation of the score. The \"large reverse beginning error\" is a strong statement here. \n\n* One relevant work that is not discussed is [1]\n\n* l.296 \"Furthermore, we emphasize that our tangent-based unified framework is not a simple extension of Bortoli [2022].\" --> Can the authors provide more details here?\n\n* Since there is a one-to-one mapping between VESDE and VPSDE could the authors have leveraged this connection? There is also a connection with Stochastic Localization as pointed out by [2]. Below we explicit the connection between VESDE and VPSDE.\n\nAssume that $(X_t)_{t \\geq 0}$ satisfies a VESDE $\\mathrm{d} X_t = g(t) \\mathrm{d} B_t$ then we have that $(Y_t)_{t \\geq 0}$ given for any $t \\geq 0$ by $Y_t = \\exp[F(\\phi(t))] X_{\\phi(t)}$ satisfies $\\mathrm{d} Y_t  = F(\\phi)'(t) Y_t \\mathrm{d}_t  + \\exp[F(\\phi(t))] \\phi'(t)^{1/2} \\mathrm{d} B_t$.\n\n[1] Conforti et al. -- \"Score diffusion models without early stopping: finite Fisher information is all you need\"\n\n[2] Montanari -- \"Sampling, Diffusions, and Stochastic Localization\"\n\nLIMITATIONS:\nLimitations are not really addressed in Section 8 (\"Conclusion\") in the paragraph \"Future Work and Limitation\". I think a more in depth discussion of the benefits of VESDE and VPSDE is needed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}