Leveraging Drift to Improve Sample Complexity of Variance Exploding Diffusion Models

Authors: ruofeng  yang
zhijie  wang
bo  jiang
shuai  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:23:01

Abstract: Variance exploding (VE) based diffusion models, an important class of diffusion models, have shown state-of-the-art (SOTA) performance. However, only a few theoretical works analyze VE-based models, and those works suffer from a worse forward convergence rate $1/\text{poly}(T)$ than the $\exp{(-T)}$ of variance preserving (VP) based models, where $T$ is the forward diffusion time and the rate measures the distance between forward marginal distribution $q_T$ and pure Gaussian noise. The slow rate is due to the Brownian Motion without a drift term. In this work, we design a new drifted VESDE forward process, which allows a faster $\exp{(-T)}$ forward convergence rate. With this process, we achieve the first efficient polynomial sample complexity for a series of VE-based models with reverse SDE under the manifold hypothesis. Furthermore, unlike previous works, we allow the diffusion coefficient to be unbounded instead of a constant, which is closer to the SOTA models. Besides the reverse SDE, the other common reverse process is the probability flow ODE  (PFODE) process, which is deterministic and enjoys faster sample speed. To deepen the understanding of VE-based models, we consider a more general setting considering reverse SDE and PFODE simultaneously, propose a unified tangent-based analysis framework, and prove the first quantitative convergence guarantee for SOTA VE-based models with reverse PFODE.
We also show that the drifted VESDE can balance different error terms and improve generated samples without training through synthetic and real-world experiments.

Main paper: 
 introduction: 
 Recently, diffusion modeling has shown impressive performance in different areas [Ho et al., 2022 , Rombach et al., 2022 , Esser et al., 2024 , Li et al., 2024] . Diffusion models consist of two processes: the forward and reverse process. The forward process gradually converts data q 0 to Gaussian noise, which can be described by an intermediate marginal distribution sequence {q t } t∈[0,T ] . The reverse process sequentially predicts noise and removes it from data to generate samples.There are two common forward processes: (1) Variance preserving (VP) SDE and (2) variance exploding (VE) SDE. The VPSDE corresponds to an Ornstein-Uhlenbeck process, and the stationary distribution is N (0, I). The VESDE has an exploding variance in the forward process. In earlier √ T ) reverse beginning error. However, they introduce an additional exp (T ) in the discretization error term. Furthermore, their results do not allow unbounded β t , which is the key point of the optimal solution trajectory and used by the SOTA models [Karras et al., 2022 , Song et al., 2023] . Therefore, the following question remains open:Is it possible to design a VESDE with a faster forward convergence rate than 1/poly(T ) and achieve the polynomial sample complexity when the diffusion coefficient is unbounded?In this work, for the first time, we propose a new drifted VESDE forward process, which enjoys a faster forward convergence rate and allows unbounded coefficients. We first show that the drifted VESDE has similar trends but performs better than the original SOTA VESDE on synthetic data (Section 7). After that, we analyze the sample complexity of drifted VESDE under the realistic manifold hypothesis. The manifold hypothesis means the data q 0 is supported on a lower dimensional compact set M, and much empirical evidence shows that image and text dataset satisfy this hypothesis [Fefferman et al., 2016 , Pope et al., 2021 , Tang and Yang, 2024] . Furthermore, as shown in Section 2, the manifold hypothesis is more realistic than previous data assumptions since it allows the blow-up phenomenon of the score function at the end of the reverse process, which matches the empirical observation [Kim et al., 2021] . Under the manifold hypothesis, we prove that the drifted VESDE with a suitable larger β t balances the reverse beginning, discretization, and approximated score errors and achieves the first efficient polynomial sample complexity for VE-based models with reverse SDE.To better understand VE-based models, we analyze reverse SDE and PFODE simultaneously after obtaining polynomial complexity for reverse SDE. Despite the great performance, a few theoretical works consider reverse PFODE [Chen et al., 2023d,b, Gao and Zhu, 2024] , and these works either focus on VPSDE or have strong assumptions. Hence, we propose the tangent-based framework for VE-based models and achieve the first quantitative convergence for the SOTA VE-based models with reverse PFODE. In conclusion, we accomplish the following results under the manifold hypothesis: q 0 be the data distribution. Given X 0 ∈ R d , the forward process is defined by dX t = f (X t , t) dt + g(t) dB t , X 0 ∼ q 0 , where (B t ) t≥0 is the standard Brownian motion in R d , f (X t , t) is a drift coefficient and g(t) is a diffusion coefficient. Let q t be the density function of X t at time t. With a suitable choice of drift and diffusion terms (e.g. Section 3.1 and 3.2), the forward process gradually converts the data distribution into Gaussian noise. More specifically, the conditional distribution X t |X 0 is exactly N (m t X 0 , σ2 t I) given X 0 , where m t is determined by the drift term and σ 2 t is determined by the diffusion term. The VPSDE forward process. Let {β t } t≥0 be a non-decreasing sequence with bounded range [1/ β, β] . The VPSDE has the following formula: dX t = -β t X t dt + 2β t dB t , where X 0 ∼ q 0 .(1)In this case, m t = exp(-t 0 β s ds) and σ 2 t = 1 -m 2 t . Note that m T ≤ exp (-T / β) ,which indicates a fast forward convergence rate TV(q T |N (0, I)) ≤ exp (-T / β) [Chen et al., 2023c] .The VESDE forward process. The VESDE forward process is defined without a drift term:dX t = dσ 2t /dt dB t , where X 0 ∼ q 0 .(2)Two common choices for σ 2 t are t and t 2 , with the latter achieving SOTA performance [Karras et al., 2022 , Teng et al., 2023] . However, VESDE only has a slow polynomial-decay forward convergence rate (Theorem 4.2), which introduces hardness to obtain an efficient sample complexity (see Section 5). This motivates us to design an improved VESDE process with a fast forward convergence rate. 
the drifted vesde forward process: 
 Note that the forward convergence rate of the general process is upper bounded by m T /σ 2 T (Theorem 4.2). In practical applications [Ho et al., 2020 , Karras et al., 2022 , Song et al., 2023] , the variance of the forward process σ 2 t at time T , which is determined by the diffusion term, does not exceed T 2 . This indicates the contribution of σ 2T to the forward convergence rate is only 1/Poly(T ). Hence, the exponential-decay forward convergence rate of VPSDE comes from the drift term, which introduces an exponential-decay m t ≤ exp (-T / β). Due to the absence of the drift term in VESDE, the data information, such as expectation E[q 0 ] and covariance Cov[q 0 ], does not decay and m t ≡ 1, which is a key to an only polynomial-decay forward convergence rate m T /σ 2 T ≤ 1/Poly(T ). With the drift term, the VPSDE gradually removes the data information from q t during the process, which makes q t quickly converge to pure Gaussian noise. Inspired by this elimination effect of the drift term, we propose a drifted VESDE forward process:EQUATIONwhere τ ∈ [T, T 2 ] is the coefficient used to balance the drift and diffusion term 2 , and {β t } t≥0 is a positive non-decreasing sequence. In this case,EQUATIONWe show that the drifted VESDE is not only an effective representation of the existing VESDE but also extends beyond it (see Section 7 and Appendix A.1). We also prove that this process with suitable β t has a exp (-T ) forward convergence rate and enjoys an efficient polynomial sample complexity.To generate samples from Gaussian noise, a diffusion model reverses the forward process. Let q τ t be the density function of the drifted VESDE forward process at time t and (Y t ) t∈[0,T ] = (X T -t ) t∈[0,T ] . As shown in Cattiaux et al. [2021] , the reverse process of drifted VESDE has the following form3 :EQUATIONThe parameter η ∈ [0, 1] is used to determine the type of reverse processes. There are two common reverse processes in the application: reverse probability flow ODE (PFODE) (η = 0) [Song et al., 2020b [Song et al., , 2023] ] and reverse SDE (η = 1) [Ho et al., 2020] .To generate distribution q 0 through running the above reverse process, diffusion models need the true score function ∇ log q τ T -t (Y t ) and the accurate reverse beginning distribution q τ T . However, ∇ log q τ T -t (Y t ) and q τ T contain the data information and usually can not be exactly calculated. For the score function, diffusion models approximate it using a score network s(T -t, •) by minimizing the score matching objective function [Vincent, 2011] . For the initial distribution of the reverse process, since q τ T should be close to a pure Gaussian, we choose q τ ∞ = N (0, σ 2 T I) as an approximation. Then, the continuous reverse process ( Y t ) t∈[0,T ] , incorporating s(T -t, •) and q τ ∞ , is defined as:d Y t = β T -t Y t /τ + (1 + η 2 )s(T -t, Y t ) dt + η 2β T -t dB t, where Y 0 ∼ q τ ∞ . Since diffusion models can not run a continuous process due to the nonlinear score function, these models usually discretize the above continuous process and freeze the approximated score at the beginning of each interval. Let {γ k } k∈[K] be the stepsize and t k+1 = k j=0 γ j . As shown in Kim et al. [2021] , ∇ log q τ T -t (Y t ) goes to +∞ at the end of the reverse process. To mitigate this issue, they use the early stopping technique t K = T -δ, and we also employ this technique in this work. With the stepsize, we choose the exponential integrator discretization scheme [Zhang and Chen, 2022 ] to discretize the above process, which runs the following process:d Y t = β T -t Y t /τ + (1 + η 2 )s(T -t k , Y t k ) dt + η 2β T -t dB t , where t ∈ [t k , t k+1 ] . (6)As shown in Karras et al. [2022] , the choice of β t significantly affects the performance of models, and we need to determine β t before running the reverse process. The state-of-the-art diffusion models adopt β t = t, which increases rapidly and has an unbounded range. However, current theoretical works assume β t to be a constant [Chen et al., 2023c] or confined to a bounded interval [1/ β, β] [Bortoli, 2022] to match the setting of VPSDE. To align more closely with practical applications of VE-based models, we allow an unbounded β t in this work. Furthermore, we make a detailed assumption on β t when considering different reverse processes. Assumption 3.1. Let {β t } t≥0 be a positive, non-decreasing sequence. For any τ ∈ [T, T 2 ], there exists constants β and C, such that for any t ∈ [0, T ]: (1) for η = 0, then 1/ β ≤ β t ≤ max{ β, t} andT 0 β t /τ dt ≤ C; (2) for η = 1, then 1/ β ≤ β t ≤ max{ β, t 2 }.As shown in Chen et al. [2023b] , due to the absence of the stochasticity, the small errors for quickly accumulate and are magnified. Hence, we assume a conservative β t for the reverse PFODE, whose growth rate is at most t, to avoid an additional exp (T ) in the convergence guarantee (see Section 6.1). We note that this choice of β t is satisfied in practical applications [Song et al., 2020b , Karras et al., 2022] . For the reverse SDE setting, we assume the growth rate of β t can depend on τ instead of at most linear. For example, when τ = T 2 , we can choose β t = t 2 , which has the same order as τ . As shown in Theorem 4.2, the drifted VESDE with aggressive β t = t 2 has an exponential-decay forward convergence rate, which leads to the first efficient polynomial complexity for VE-based models.Notations. For x ∈ R d and A ∈ R d×d , we denote by ∥x∥ and ∥A∥ the Euclidean norm for vector and the spectral norm for matrix. We denote by γK = argmax k∈{0,...,K-1} γ k the maximum stepsizefor k ∈ [0, K -1]. We denote by q 0 P T the distribution of X T , Q q τ ∞ t K the distribution of Y t K , R q τ ∞ K the distribution of Y t K and Q q0P T t Kthe distribution which does reverse process starting from q τ T (Eq. 5). We denote by W 1 and W 2 the Wasserstein distance of order one and two, respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All assumptions, Theorem, Corollary, and proof sketch have been clearly stated in the main content. The detailed proof appears in the appendix.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
the faster forward convergence rate for the drifted vesde: 
 This section shows that the drifted VESDE has a fast forward convergence rate. Since q τ T contains the data information, we first introduce the manifold hypothesis before controlling TV(q τ T , q τ ∞ ). Assumption 4.1. q 0 is supported on a compact set M and 0 ∈ M.We denote R the diameter of the manifold by R = sup{∥x -y∥ : x, y ∈ M} and assume R > 1.As shown in Section 1, the manifold hypothesis is supported by much empirical evidence [Bengio et al., 2013 , Fefferman et al., 2016 , Pope et al., 2021] and allows the blow-up phenomenon of the score. Recently, Tang and Yang [2024] show that diffusion models can adapt to the intrinsic manifold structure. With Assumption 4.1, we obtain the forward process guarantee for the drifted VESDE. Theorem 4.2. Assume Assumption 4.1 and 3.1. Let q τ ∞ = N (0, σ 2 T I). With m T , σ T defined in Equation (4), we have TV(q τ T , q τ ∞ ) ≤ √ m T D/σ T , where D = d|c| + E[q 0 ] + R and c is the eigenvalue of Cov[q 0 ] with the largest absolute value.Recall that m T = exp(-T 0 β t /τ dt), the previous VESDE [Song et al., 2020b , Karras et al., 2022 , Lee et al., 2022] chooses a conservative β t satisfies T 0 β t /τ dt ≤ C. However, with an aggressive β t , the drifted VESDE will have a faster convergence rate. To illustrate the accelerated forward process, we use τ = T 2 as an example and discuss different β t = t α1 , α 1 ∈ [1, 2] . Due to the definition of σ T , σ T ≈ T , and the forward convergence rate mainly depends on√ m T . When α 1 = 1 is conservative,m T is a constant, and the convergence rate is 1/T . When α 1 = 1 + ln(2r ln(T ))/ ln(T ) is slightly aggressive, the convergence rate is 1/T r+1 for r > 0. When α 1 ≥ 1 + ln(T -ln(T ))/ ln(T ) is aggressive, the convergence rate is faster than exp(-T ). In our analysis, whether β t can be aggressive depends on the reverse process (see Section 6). When choosing an aggressive β t , the drifted VESDE achieves an improved sample complexity compared with pure VESDE (see Section 5).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] We has shown all experiments detail including dataset and training detail in Appendix G. Furthermore, we discuss why these experiments results support our theoretical results in Section 7. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
the polynomial complexity for a series of vesde with reverse sde: 
 In this section, we first pay attention to the reverse SDE (η = 1) to show the power of the drifted VESDE. More specifically, we show that our general drifted VESDE form covers the current models (VP and VESDE). After that, we show that drifted VESDE can go beyond the current models and achieve an improved complexity with an aggressive β t . Since the objective function minimizes the L 2 distance between the ground truth and the approximated score, we assume that the approximated score is L 2 -accuracy, which is exactly the same with Chen et al. [2023c] and Benton et al. [2023] .Assumption 5.1. E qt k s t k -∇ ln q τ t k 2 ≤ ϵ 2 score for ∀k ∈ [K].With this assumption, we provide a universe convergence guarantee for τ ∈ [1, T 2 ] and β t ∈ [1, t 2 ] and discuss the sample complexity of VP and VE-based models in detail. Theorem 5.2. Assume Assumption 3.1, 4.1, 5.1. Let D defined in Theorem 4.2, γK = argmax k∈{0,...,K-1} γ k , τ = T 2 and β t ∈ [1, t 2 ]. Then, we have thatTV R q τ ∞ K , q δ ≤ D√ m T σ T + R 2 √ d σ 4 δ γK β T τ T + ϵ score β T T .To guarantee the above convergence guarantee smaller than O(ϵ TV ), each component of the result needs to be smaller than ϵ TV . As shown in Remark 5.3, it is difficult for pure VESDE to balance the approximated score and the first two error terms to achieve an efficient sample complexity. Hence, we discuss how to balance the reverse beginning and discretization error. More specifically, we require D√ m T /σ T ≤ ϵ TV and γK ≤ σ 8 δ ϵ 2 TV / R4 dβ T τ T , where the first inequality determines the order of T and the second inequality determines the stepsize γK . After that, with sample complexity K = T /γ K , we have that TV(Rq τ ∞ K , q δ ) ≤ O(ϵ TV ). The last step is to guarantee q δ and q 0 is close enough W 2 2 (q 0 , q δ ) ≤ ϵ 2 W2 , which requires σ 2 δ ≤ ϵ 2 W2 /(d + R √ d).Following the above process, this general convergence guarantee leads to the polynomial sample complexity for VP and VE-based models. When β t = 1 and τ = 1, the drifted VESDE becomes VPSDE and achieve the complexity O(1/(ϵ 8 W2 ϵ 2 TV )), which achieve exactly the same order compared with Chen et al. [2023c] . When β t = 1 and τ = T , our formula is similar but slightly better (Figure 2 and 3) to pure VESDE (σ 2 t = t) and achieves O(1/ϵ 8 W2 ϵ 8 TV ) result. For β t = t and τ = T 2 , the general formula is similar to SOTA pure VESDE (σ 2 t = t 2 ) and achieves the first polynomial results O(1/ϵ 8 W2 ϵ 7 TV ) for this model under the manifold hypothesis 4 . Although we achieve the first polynomial sample complexity for VE-based models under the manifold hypothesis, it is clear that the results of the VE-based models are significantly worse than the result of VP-based models since the slow 1/Poly(T ) forward convergence guarantee. More specifically, the forward convergence rate of VPSDE is exp (-T ), which means T has the order log(1/ϵ TV ) and can be ignore. When considering the pure VESDE with σ 2 t = t 2 , the forward convergence rate is D/T , which indicates T ≥ D/ϵ TV is a polynomial term and can not be ignored. Hence, the results K = R 4 dT 5 /(σ 8 δ ϵ 2 TV ) of pure VESDE (σ 2 t = t 2 ) is heavily influenced by T . When considering pure VESDE with σ 2 t = t, it suffers from a slower forward convergence guarantee D/ √ T , which indicates T ≥ D2 /ϵ 2 TV and K = R 4 dT 3 /(σ 8 δ ϵ 2 TV ). This is the first work to explain why pure VESDE (σ 2 t = t 2 ) performs better than pure VESDE (σ 2 t = t 2 ) from a theoretical perspective. Remark 5.3. In this part, we explain the reason why the pure VESDE fails to balance the reverse beginning and the approximated score. We use the pure VESDE with σ 2 t = t 2 as an example. Under this setting, the guarantee has the form 1/T + γK T 4 /δ 4 + ϵ scoreT 2 , which requires T ≥ 1/ϵ TV . Then, ϵ scoreT 2 is larger than ϵ score / ϵ 2 TV . Hence, it is hard to achieve non-asymptotic results. Drifted VESDE with an aggressive β t balances different error terms. This part shows that our drifted VESDE with a suitable aggressive β t can balance the above three error terms. More specifically, we show that introducing aggressive β t only slightly affects the discretization error and significantly benefits in balancing reverse beginning and approximated errors. As a result, we obtain an efficient polynomial complexity for a series of VE-based models with unbounded β t .Corollary 5.4. Following the setting of Theorem 5.2. When considering τ = T 2 , β t = t 2 , bychoosing δ ≤ ϵ 2/3 W 2 (d+R √ d) 1/3 , T ≥ 2 ln( D/ϵ T V ), γK ≤ δ 12 ϵ 2 TV ln 5 ( D/ϵ TV )/(R 4 d) and assuming ϵ score ≤ O(ϵ TV ), R q τ ∞ K is (ϵ TV + ϵ score ) close to q δ ,which is ϵ W2 close to q 0 , with sample complexityK ≤ O dR 4 (d + R √ d) 4 ϵ 8 W2 ϵ 2 TV . Defined by R q τ ∞ K,R0 the output R q τ ∞ K projected onto B (0, R 0 ) for R 0 = Θ(R). Then, we achieve pure W 2 guarantee W 2 (R q τ ∞ K,R0 , q 0 ) ≤ ϵ W2 with sample complexity O dR 8 (d+R √ d) 4 ϵ 12 W 2 .Since our drifted VESDE with τ = T 2 and β t = t 2 has a fast forward convergence rate exp (-T )/T 2 , T becomes a logarithmic term and does not influence the discretization term, which is the source of the improved sample complexity. Furthermore, the requirement of ϵ score has the same order with ϵ TV , which indicates the drifted VESDE balances the reverse beginning and approximated score error. In fact, we only require the forward convergence rate of drifted VESDE is exp (-T ), which indicates a series of VE-based models can achieve this sample complexity. We use τ = T 2 as an example. When considering the β t = t α1 , we require 2 ≥ α 1 ≥ 1 + ln(T -ln(T ))/ ln(T ) to enjoy exp (-T ) forward convergence rate and achieve K ≤ O 1/(ϵ 8 W2 ϵ 2 TV ) sample complexity. For β t = t and τ = T , we also obtain complexity K ≤ O 1/ ϵ 8 W2 ϵ 2 TV (Appendix C.1). Remark 5.5. Recently, Lee et al. [2022] and Gao et al. [2023] consider the sample complexity of VESDE with reverse SDE under strong assumption. Lee et al. [2022] consider VESDE with σ 2 t = t and achieve Õ(L 2 /ϵ 4 TV ) result under the LSI assumption. Under the manifold hypothesis, the result is Õ(1/ϵ 8 W2 ϵ 4 TV ), which is worse than Corollary 5.4. Gao et al. [2023] achieve pure W 2 guarantee O(1/ϵ 2.5 W2 ) under the log-concave distribution, which is even stronger than LSI assumption and ignore the influence of δ. Hence, they ignore an additional 1/Poly(W 2 ) (Detail in Appendix A.2).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: As a theoretical work, we only do simple synthetic experiments to support our results. All detail and the used checkpoint are shown in Appendix G. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We have checked the code of ethics and make sure that our work satisfies the code of ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed the broader impacts of our work at the end of main paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
the tangent-based analysis framework: 
 To deepen the understanding of VE-based models instead of the specific reverse process, we introduce the unified framework for VESDE with reverse SDE and PFODE. Similar to previous PFODE work [Chen et al., 2023d] , we assume an accurate score and consider the other errors. Theorem 6.1. Assume Assumption 3.1 and 4.1, δ ≤ 1/32 andγ k sup v∈[T -t k+1 ,T -t k ] β v /σ 2 v ≤ 1/28 for ∀k ∈ {0, ..., K -1}. Let γ K = δ. Then, for ∀τ ∈ [T, T 2 ]: (1) If η = 1 (the reverse SDE), choosing β t = t 2 , W 1 R q τ ∞ K , q 0 is bounded by ( R τ + √ d) √ δ + exp R 2 2 ( β δ 3 + 1 τ ) C 1 (τ )T κ 2 1 (τ ) ( β δ 3 + 1 τ )γ 1/2 K + 1 γ1/2 K + De -T /2 √ τ ,where κ 1 (τ ) = T 2 (1/τ + β/δ 3 ) and C 1 (τ ) is linear in τ 2 .(2) If η = 0 (PFODE), choosing a conservative β t (Assumption 3.1),W 1 R q τ ∞ K , q 0 is bounded by ( R τ + √ d) √ δ + exp R 2 2 ( β δ 2 + 1 τ ) C 2 (τ )κ 2 2 (τ )T ( β δ 2 + 1 τ )γ 1/2 K + 1 γ1/2 K + D √ τ ,where κ 2 (τ ) = T 1/τ + β/δ 2 and C 2 (τ ) is linear in τ 2 .Theorem 6.1 proves the first quantitative guarantee for VE-based models with reverse PFODE using the unified tangent-based framework. Correspondingly, the Girsanov-based method [Chen et al., 2023c,a] can not deal with reverse PFODE since the reverse process diffusion term is not well-defined.Recently, Chen et al. [2023d] employ the Restoration-Degradation framework to analyze VESDE with reverse PFODE, which also has exponential dependence on R and δ. Furthermore, their results have exponential dependence on β t (g max in Chen et al. [2023d] ), which corresponds to τ . However, our dependence on τ appears in the polynomial term. Hence, our framework is a suitable unified framework. Furthermore, we emphasize that our tangent-based unified framework is not a simple extension of Bortoli [2022] . We carefully control the tangent process according to the variance exploding property of VESDE to avoid exp (T ) term when considering PFODE (Section 6.1).Theorem 6.1 has exponential dependence on R and δ, which is introduced by the tangent process. Similar to Bortoli [2022] , if we assume the Hessian ∇ 2 log q t (x t ) ≤ Γ/σ 2 t , we obtain a better control on the tangent process and replace the exponential dependence on δ by a polynomial dependence on δ and exponential dependence on Γ when considering reverse PFODE. Corollary 6.2. Assume Assumption 3.1, 4.1 and ∇ 2 log q t (x t ) ≤ Γ/σ 2 t . Let η = 0 (reverse PFODE), δ ∈ (0, 1/32), τ = T 2 , β t = t and κ 2 (τ ), C 2 (τ ) defined in Theorem 6.1, we haveW 1 R q τ ∞ K , q 0 ≤ ( R τ + √ d) √ δ + β Γ 2 δ Γ exp Γ + 2 2 C 2 (τ )κ 2 2 (τ )T (( β δ 2 + 1 τ )γ 1/2 K + 1)γ 1/2 K + D √ τ .Though the additional assumption is strong, many special cases, such as hypercube M = [-1/2, 1/2] p satisfy this assumption. We emphasize that our analysis also holds for VESDE (σ 2 t = t 2 ) with reverse PFODE, which means our results can explain the SOTA model in Karras et al. [2022] .In this section, we introduce the unified tangent-based framework for reverse SDE and PFODE and discuss key steps to achieve the quantitative guarantee for PFODE. Firstly, we decompose the goalW 1 R q τ ∞ K , q 0 into three terms: W 1 R q τ ∞ K , Q q τ ∞ t K + W 1 Q q τ ∞ t K , Q q0P T t K + W 1 Q q0P T t K , q 0 .These terms correspond to the discretization scheme, reverse beginning distribution, and the early stopping parameter δ. We focus on most difficult discretization term and first recall the stochastic flow of the reverse process for any x ∈ R d and s, t ∈ [0, T ] with t ≥ s:dY x s,t = β T -t Y x s,t /τ + 1 + η 2 ∇ log q T -t Y x s,t dt + η 2β T -t dB t , where Y x s,s = x . With ∇Y x s,s = I, the corresponding tangent process is d∇Y x s,t =β T -t ∇Y x s,t /τ dt + β T -t 1 + η 2 ∇ 2 log q T -t (Y x s,t )∇Y x s,t dt .The key of the discretization error is to bound tangent process ∇Y x s,t K . For this term, we consider the reverse SDE and PFODE simultaneously and propose a general version of Bortoli [2022] . Lemma 6.3. Assume Assumption 3.1 and 4.1. For ∀s ∈ [0, t K ] and x ∈ R d , we have We emphasize that the general bound for the tangent process is the key to achieving the guarantee for VESDE with the reverse ODE. Recall that in the original lemma for the tangent processes, since τ is independent of T and β t is bounded in a small interval [1/ β, β],∥∇Y x s,t K ∥ ≤ exp R 2 2σ 2 T -t K + (1 -η 2 ) 2 t K 0 β T -u τ du .t K 0 β T -u /τ du = Θ(T ), which means there is an additional exp (T ) when considering VPSDE with revere PFODE. However, our tangent-based lemma makes use of the variance exploding property of VESDE to guarantee that T 0 β t /τ dt ≤ C with a conservative β t = t when considering reverse PFODE. When η = 1, we choose aggressive β t = t 2 since the choice of β t does not affect the bound of the tangent process.For the early stopping term, it corresponds to δ and is smaller than2(R/τ + √ d) √ δ.Since we can not use the data processing inequality in Wasserstein distance, the reverse beginning terms consists of the bound of tangent process term and the forward process term:W 1 Q q τ ∞ t K , Q q0P T t K ≤ √ m T D σ T exp R 2 2σ 2 T -t K + (1 -η 2 ) 2 t K 0 β T -u τ du .One notable future work is introducing the short regularization technique [Chen et al., 2023b] and suitable corrector to remove the above exponential dependence. 
experiments: 
 In this section, we show the power of the drifted VESDE forward process through experiments. Section 7.1 shows that aggressive one achieves good balance in different error terms. After that, we consider the approximated score and show that the conservative one can improve the quality of the generated distribution without training in the synethetic and real-world setting.In this section, we do experiments on 2-D Gaussian to show that the aggressive drifted VESDE balances different errors. Since the ground truth score of the Gaussian can be directly calculated, we use the accurate score function to discuss the balance between the other two error terms clearly. We show how to use approximated score in Section 7.2. As shown in Figure 2 , the process with aggressive β t = t 2 achieves the best and second performance in EI and EM discretization, which supports our theoretical result (Corollary 5.4).The third best process is conservative β t = t with the small drift term. The reason is that though it can not achieve a exp (-T ) forward guarantee, it also has a constant decay on prior information, which slightly reduces the effect of the reverse beginning error (Section 3.1). The worst process is pure VESDE since it is hard to balance different error sources. Our experimental results also show that EI is better than EM. As shown in Figure 2 , the red and orange lines have similar trends. Hence, for conservative drifted VESDE, which satisfies (2) of Assumption 3.1, we can directly use the models trained by pure VESDE to improve the quality of generated distribution. We confirm our intuition by training the model with pure VESDE with σ 2 t = t and directly use the models to conservative drifted VESDE with β t = 1 and τ = T . From the experimental results (Figure 1 ), it is clear that pure VESDE has a low density on the Swiss roll except for the center one, which indicates pure VESDE can not deal with large dataset variance, as we discuss in Section 4. For conservative drift VESDE, as we discuss in the above section, it can reduce the influence of the dataset information. Figure 1 (c) supports our augmentation and shows that the density of the generated distribution is more uniform compared to pure VESDE, which means that the drift VESDE can deal with large dataset mean and variance.Beyond the synthetic data, we show that our conservative drifted VESDE can improve the generated images of pure VESDE without training on the real-world CelebA256 dataset. From the qualitative perspective, as shown in Figure 3 , the images generated by our drifted VESDE have more detail (such as hair and beard details). On the contrary, since pure VESDE can not deal with large variance, the images generated by pure VESDE appear blurry and unrealistic in these details. From the quantitative results, we use aesthetic score [Schuhmann et al., 2022] and Inception Score to measure the quality of generated images. Our drifted VESDE achieves aesthetic score 5.813, and IS 4.174, which is better than the results of baseline pure VESDE (aesthetic score 5.807 and IS 4.082). There are more examples on CelebA256 and more experiments on Swiss roll and 1D-GMM to explore different sampling methods (RK45, PFODE) and different T . We refer to Appendix G for more details. 
conclusion: 
 In this work, we analyze the VE-based models under the manifold hypothesis. Firstly, we propose a new forward drifted VESDE process, which enjoys a faster forward convergence rate. Then, we show that with an aggressive β t , the new process balances different errors and achieve the first efficient polynomial sample complexity for a series of VE-based models with reverse SDE.After achieving the above results, we go beyond the reverse SDE and propose the tangent-based unified framework, which considers reverse SDE and PFODE at the same time. Under this framework, we make use of the variance exploding property of VESDE and achieve the first quantitative convergence guarantee for SOTA VE-based models with reverse PFODE. Finally, we show the power of the new drifted forward process through synthetic and real-world experiments.Future Work and Limitation. This work proposes the first unified framework for VE-based models with an accurate score. After that, we plan to consider the approximated score error and provide a polynomial complexity for the VE-based models with reverse PFODE under the manifold hypothesis.Broader Impact. Our work focuses on the convergence guarantee of the SOTA diffusion models and deepens the understanding of diffusion models. Therefore, this work can be viewed as a fundamental step in improving the quality of diffusion models and the societal impact is similar to general generative models [Mirsky and Lee, 2021] . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: A.1 The Drifted VESDE is Representative Enough.In this work, we consider τ ∈ [T, T 2 ] and show that this choice is enough to represent the current VESDE. More specifically, since VESDE withσ 2 t = t has q T = N (E[q 0 ], Cov[q 0 ] + T I) and drifted VESDE with τ = T 2 and β t ≡ 1/2 has q τ T = N (exp (-1 2T )E[q 0 ], exp (-1 T )Cov[q 0 ] + (1 - exp (-1 T ))T 2 I), these two setting is almost identical when T → +∞. The second choice of VESDE σ 2 t = t 2 , which achieves the state-of-the-art performance [Karras et al., 2022] , is almost identical to τ = T 2 , β t = t. The simulation experiments also show that VESDE and drifted VESDE with specific β t and τ have similar performance (Figure 2 ).The results of Lee et al. [2022] Lee et al. [2022] consider VESDE (σ 2 t = t ) with reverse SDE under the LSI assumption with parameter C LS . The LSI assumption does not allow the presence of substantial non-convexity and is far away from the multi-modal real-world distribution. Furthermore, they use unrealistic assumption ϵ score ≤ 1/(C LS + T ) to avoid the effect of the approximated score, which is stronger than Assumption 5.1. Under the above strong assumption, Lee et al. [2022] achieve the polynomial sample complexity Õ(L 2 d(d|c| + R) 2 /ϵ 4 TV ). Under the manifold hypothesis, by Lemma E.2, we know thatL = R 2 d 2 /ϵ 4 W2 Then, the result is Õ(R 4 d 5 (d|c| + R) 2 /ϵ 8 W2 ϵ 4TV ), which is worse than Corollary 5.4.The results of Gao et al. [2023] . Gao et al. [2023] analyze a series of VESDE with reverse SDE and achieve 1/ϵ 2.5 W2 sample compelxity for VESDE with σ 2 t = t 2 in 2-Wasserstein distance. However, they assume the data distribution is log-concave, which is even stronger than LSI assumption. Furthermore, under this assumption, ∇ log q T -t (•) do not blow-up at the end of the reverse process, which do not match the empirical phenomenon and ignore the influence of early stopping parameters. To transfer their results to our the results under the manifold hypothesis, we need to consider the influence of δ, which would introduce an additional 1/Poly(W 2 ) term.Lemma B.1. The minimization problem min mt,Vt KL (q τ t | N ( mt , V t )) is minimized by mt = m t E [q 0 ] and V t = m 2 t Cov [q 0 ] + σ 2 t I, where m t and σ t defined in Equation (4).Proof. For simplicity, we denote the mean and covariance of q 0 by a and C ′ . We also define the optimize variable n t = N ( mt , C t ). We can directly compute the KL divergence KL(q t |n t ):KL (q t |n t ) = -H (q t ) -log (n t (x)) q t (x)dx = -H (q t ) + d 2 log(2π) + 1 2 log (det (V t )) + 1 2 (x -mt ) T V -1 t (x -mt )q t (x)dx .For the last term, we directly compute(x -mt ) T V -1 t (x -mt ) p t (x)dx =E (X t -mt ) T V -1 t (X t -mt ) = E (m t X 0 + σ t Z -mt ) T V -1 t (m t X 0 + σ t Z -mt ) =E m 2 t (X 0 -a) T V -1 t (X 0 -a) + (m t a -mt ) T V -1 t (m t a -mt ) + σ 2 t E Z T V -1 t Z =m 2 t tr C ′ V -1 t + σ 2 t tr V -1 t + (m t a -mt ) T V -1 t (m t a -mt ) ,where the second inequality follows that X t = m t X 0 + σ t Z. It is clear that the optimal solution of mt is m t a. In the next step, we focus on the optimization problem for V t :L V -1 t = log (det (V t )) + tr m 2 t C ′ + σ 2 t I V -1 t = -log det V -1 t + tr m 2 t C ′ + σ 2 t I V -1 t .Since the above optimization is a convex optimization problem, we use the method similar to Pidstrigach [2022] , we obtain that the optimal solution of V t is m 2 t C ′ + σ 2 t I. ■ Lemma B.2. Let mt and V t be the optimal mean and covariance operator from Lemma B.1. ThenKL (q t |N ( mt , V t )) ≤ 1 2 log d i=1 m 2 t c i + σ 2 t (σ 2 t ) d + R 2 m t σ 2 t ≤ dm 2 t c 2σ 2 t + R 2 m t σ 2 t + o( m 2 t c σ 2 t ) , KL N ( mt , V t )|(N (0, σ 2 t ) ≤ m 2 t d i=1 c i 2σ 2 t + m 2 t (E[q 0 ]) 2 2σ 2 t + 1 2 log (σ 2 t ) d d i=1 (m 2 t c i + σ 2 t ) ≤ m 2 t d i=1 c i 2σ 2 t + m 2 t (E[q 0 ]) 2 2σ 2 t + dm 2 t c 2σ 2 t + o( m 2 t c σ 2 t ) ,where c i are the eigenvalues of Cov [q 0 ], and c is the eigenvalue with the largest absolute value.Proof. For t ≥ 0, we directly calculate the KL divergence for this term:KL (q t | N ( mt , V t )) = -H (q t ) + 1 2 log (det (2πV t )) + 1 2 tr m 2 t C ′ + σ 2 t I V -1 t = -H (q t ) + 1 2 log (det (2πV t )) + d 2 = -H (q t ) + d 2 log(2π) + 1 2 log d i=1 m 2 t c i + σ 2 t + d 2 ,where c i are the eigenvalues of Cov [q 0 ]. Now, we only need to calculate H(q t ):-H (q t ) = E Xt [log q t (X t )] = E Xt log E X0 (2πσ 2 t ) -d/2 exp - 1 2σ 2 t ∥X t -X 0 ∥ 2 . By Assumption 4.1, it is clear that exp - 1 2σ 2 t ∥X t -X 0 ∥ 2 ≤ exp - 1 2σ 2 t ∥X t ∥ 2 + 2⟨X t , X 0 ⟩ .Then, we know thatE log E X0 (2πσ 2 t ) -d/2 exp - 1 2σ 2 t ∥X t -X 0 ∥ 2 ≤E log (2πσ 2 t ) -d/2 - 1 2σ 2 t ∥X t ∥ 2 + 2⟨X t , X 0 ⟩ ≤ - d 2 log(2π) - 1 2 log (σ 2 t ) d - 1 2σ 2 t E ∥X t ∥ 2 + R 2 m t σ 2 t .we also know thatE ∥X t ∥ 2 = m 2 t E ∥X 0 ∥ 2 + σ 2 t E ∥Z∥ 2 = E ∥X 0 ∥ 2 + tE ∥Z∥ 2 = m2 0 + V 0 + σ 2 t d .Finally, put these terms together, we have:KL (q t |N ( mt , V t )) ≤ 1 2 log d i=1 m 2 t c i + σ 2 t (σ 2 t ) d + R 2 m t σ 2 t ,where c i are the eigenvalues of Cov [q 0 ]. Then by choosing the largest absolute value eigenvalue largest absolute value, we can use the Taylor expansion to obtain the first results of this lemma. For the second result of this lemma, we directly compute the KL divergence between N ( mt , V t ) and N (0, σ 2 t ) to obtain the final results. ■ Theorem 4.2. Assume Assumption 4.1 and 3.1. Let q τ ∞ = N (0, σ 2 T I). With m T , σ T defined in Equation (4), we have TV(q τ T , q τ ∞ ) ≤ √ m T D/σ T , where D = d|c| + E[q 0 ] + R and c is the eigenvalue of Cov[q 0 ] with the largest absolute value.Proof. We know that ∥q T -q τ ∞ ∥ TV ≤ ∥q T -N (m T E[q 0 ], m 2 T Cov[q 0 ] + σ 2 T I)∥ TV + ∥N (m T E[q 0 ], m 2 T Cov[q 0 ] + σ 2 T I) -q τ ∞ ∥ TV .dX t = b t dt + α t dB twhere B is a Q T -Brownian motion. We assume that for each t > 0, α t is a d × d symmetric positive definite matrix. Then, provided that Novikov's condition holds,E Q T exp 1 2 T 0 α -1 t bt -b t 2 dt < ∞,we have thatdP T dQ T = exp T 0 α -1 t bt -b t dB t - 1 2 T 0 α -1 t bt -b t 2 dt .If the Novikov's condition is satisfied, we apply the Girsanov theorem by choosingP T = R q τ T K , Q T = Q q τ T t K , bt = β T -t 1 τ Y t + 2s(T -t k , Y t ) (for t ∈ [t k , t k+1 ]), b t = β T -t 1 τ Y t + 1 + η 2 ∇ log q T -t (Y t ) , and α t = 2β T -t I d .Then, similar to Chen et al. [2023c] , we have the following lemma.Lemma C.2. Assuming that R q τ T K and Q q τ T t K satisfy Novikov's condition, it holds that KL Q q τ T t K ∥R q τ T K = E Q q τ T t K ln dQ q τ T t K dR q τ T K = K-1 k=0 E Q q τ T t K t k+1 t k 2β T -t ∥s (T -t k , Y t k ) -∇ ln q T -t (Y t )∥ 2 dt .Before using the Girsanov's Theorem, we need to check the Novikov's condition. We use almost the same proof process compare to Chen et al. [2023c] . The key proof of the Novikov's condition is Lemma 19 of Chen et al. [2023c] . Hence, we give a complete proof of Lemma 19 in Chen et al.[2023c] under our drifted VESDE. Before the proof, we first introduce a smooth cuttoff function for truncating the drift terms.Lemma C.3 (lemma 17 of Chen et al. [2023c] ). For any R > 0, there is a smooth functionϕ R : R d → [0, 1] satisfying: 1. ϕ R(x) = 1 for all ∥x∥ ≤ R, 2. ϕ R(x) = 0 for all ∥x∥ ≥ 2 R, 3. ϕ R is O(1/ R)-Lipschitz.Note that R is not R in Assumption 4.1 and will goes to +∞ in the proof of Chen et al. [2023c] . Similar to Chen et al. [2023c] , we also introduce a L ∞ and a modified process with truncation argument for t ∈ [t k , t k+1 ]. Define the bad setB t := {∥s t -∇ ln q t ∥ ≥ ε score ,∞ } ,where ε score, ∞ > 0 is a parameter to be chosen later. We define the L ∞ -accurate score estimate to bes ∞ t := s t 1 B c t + ∇ ln q t 1 Bt .We note that ∥s ∞ t -∇ ln q t ∥ ≤ ε score ,∞ . The modified process with truncation argument for t∈ [t k , t k+1 ] is dY ∞ t = β T -t Y ∞ t /τ + 2∇ log q τ T -t (Y ∞ t ) dt + 2β T -t dB t , d Y ∞ t = β T -t Y ∞ t /τ + 2s(T -t k , Y ∞ t ) dt + 2β T -t dB t ,whereY ∞ 0 = Y ∞ 0 is obtained by sampling Y ∞ 0 ∼ q τ T and setting Y ∞ 0 = X T if ∥X T ∥ ≤ R and setting Y ∞ 0 = 0 otherwise.Then, we ara ready to prove the following lemma. Lemma C.4 (Modified key lemma for Novikov's condition).E Q q τ T ,∞ t K exp K-1 k=0 t k+1 t k β T -t k ϕ R Y ∞ T -t k s ∞ T -t k Y ∞ T -t k -ϕ R Y ∞ T -t k ∇ ln q T -t Y ∞ T -t k 2 dt < ∞Proof. We note that due to the manifold hypothesis(Assumption 4.1), if R ≥ √ dT 2 + R 2 , then the marginal distribution of Y ∞T -t k is exactly the same compared to X t k . We also recall that R → +∞ in Chen et al. [2023c] (Theorem 21 of Chen et al. [2023c] ). Hence, we can use Lemma E.1 to prove thatβ T -t k ϕ R Y ∞ T -t k s ∞ T -t k Y ∞ T -t k ≤ sup t * ∈[0,T -δ] β T -t * s ∞ T -t * (Y T -t * ) =: A ′ < ∞ . and β T -t k ϕ R Y ∞ T -t k ∇ ln q T -t Y ∞ T -t k ≤ sup t * ∈[0,T -δ] β T -t * ∥∇ ln q T -t * (Y T -t ⋆ )∥ =: B ′ < ∞ .Then, the left hand of this lemma is at most exp 2T A ′2 + B ′2 < ∞ as claimed. ■After obtaining the above inequality, the remaining proof for Novikov's condition are exactly compared to Chen et al. [2023c] .Since we assume the accurate score function in this work, this lemma need to controlsup x * ∈B(0,R),t * ∈[0,T -δ] 2β T -t * ∥∇ ln q T -t * (x * )∥ =: B < ∞ .As we shown in Lemma E.1, we know that with the early stopping parameter δ, ∥∇ ln q T -t * (x * )∥ is controlled. By using Assumption 4.1, we know that 1 β T -t * ≤ β. Finally, with similar process to Chen et al. [2023c] , we can proof that the Novikov's condition is satisfied. The following lemma show the discretization error for our drifted VESDE with reverse SDE. .,K-1} γ k , γ K = δ. If τ ∈ [1, T 2 ] and β t ∈ [1, t 2 ], then with Q q τ T t K and R q τ T K defined in Lemma C.2, TV R q τ T K , Q q τ T t K 2 ≲ R 4 T τ β T d σ 8 δ γK + R 6 T τ β T σ 8 δ γ2 K + ϵ 2 score T β T .Proof. First, we control the discretization error in an interval t ∈ [t k , t k+1 ]:E Q q τ T t K ∥s (T -t k , Y t k ) -∇ ln q T -t (Y t )∥ 2 ≲ ϵ 2 score + E Q q τ T t K ∥∇ ln q T -t k (Y t k ) -∇ ln q T -t (Y t k )∥ 2 + E Q q τ T t K ∥∇ ln q T -t (Y t k ) -∇ ln q T -t (Y t )∥ 2 ≲ E Q q τ T t K ∇ ln q T -t k q T -t (Y t k ) 2 + L 2 E Q q τ T t K ∥Y t k -Y t ∥ 2 + ϵ 2 score ≲ τ L 2 dγ K + τ L 2 γ2 K dτ + R 2 + τ L 3 γ2 K + L 2 (β T dγ K + R 2 γ2 K ) + ϵ 2 score ≲ τ L 2 dγ K + τ L 2 R 2 γ2 K + ϵ 2 score ,whereL = max t∈[0,T -δ] ∇ 2 log q T -t (Y t ) ≤ 1 + R 2 /σ 4δ and the third inequality follows Lemma E.5. Then, we know thatK-1 k=0 E Q q τ T t K t k+1 t k 2β T -t ∥s (T -t k , Y t k ) -∇ ln q T -t (Y t )∥ 2 dt ≲ τ T β T L 2 dγ K + L 2 R 2 τ T β T γ2 K + ϵ 2 score T β T ≲ R 4 τ T β T d σ 8 δ γK + R 6 τ T β T σ 8 δ γ2 K + ϵ 2 score T β T .After obtaining the general discretization error for our drifted VESDE, we focus on two special cases.For τ = T 2 and β t = t 2 , we have thatK-1 k=0 E Q q τ T t K t k+1 t k 2β T -t ∥s (T -t k , Y t k ) -∇ ln q T -t (Y t )∥ 2 dt ≲ T 3 β T L 2 dγ K + L 2 R 2 T 3 β T γ2 K + ϵ 2 score T β T ≲ R 4 T 3 β T d σ 8 δ γK + R 6 T 3 β T σ 8 δ γ2 K + ϵ 2 score T β T = R 4 T 5 d σ 8 δ γK + R 6 T 5 σ 8 δ γ2 K + ϵ 2 score T 3 .For τ = T and β t = t, we know thatK-1 k=0 E Q q τ T t K t k+1 t k 2β T -t ∥s (T -t k , Y t k ) -∇ ln q T -t (Y t )∥ 2 dt ≲ T 3 L 2 dγ K + L 2 R 2 T 3 γ2 K + ϵ 2 score T 2 ≲ R 4 T 3 d σ 8 δ γK + R 6 T 3 σ 8 δ γ2 K + ϵ 2 score T 2 .Combined with the reversing beginning error controlled by Theorem 4.2, we can obtain the convergence guarantee for the general drifted VESDE with reverse SDE.Theorem 5.2. Assume Assumption 3.1, 4.1, 5.1. Let D defined in Theorem 4.2, γK = argmax k∈{0,...,K-1} γ k , τ = T 2 and β t ∈ [1, t 2 ]. Then, we have thatTV R q τ ∞ K , q δ ≤ D√ m T σ T + R 2 √ d σ 4 δ γK β T τ T + ϵ score β T T .Proof. By the data processing inequality, we know thatTV R q τ ∞ K , q δ ≤ TV R q τ ∞ K , R q τ T K + TV R q τ T K , Q q τ T t K ≤ TV (q τ T , q τ ∞ ) + TV R q τ ∞ K , Q q τ T t K .Combined with Theorem 4.2 and Lemma C.5, we achieve the final result. ■As shown in Theorem 5.2, the general convergence guarantee isD√ m T σ T + R 2 √ d σ 4 δ γK β T τ T + ϵ score β T T .In this section, we provide the sample complexity for different β t and τ .The results of τ = 1 and β t = 1. When considering β t = 1 and τ = 1, the drifted VESDE becomes VPSDE and m T = exp (-T ), which indicates T is a logarithmic term and the dominated term of the convergence guarantee is the discretization term O(R 2 √ dγ K /σ 4 δ ). To make this term smaller than ϵ TV , we require γK ≤ σ 8 δ ϵ 2 TV /(R 4 d). To make sure that W 2 2 (q 0 , q δ ) ≤ ϵ 2 W2 , we requireσ 2 δ ≤ ϵ 2 W 2 d+R √ d .Then, we achieve the final sample complexityK ≤ O dR 4 (d + R √ d) 4 ϵ 8 W2 ϵ 2 TV .We note that this results is exactly the same with Chen et al. [2023c] , which means our drifted VESDE covers VPSDE setting.C.1.1 The results of τ = T 2 with different β tIn this part, we analyze the influence of β t under setting τ = T 2 and show the power of our drifted VESDE.The aggressive β t (τ = T 2 ). When considering aggressive β t = t α1 where 2 ≥ α 1 ≥ 1 + ln(Tln(T ))/ ln(T ), √ m T /σ T ≥ exp (-T /2), which means T is a logarithmic term and can be ignored.After that, the analysis process is exactly the same with the above VPSDE setting, and we achieve the sample complexityK ≤ O dR 4 (d + R √ d) 4 ϵ 8 W2 ϵ 2 TV . Defined by R q τ ∞ K,R0 the output R q τ ∞ K projected onto B (0, R 0 ) for R 0 = Θ(R).Following exactly the same proof process of Chen et al. [2023c] (Corollary 5), we have thatW 2 (R q τ ∞ K,R0 , q 0 ) ≤ ϵ W2 with sample complexity K ≤ O dR 8 (d + R √ d) 4 ϵ 12 W2 .The conservative β t = t. In this case, the first term is D/T . To make this term smaller than ϵ TV , we require T ≥ D/ϵ TV . For the stepsize, we require γK ≤ σ 8 δ ϵ 2 TV /(R 4 dT 4 ), which means the sample complexity isK ≤ O dR 4 (d + R √ d) 4 D5 ϵ 8 W2 ϵ 7 TV .The most conservative β t = 1. In this case, m T = exp (-1/T ) andσ 2 t = τ (1 -m 2 t ) = T 2 (1 -exp (-2/T ). When T is large enough, m T = Θ(1) and σ 2 T = T , which indicates the first term is D/ √ T .To make this term smaller than ϵ TV , we require T ≥ D2 /ϵ 2 TV . For the second term, we require γK ≤ σ 8 δ ϵ 2 TV /(R 4 dT 3 ) Then, the final complexity isK ≤ O dR 4 (d + R √ d) 4 D8 ϵ 8 W2 ϵ 10 TV . C.1.2 The results of τ = T with different β tAt the remaining part, we show the sample complexity of setting τ = T with different β t .The results for setting τ = T and β t = t. For this setting, as shown in Lemma C.5, the discretization error isTV R q τ T K , Q q τ T t K 2 ≲ R 4 T 3 d σ 8 δ γK + R 6 T 3 σ 8 δ γ2 K + ϵ 2 score T 2 .Furthermore, we choose an aggressive β t , which indicates T is a logarithmic term. Then, by choosing d) and γK ≤ σ 8 δ ϵ 2 TV ln 3 D/ϵ TV / R 4 d , we obtain the sample complexityEQUATIONK = T γK ≤ O dR 4 (d + R √ d) 4 ϵ 8 W2 ϵ 2 TV .The results for setting τ = T and β t = 1. In this setting, the reverse beginning error is bounded by D/T , which indicates T ≥ D2 /ϵ 2 TV . For the discretization term, we require γK ≤ σ 8 δ ϵ 2 TV /(R 4 dT 2 ). Then, the sample complexity is bounde byK = T γK ≤ O dR 4 (d + R √ d) 4 D6 ϵ 8 W2 ϵ 8 TV .In this work, we introduce an indicator i ∈ {1, 2} for σ T -t K to represent different β t . We use τ = T 2 as an example. When β t = t 2 is aggressive, we choose i = 1 , η = 1 and σ -2T -t K (i = 1) ≤ 1 τ + β δ 3 . When β t = t is conservative, we choose i = 2, η ∈ [0, 1) and σ -2 T -t K (i = 2) ≤ 1 τ + β δ 2 .In the proof process of Lemma 6.3, Lemma D.1, Lemma D.2 and Lemma D.3, we ignore the indicator i since this lemma does not involve the specific value of σ 2 T -t K (i). Before the proof of this section, we first recall the stochastic flow of the reverse process for any x ∈ R d and s, t ∈ [0, T ] with t ≥ s:dY x s,t = β T -t Y x s,t /τ + 1 + η 2 ∇ log q T -t Y x s,t dt + η 2β T -t dB t , Y x s,s =x , and the interpolation of its discretization for any k ∈ {0, ..., K} and t ∈ [s k , t k+1 ):d Ȳx s,t (k) = β T -t Ȳx s,t /τ + 1 + η 2 s T -s k , Ȳx s,t dt + η 2β T -t dB t , Ȳx s,s = x, where s k = max (s, t k ). To deal with the discretization error, we use the approximation technique used in Bortoli [2022] . Hence, we introduce the tangent process:d∇Y x s,t = β T -t I/τ + 1 + η 2 ∇ 2 log q T -t (Y x s,t ) ∇Y x s,t dt, ∇Y xs,s = I . Then, we discuss the interpolation formula, which is used to control the discretization error. Proposition 1. For s, t ∈ [0, T ) with s < t, any k ∈ {0, ..., K} and (ω v ) v∈[s,T ] , we define thatb u (ω) = β T -u ( 1 τ ω u + (1 + η 2 )∇ log q T -u (ω u )) , bu (ω) = β T -u ( 1 τ ω u + (1 + η 2 )s(T -s k , ω s k )) , ∆b u (ω) = b u (ω) -bu (ω) ,where s k = max(s, t k ) and u ∈ [s k , t k+1 ). Then, for any x ∈ R d , we have thatY x s,t -Ȳx s,t = t s ∇Y x u,t Ȳx s,u ⊤ ∆b u Ȳx s,v v∈[s,T ] du,where for any u ∈ [0, T ), there exists a k ∈ {0, ..., K} satisfies u ∈ [s k , t k+1 ).For reverse SDE, the augmentation is similar to Bortoli [2022] (Appendix E). When η = 0, the stochastic extension of the Alekseev-Gröbner formula [Del Moral and Singh, 2022] degenerates into the original version [Alekseev, 1961] . After that, we control the tangent process.Lemma 6.3. Assume Assumption 3.1 and 4.1. For ∀s ∈ [0, t K ] and x ∈ R d , we have∥∇Y x s,t K ,i ∥ ≤ exp R 2 2σ 2 T -t K + (1 -η 2 ) 2 t K 0 β T -u τ du . If ∇ 2 log q t (x t ) ≤ Γ/σ 2 t , ∥∇Y x s,t K ,i ∥ ≤ σ -(1+η 2 )Γ T -t K exp 1 + η 2 Γ + 2 t K 0 β T -u τ du .Proof. Using the definition of the tangent process and Lemma E.1, we haved ∇Y x s,t 2 ≤ 2β T -t 1 τ ∇Y x s,t 2 -1 + η 2 1 -m 2 T -t R 2 / 2σ 2 T -t /σ 2 T -t ∇Y x s,t 2 dt .Using Lemma F.1, we havet s β T -u 1 τ -1 + η 2 /σ 2 T -u + 1 + η 2 m 2 T -u R 2 /2σ 4 T -u du ≤ 1 + η 2 R 2 /4 σ -2 T -t -σ -2 T -s + 1 -η 2 2 t s β T -u τ du ≤ 1 + η 2 R 2 4σ 2 T -t + 1 -η 2 2 t s β T -u τ du .Note that ∇Y s,s = I, we get∥∇Y x s,t K ∥ 2 ≤ exp 1 + η 2 R 2 2σ 2 T -t + (1 -η 2 ) t K 0 β T -u τ du .When we assume ∇ log q 2 t (x t ) ≤ Γ/σ 2 t , we know thatd ∇Y x s,t 2 ≤ 2β T -t 1 τ - 1 + η 2 Γ σ 2 T -t ∇Y x s,t 2 dt .Using Lemma F.1, we have2 t s β T -u /σ 2 T -u du ≤ log exp 2 T -s 0 β T -u τ du -1 -log exp 2 T -t 0 β T -u τ du -1 ≤ log σ 2 T -s -log σ 2 T -t + T -s T -t β u τ du .Then we have∥∇Y x s,t K ∥ 2 ≤ σ -(1+η 2 )Γ T -t K exp 1 + η 2 Γ + 2 t K 0 β T -u τ du .Thus we complete our proof. ■After bounding the gradient of the tangent process, the remaining term is ∥∆b∥:∥∆b∥ ≤ ∥∆ (a,b) b∥ + ∥∆ (b,c) b∥ + ∥∆ (c,d) b∥ ,where b (a) = b and b d) .(d) = b. Moreover, b (b) u (ω) = β T -u ( 1 τ ω u + (1 + η 2 )∇ log q T -s k (ω u )) , b (c) u (ω) = β T -u ( 1 τ ω u + (1 + η 2 )∇ log q T -s k (ω s k )) , ∆ a,b b = b (a) -b (b) , ∆ b,c b = b (b) -b (c) , ∆ c,d b = b (c) -b (We then control ∥∆ (a,b) b∥, ∥∆ (b,c) b∥, ∥∆ (c,d) b∥ separately. In this section, ∥∆ (c,d) b∥ = 0 since we assume that the accurate score function is achieved. For ∆ (a,b) b u (ω) , we have the following lemma.Lemma D.1. For s, u ∈ [0, T ) such that u ≥ s, u ∈ [s k , t k+1 ) and ω = (ω v ) v∈[s,T ] we have ∥∆ (a,b) b u (ω) ∥ ≤ 1 + η 2 β T -u sup v∈[T -u,T -t k ] β v /σ 6 v 2 + R 2 (R + ∥ω u ∥) γ k .Proof. Without loss of generality, we assume s ≤ t k . Then∥∆ (a,b) b u (ω) ∥ ≤ 1 + η 2 β T -u ∥∇ log q T -u (ω u ) -∇ log q T -t k (ω u ) ∥ ≤ 1 + η 2 β T -u γ k sup v∈[T -u,T -t k ] ∥∂ v ∇ log q T -v (ω u ) ∥.Then by Lemma E.4, we have∥∆ (a,b) b u (ω) ∥ ≤ 1 + η 2 β T -u sup v∈[T -u,T -t k ] β v /σ 6 v 2 + R 2 (R + ∥ω u ∥) γ k .For ∆ (b,c) b u (ω) , we have the following lemma.Lemma D.2. For s, u ∈ [0, T ) such that u ≥ s, u ∈ [s k , t k+1 ) and ω = (ω v ) v∈[s,T ] we have ∥∆ (b,c) b u (ω) ∥ ≤ 1 + η 2 β T -u /σ 4 T -u 1 + R 2 ∥ω u -ω s k ∥ .Proof. Without loss of generality, we assume s ≤ t k . In this case s k = t k , Then∥∆ (b,c) b u (ω) ∥ ≤ 1 + η 2 β T -u ∥∇ log q T -t k (ω t k ) -∇ log q T -t k (ω u ) ∥ ≤ 1 + η 2 β T -u sup v∈[u,T -t k ] ∥∇ 2 log q T -t k (ω v ) ∥∥ω u -ω t k ∥ .Using Lemma E.2, we have that∥∆ (b,c) b u (ω) ∥ ≤ 1 + η 2 β T -u /σ 4 T -u 1 + R 2 ∥ω u -ω t k ∥ .Then the proof is complete. ■We need to control the reverse process when dealing with ∆b. The following lemma shows an upper bound for the reverse Y k .Lemma D.3. Assume Assumption 3.1 ,Assumption 4.1, and there exists δ > 0 such thatγ k β T -t k σ 2 T -t k ≤ δ ≤ 1/28 for any k ∈ {0, • • • , K}, then we have E[∥Y k ∥ 2 ] ≤ U (τ ) = τ d + B(1/A + δ) ,whereA = 4η 2 + 2 -2δ -4(1 + η 2 )(1 + δ)µR B = 4(1 + η 2 )R 2 δ + 2(1 + η 2 )(1 + δ) R µ + 4η 2 τ dand µ is an arbitrary positive number which makes A > 0. In particular, if δ ≤ 1/28, thenE[∥Y k ∥ 2 ] ≤ U 0 (τ ) = 111R 2 + 13τ d .Proof. Recall the discretization of the backward process (the explicit form of Equation ( 6))Y k+1 = Y k + γ 1,k 1 τ Y k + (1 + η 2 )s (T -t k , Y k ) + η 2γ 2,k Z k , γ 1,k = exp T -t k T -t k+1 β s ds -1, γ 2,k = exp 2 T -t k T -t k+1 β s ds -1 /2 ,where {Z k } k∈K are independent Gaussian random variables. It is clear that γ 1,k ≤ γ 2,k ≤ 2γ 1,k , and using Lemma E.1 we have⟨x t , s(t, x t )⟩ = ⟨x t , ∇ log q t (x t )⟩ ≤ -∥x t ∥ 2 /σ 2 t + m t R∥x t ∥/σ 2 t ≤ (-1 + µm t R)∥x t ∥ 2 /σ 2 t + (m t R/µ)/σ 2 t ,where the first equality follows that we assume the accurate score function. For any µ > 0. Again using Lemma E.1, we have∥s(t, x t )∥ 2 = ∥∇ log q t (x t )∥ 2 ≤ 2∥x t ∥ 2 /σ 4 t + 2m 2 t R 2 /σ 4 t .Combining the results above, we haveE[∥Y k+1 ∥ 2 ] = (1 + γ 1,k τ ) 2 E[∥Y k ∥ 2 ] + (1 + η 2 ) 2 γ 2 1,k E[∥s(T -t k , Y k )∥ 2 ] + 2(1 + η 2 )(1 + γ 1,k τ )γ 1,k E[⟨Y k , s(T -t k , Y k )⟩] + 2η 2 γ 2,k d ≤ ((1 + γ 1,k τ ) 2 + 2(1 + η 2 ) 2 γ 2 1,k /σ 4 T -t k + 2(1 + η 2 )(1 + γ 1,k τ )γ 1,k (-1 + µm T -t k R)/σ 2 T -t k )E[∥Y k ∥ 2 ] + 2m 2 T -t k R 2 σ 4 T -t k (1 + η 2 ) 2 γ 2 1,k + m T -t k R µσ 2 T -t k (1 + η 2 )(1 + γ 1,k τ )γ 1,k + 4η 2 γ 1,k d . If we denote δ k = γ 1,k /σ 2 T -t k and notice the fact that m t ∈ [0, 1], σ 2 t ∈ [0, τ ], η ∈ [0, 1], then we have E[∥Y k+1 ∥ 2 ] ≤ (1 + 2δ k + δ 2 k )E[∥Y k ∥ 2 ] + 8δ 2 k E[∥Y k ∥ 2 ] + 2(1 + δ k )δ k (-1 + µR)E[∥Y k ∥ 2 ] + 8R 2 δ 2 k + 2R µ δ k (1 + δ k ) + 4τ δ k d .We also have thatγ 1,k = exp[ T -t k T -t k+1 β s ds] -1 ≤ exp[β T -t k γ k ] -1 ≤ 2β T -t k γ k ,where the last inequality follows that γ k = exp (-T ), β T -t k γ k ≤ 1/2 for small enough stepsize, and e ω -1 ≤ 2ω for any ω ∈ [0, 1/2]. We getδ k ≤ 2γ k β T -t k /σ 2 T -t k ≤ 2δ. Thus E[∥Y k+1 ∥ 2 ] ≤ (1 + 2δ k + 2δ k δ)E[∥Y k ∥ 2 ] + 16δ k δE[∥Y k ∥ 2 ] + 4(1 + δ)(-1 + µR)δ k E[∥Y k ∥ 2 ] + 16R 2 δ k δ + 4(1 + δ) R µ δ k + 4τ dδ k .Hence, we haveE[∥Y k+1 ∥ 2 ] ≤ (1 + δ k [-2 + 14δ + 4(1 + δ)µR])E[∥Y k ∥ 2 ] + δ k [16R 2 δ + 4(1 + δ) R µ + 4τ d] .We denote A = 2 -14δ -4(1 + δ)µR and B = 16R2 δ + 4(1 + δ) R µ + 4τ d, then E[∥Y k+1 ∥ 2 ] ≤ (1 -δ k A)E[∥Y k ∥ 2 ] + δ k B . Notice that E[∥Y 0 ∥ 2 ] = dτ and if E[∥Y k ∥ 2 ] ≥ B/A it is decreasing, if E[∥Y k ∥ 2 ] ≤ B/A we have E[∥Y k+1 ∥ 2 ] ≤ B/A + δB. so E[∥Y k ∥ 2 ] ≤ τ d + B(1/A + δ) .Notice that when δ ≤ 1/28,if we choose µ = 1/(4(1 + δ)R), A ≥ 1/2, andB ≤ 37R 2 + 4τ d .Then, the proof is complete. ■The following lemma shows a discretization error in the k-the interval. Lemma D.4. Assume Assumption 3.1,Assumption 4.1 andγ k β T -t k /σ 2 T -t k ≤ 1/28 for any k ∈ {0, • • • , K -1}. Then for any k, t ∈ [t k , t k+1 ] and i ∈ {1, 2}, we have that E[∥ Ȳt -Ȳt k ∥ 2 ] ≤ L i (τ )β T -t k γ k , where L i (τ ) = γK κ i (τ )( 64 σ 2 T -t K (i) + 8 τ )U 0 (τ ) + 64R 2 γK κi(τ ) σ 2 T -t K (i) + 4d, γK , κ i (τ ) is defined in Lemma D.5 and U 0 (τ ) is defined in Lemma D.3.Ȳt = Ȳt k + (exp[ T -t k T -t β s ds] -1)( 1 τ Ȳt k + (1 + η 2 )s(T -t k , Ȳt k )) + η(exp[2 T -t k T -t β s ds -1]) 1/2 Z ,where Z is a standard Gaussian random variable. By directly calculating, we have thatE[∥ Ȳt -Ȳt k ∥ 2 ] = 2(exp[ T -t k T -t β s ds] -1) 2 ( 1 τ 2 E[∥ Ȳt k ∥ 2 ] + (1 + η 2 ) 2 E[∥s(T -t k , Ȳt k )∥ 2 ]) + η 2 (exp[2 T -t k T -t β s ds] -1)d .By Lemma E.1 and accurate score function assumption,∥s(T -t k , Ȳt k )∥ 2 ≤ 2∥ Ȳt k ∥ 2 /σ 4 T -t k (i) + 2m 2 T -t k R 2 /σ 4 T -t k (i) . So we have that E[∥ Ȳt -Ȳt k ∥ 2 ] ≤ 2(exp[ T -t k T -t β s ds] -1) 2 (( 8 σ 4 T -t k (i) + 1 τ 2 )E[∥ Ȳt k ∥ 2 ] + 8R 2 σ 4 T -t k (i) ) + (exp[2 T -t k T -t β s ds] -1)d . By e 2w -1 ≤ 1 + 4w for any w ∈ [0, 1/2] and γ k sup v∈[T -t k+1 ,T -t k ] β v /σ 2 v ≤ 1/28 for any k ∈ {0, ..., K -1}, we have exp[ρ T -t k T -t β s ds] -1 ≤ 2ρβ T -t k γ k .for ρ = 1, 2. And using Lemma D.3 and Lemma F.2 we haveE[∥ Ȳt -Ȳt k ∥ 2 ] ≤ ( 64γ k σ 4 T -t k (i) + 8β T -t k γ k τ 2 )U 0 (τ )β T -t k γ k + 64R 2 γ k σ 4 T -t k (i) β T -t k γ k + 4dβ T -t k γ k .We denote L i (τ ) = γK κ i (τ )( 64 σ 2 T -t K (i) + 8 τ )U 0 (τ ) + 64R 2 γK κi(τ ) σ 2 T -t K (i) + 4d for i ∈ {1, 2}EQUATIONC i (τ ) = 2(2 + R 2 )(R + U 1/2 0 (τ )) + 2L 1/2 i (τ )τ 3/2 (1 + R 2 ) , for i ∈ {1, 2}.Then, for any s, u ∈ [0, t K ] with u ≥ s and i ∈ {1, 2}, we haveE[∥∆b u,i (( Ȳs,v ) v∈[s,T ] )∥] ≤ C i (τ )[κ 2 i (τ )σ -2 T -t K (i)γ 1/2 K + κ 2 i (τ )]γ 1/2 K ,where Ȳs,s ∼ N(0, I).Proof. Combining Lemma D.1, Lemma D.2 and the exact score function, we get∥∆b u,i (ω)∥ ≤ (1 + η 2 ) sup v∈[T -t k+1 ,T -t k ] (β 2 v /σ 6 v (i))(2 + R 2 )(diam(M + ∥ω u ∥))γ k + (1 + η 2 )(β T -u /σ 4 T -u (i))(1 + diam(M 2 ))∥ω u -ω s k ∥.For any u ∈ [T -t K , T ], using Lemma F.3 we have β u /σ 2 u (i) ≤ κ i (τ ). Hence,∥∆b u,i (ω)∥ ≤ (1 + η 2 ) sup v∈[T -t k+1 ,T -t k ] (β 2 v /σ 6 v (i))(2 + diam(M 2 ))(R + ∥ω u ∥)γ k + (1 + η 2 )(β T -u /σ 4 T -u (i))(1 + diam(M 2 ))(∥ω u -ω t k ∥) ≤ (1 + η 2 )(κ 2 i (τ )/σ 2 T -t k+1 (i))γ k (2 + diam(M 2 ))(R + ∥ω u ∥) + (1 + η 2 )κ 2 i (τ )(1 + R 2 )∥ω u -ω t k ∥/β T -u .Combining this with Lemma D.3 and Lemma D.4,E[∥∆b u,i (( Ȳs,v ) v∈[s,T ] )∥] ≤ (1 + η 2 )(κ 2 i (τ )/σ 2 T -t k+1 (i))γ K (2 + R 2 )(R + U 1/2 0 (τ )) + (1 + η 2 )κ 2 i (τ )(1 + R 2 )L 1/2 i (τ ) max{ β, τ } 3/2 γ1/2 K .We denoteC i (τ ) = 2(2 + R 2 )(R + U 1/2 0 (τ )) + 2L 1/2 i (τ )τ 3/2 (1 + R 2 ), for i ∈ {1, 2}, then we have E[∥∆b u,i (( Ȳs,v ) v∈[s,T ] )∥] ≤ C i (τ )((κ 2 i (τ )/σ 2 T -t k+1 )γ K + κ 2 i (τ )γ 1/2 K ).Lemma D.6. Assume Assumption 3.1 and Assumption 4.1,γ k sup v∈[T -t k+1 ,T -t k ] β v /σ 2 v ≤ 1/28 for any k ∈ {0, ..., K -1}. Let γK = argmax k∈{0,...,K-1} γ k , γ K = δ, and δ ≤ 1/32. Then W 1 R q τ ∞ K , Q q τ ∞ t K ≤ C i (τ )κ 2 i (τ )T exp R 2 2σ 2 T -t K (i) + (1 -η 2 ) 2 [ γ1/2 K σ 2 T -t K (i) + 1]γ 1/2 K ,where C i (τ ), κ i (τ ) for i ∈ {1, 2} are the same terms to Theorem 6.1.Proof. By Proposition 1 we have∥Y t K -Y K ∥ = ∥Y t K -Ȳt K ∥ ≤ t K 0 ∥∇Y u,t K ,i ( Ȳ0,u )∥∥∆b u,i (( Ȳ0,v ) v∈[0,T ] )∥du. ∥Y t K -Y K ∥ ≤ exp 1 + η 2 R 2 4σ 2 T -t (i) + (1 -η 2 ) 2 t K 0 β T -u τ du t K 0 ∥∆b u,i (( Ȳ0,v ) v∈[0,T ] )∥du .Then by definition of Wasserstein distance, we haveW 1 (q ∞ Q t K , q ∞ R K ) ≤ E[∥Y t K -Y K ∥] ≤ exp 1 + η 2 R 2 4σ 2 T -t K (i) + (1 -η 2 ) 2 t K 0 β T -u τ du t K 0 E[∥∆b u,i (( Ȳ0,v ) v∈[0,T ] ∥]du ≤ C i (τ )T exp 1 + η 2 R 2 4σ 2 T -t K (i) + (1 -η 2 ) 2 [κ 2 i (τ )σ -2 T -t K (i)γ 1/2 K + κ 2 i (τ )]γ 1/2 K . ■ Theorem 6.1. Assume Assumption 3.1 and 4.1, δ ≤ 1/32 and γ k sup v∈[T -t k+1 ,T -t k ] β v /σ 2 v ≤ 1/28 for ∀k ∈ {0, ..., K -1}. Let γ K = δ. Then, for ∀τ ∈ [T, T 2 ]: (1) If η = 1 (the reverse SDE), choosing β t = t 2 , W 1 R q τ ∞ K , q 0 is bounded by ( R τ + √ d) √ δ + exp R 2 2 ( β δ 3 + 1 τ ) C 1 (τ )T κ 2 1 (τ ) ( β δ 3 + 1 τ )γ 1/2 K + 1 γ1/2 K + De -T /2 √ τ ,whereκ 1 (τ ) = T 2 (1/τ + β/δ 3 ) and C 1 (τ ) is linear in τ 2 .(2) If η = 0 (PFODE), choosing a conservative β t (Assumption 3.1),W 1 R q τ ∞ K , q 0 is bounded by ( R τ + √ d) √ δ + exp R 2 2 ( β δ 2 + 1 τ ) C 2 (τ )κ 2 2 (τ )T ( β δ 2 + 1 τ )γ 1/2 K + 1 γ1/2 K + D √ τ ,where κ 2 (τ ) = T 1/τ + β/δ 2 and C 2 (τ ) is linear in τ 2 .Proof. To obtain the convergence guarantee, we need to control three error terms:W 1 R q τ ∞ K , q 0 ≤ W 1 R q τ ∞ K , Q q τ ∞ t K + W 1 Q q τ ∞ t K , Q q0P T t K + W 1 Q q0P T t K , q 0 . For term W 1 R q τ ∞ K , Q q τ ∞ t K , we use Lemma D.6.For the second term, we define Y x 0,t t∈[0,T ] and Y y 0,t t∈[0,T ] be the reverse processes with initial condition x and y. Then we have∥Y x 0,t -Y y 0,t ∥ ≤ ∥x -y∥ 1 0 ∥∇Y z λ 0,t ∥dλ ,where z λ = λx + (1 -λ)y. In this work, we choose x ∼ q τ ∞ and y ∼ q 0 P T . Combined with the above inequality, Theorem 4.2 and Lemma 6.3, we know that:W 1 Q q τ ∞ t K , Q q0P T t K ≤ exp R 2 2σ 2 T -t K (i) + (1 -η 2 ) 2 t K 0 β T -u τ du ∥q 0 P T -q τ ∞ ∥ ≤ √ m T D σ T exp R 2 2σ 2 T -t K (i) + (1 -η 2 ) 2 t K 0 β T -u τ du .For the last term, we use exactly the same process with Bortoli [2022] with bounded σ 2 T -t K :W 1 Q q0P T t K , q 0 ≤ E [∥X -m T -t K X + σ T -t K Z∥] ≤ ( R τ + √ d)σ T -t K ≤ 2( R τ + √ d) √ δ ,where the second inequality follows that σ 2T -t K + τ m T -t K = τ . ■In the end of the section, we provide the proof of Corollary 6.2. Corollary D.7. Assume Assumption 3.1, 4.1 and ∇ 2 log q t (x t ) ≤ Γ/σ 2 t . Let η = 0 (reverse PFODE), δ ∈ (0, 1/32), τ = T 2 , β t = t and κ 2 (τ ), C 2 (τ ) defined in Theorem 6.1, we haveW 1 R q τ ∞ K , q 0 ≤ ( R τ + √ d) √ δ + β Γ 2 δ Γ exp Γ + 2 2 C 2 (τ )κ 2 2 (τ )T (( β δ 2 + 1 τ )γ 1/2 K + 1)γ 1/2 K + D √ τ .Proof. The proof of this corollary is almost identical to the proof of Theorem 6.1. We just need to replace the first bound for the tangent process in Lemma 6.3 by the second bound. ■In this section, we introduce auxiliary lemmas to control the gradient and Hessian of the logarithmic density under the manifold hypothesis. Lemma E.1, Lemma E.2 and Lemma E.3 come from Lemma C.1, Lemma C.2, and Lemma C.5 of Bortoli [2022] . Since these lemmas do not involve the relationship between m t and σ t , we can directly use the results from Bortoli [2022] . Following Bortoli [2022] , we also define a empirical version of q 0 with N datapoints, i.e.q N 0 = (1/N ) N k=1 X k , with X k N k=1 ∼ q ⊗N 0 .We denote by q N t t>0 such that for any t > 0 the density w.r.t. the Lebesgue measure of the distribution of X N t , and when N → +∞, q N t = q t . Lemma E.1. Assume Assumption 4.1. Then for any t ∈ (0, T ] and x t ∈ R d we have that⟨∇ log q t (x t ), x t ⟩ ≤ -∥x t ∥ 2 /σ 2 t + m R ∥x t ∥/σ 2 t .In addition, we have∥∇ log q t (x t ) ∥ 2 ≤ 2∥x t ∥ 2 /σ 4 t + 2m 2 t R 2 /σ 4 t . Lemma E.2. Assume Assumption 4.1. Then for any t ∈ (0, T ], x t ∈ R d and M ∈ M d R d M, ∇ 2 log q t (x t ) M ≤ -1 -m 2 t R 2 / 2σ 2 t /σ 2 t ∥M ∥ 2 .In addition, we have∇ 2 log q t (x t ) ≤ 1 + R 2 /σ 4 t .The following lemma shows that the derivatives up to the fourth order are uniformly bounded since τ ∈ [T, T 2 ]. Thus we can use the stochastic extension of the Alekseev-Gröbner formula [Del Moral and Singh, 2022] . Lemma E.3. Assume Assumption 4.1. Then, there exists C ≥ 0 such that for any t ∈ (0, T ] we have∇ 2 log q t (x) + ∇ 3 log q t (x) + ∇ 4 log q t (x) ≤ C/σ 8 t .The following lemma shows that ∥∂ t ∇ log q t (x t ) ∥ is bounded. The proof before using the relationship between σ t and m t is identical compared to Lemma C.3 in Bortoli [2022] . For the sake of completeness, we also give the proof process of this part. Lemma E.4. Assume Assumption 4.1. Then for any t ∈ (0, T ] and x t ∈ R d we have∥∂ t ∇ log q t (x t ) ∥ ≤ β t /σ 6 t 2 + R 2 (R + ∥x t ∥) .Proof. Let N ∈ N and t ∈ (0, T ]. We denote for anyx ∈ R d , q N t (x) = qN t (x) / 2πσ 2 t d/2 with qN t (x) = (1/N ) N k=1 e k t (x) , e k t (x) = exp -∥x -m t X k ∥ 2 / 2σ 2 t .Next we denote f k t ≜ log e k t . Then we have∂ t log qN t (x) N k=1 ∂ t f k t (x) e k t (x) / N k=1 e k t (x) .Therefore we have∂ t ∇ log qN t (x) = N k=1 ∂ t ∇f k t (x)e k t (x)/ N k=1 e k t (x) + N k=1 ∂ t f k t (x)∇f k t (x)e k t (x)/ N k=1 e k t (x) - N k,j=1 ∂ t f k t (x)∇f j t (x)e k t (x)e j t (x)/ N k,j=1 e k t (x)e j t (x) = N k=1 ∂ t ∇f k t (x)e k t (x)/ N k=1 e k t (x) + (1/2) N k,j=1 ∂ t f k t (x) -∂ t f j t (x) ∇f k t (x) -∇f j t (x) e k t (x)e j t (x)/ N k,j=1 e k t (x)e j t (x).In what follows, we provide upper bounds for|∂ t f k t -∂ t f j t |, ∥∇f k t -∇f j t ∥ and ∂ t ∇f k t . First we notice that ∇f k t (x) = -x -m t X k /σ 2 t, and using m t ≤ 1 we get∥∇f k t (x) -∇f j t (x)∥ ≤ m R /σ 2 t ≤ R/σ 2 t . and ∂ t f k t (t) = ∂ t σ 2 t / 2σ 4 t ∥x -m t X k ∥ 2 + ∂ t m t /σ 2 t X k , x -m t X k . Notice the fact that ∂ t σ 2 t = -2τ m t ∂ t m t = 2β t m 2 t and ∂ t m t = - β t τ m t, combined with the above equality, we know that∂ t f k t (t) = -β t m t /σ 2 t -m t /σ 2 t ∥x -m t X k ∥ 2 + 1 τ x -m t X k , X k = -β t m t /σ 2 t x -m t X k , -m t /σ 2 t x -m t X k + 1 τ X k = -β t m t /σ 4 t x -m t X k , -m t x + m 2 t + σ 2 t τ X k = β t m t /σ 4 t m t ∥x∥ 2 + m t X k 2 + 1 + m 2 t x, X k ,where the last equality holds that τ m 2 t + σ 2 t = τ . The rest of the proof is identical to the Lemma C.3 in Bortoli [2022] .So using m t ≤ 1 we have∂ t f k t (x) -∂ t f j t (x) ≤ 2β t m 2 t R 2 /σ 4 t + β t m t 1 + m 2 t R∥x∥/σ 4 t ≤ 2 β t /σ 4 t R(R + ∥x∥) Now we compute ∇∂ t f k t (x) for any x ∈ R d ∇∂ t f k t (x) = 2β t m 2 t /σ 4 t x + β t m t /σ 4 t 1 + m 2 t X k .So we can bound the norm of it by∥∂ t ∇f k t (x) ∥ ≤ 2 β t /σ 4 t (R + ∥x∥).Combining results above we get for anyx ∈ R d ∂ t ∇ log qN t (x) ≤ 2 β t /σ 4 t (R + ∥x∥) + β t /σ 6 t R 2 (R + ∥x∥) ≤ β t /σ 6 t 2 + R 2 (R + ∥x∥) Note that lim N →+∞ ∂ t ∇ log q N t (x t ) = ∂ t ∇ log q tand the proof is complete. ■In the following lemma, similar to Chen et al. [2023c] , we obtain a better control on the time discretization error instead of controlling ∥∂ t ∇ log q t (x t )∥ for ∀x t ∈ R d .Lemma E.5. Assume Assumption 4.1 and X t satisfies the forward process Equation (3). DefineL = max t∈[0,T -δ] ∇ 2 log q T -t (Y t ) ≤ 1 + R 2 /σ 4 δ , then we have that E Q q τ T t K ∇ ln q T -t k q T -t (Y t k ) 2 ≲ τ L 2 dγ K + τ L 2 γ2 K (dτ + R 2 ) + τ L 3 γ2 K + τ L 4 γ2 K (β T dγ K + R 2 γ2 K ) .Proof. Due to the property of the forward process, we know that ifS : R d → R d is the mapping S(x) := exp(-(t -t k ))x, then q T -t k = S # q T -t * normal 0, τ 1 -exp(-2 t k +1 t k β s /τ ds)Similar to Chen et al. [2023c] , we define α = expt k+1 t k βs τ ds = 1 + O(γ K ) and σ 2 = τ 1 -exp(-2 t k +1 t k β s /τ ds) = O(τ γK ).Then we can use Lemma C.12 of Lee et al. [2022] to obtainE Q q τ T t K ∇ ln q T -t k q T -t (Y t k ) 2 ≲ τ L 2 dγ K + τ L 2 γ2 K ∥Y t k ∥ 2 + τ L 2 γ2 K ∥∇ ln q T -t (Y t k )∥ 2 ≲ τ L 2 dγ K + τ L 2 γ2 K (dτ + R 2 ) + τ L 3 γ2 K + τ L 4 γ2 K (β T dγ K + R 2 γ2 K ). The last inequality follows Lemma F.4 and the fact that∥∇ ln q T -t (Y t k )∥ 2 ≲ ∥∇ ln q T -t (Y t )∥ 2 + ∥∇ ln q T -t (Y t k ) -∇ ln q T -t (Y t )∥ 2 ≲ ∥∇ ln q T -t (Y t k )∥ 2 + L 2 (β T dγ K + R 2 γ2 K ) ≲ L + L 2 (β T dγ K + R 2 γ2 K ) . ■ F Auxiliary Lemmas Lemma F.1. For any s, t ∈ [0, T ] we have t s β T -u /σ 2 T -u du = - 1 2 log exp 2 T -u 0 β v τ dv -1 t s , t s β T -u m 2 T -u /σ 4 T -u du = (1/2τ ) / 1 -exp -2 T -u 0 β v τ dv t s .Proof. We directly computet s β T -u /σ 2 T -u du = 1 τ t s β T -u / 1 -exp -2 T -u 0 β v τ dv du = 1 τ t s β T -u exp 2 T -u 0 β v τ dv / exp 2 Tu 0 β v τ dv -1 du = - 1 2 t s ∂ u log exp 2 T -u 0 β v τ dv -1 du . Similarly t s β T -u m 2 T -u /σ 4 T -u = 1 τ 2 t s β T -u exp -2 T -u 0 β v τ dv / 1 -exp -2 T -u 0 β v τ dv 2 du = (1/2τ ) s t ∂ u 1 -exp -2 T -u 0 β v τ dv -1 du. ■ Lemma F.2. Assume Assumption 3.1. For i ∈ {1, 2}, we have σ 2 T -t K (i) ≤ 2δ and σ -2 u (i) ≤ σ -2 T -t K (i) ≤ 1 τ + β δ 4-i , ∀u ∈ [T -t K , T ].Proof.σ 2 T -t K (i) = τ 1 -exp -2 T -t K 0 β s τ ds ≤ 2 T -t K 0 β s ds ≤ 2δ ,where the first inequality follows from for any a ≥ 0, exp[-a] ≥ 1 -a; the second inequlity follows from Assumption 3.1 and δ ≤ 1.σ -2 T -t K (i) = 1 τ 1 -exp -2 T -t K 0 β s τ ds -1 ≤ 1 τ   1 + 2 T -t K 0 β s τ ds -1   ≤ 1 τ + β δ 4-i, where the first inequality follows from for any a ≥ 0, 1/(1+exp[-a]) ≤ 1+1/a, the second inequality follows from Assumption 3.1. It is easy to check that σ -2u (i) ≤ σ -2 T -t K (i), ∀u ∈ [T -t K , T ]. ■Using the bound on σ -2 T -t K (i) immediately yields the following control of β u /σ 2 u (i). Lemma F.3. Assume Assumption 3.1. Then, we have for anyu ∈ [T -t K , T ]: (1) if i = 1, then β u σ 2 u (i = 1) ≤ κ 1 (τ ) = max{ β, T 2 } 1 τ + β δ 3 ;(2) if i = 2, thenβ u σ 2 u (i = 2) ≤ κ 2 (τ ) = max{ β, T } 1 τ + β δ 2 .Generally speaking, T ≥ β ≥ 1. Hence, We can further simplify the above inequality by removing max.In the rest of this section, we provide the useful lemma to achieve polynomial sample complexity for VE-based models with reverse SDE. As shown in Lemma E.1, we also need to control E[∥X t ∥ 2 ] in the forward process. The following lemmas shows that this term is bounded by the R 2 and exploding variance.Lemma F.4. Suppose that Assumption 4.1hold. Let (X t ) t∈[0,T ] denote the forward process Equation (3). Then, for all t ≥ 0,E ∥X t ∥ 2 ≤ dσ 2 t ∨ R 2 .Proof. As shown in Equation ( 4),E ∥X t ∥ 2 ≤ E ∥X 0 ∥ 2 + σ 2 t d ≤ dσ 2 t ∨ R 2 .■ Lemma F.5 (movement bound for VESDE). Let (X t ) t∈[0,T ] denote the forward process Equation (3). For 0 ≤ s < t with δ := t -s, if δ ≤ 1, thenE ∥X t -X s ∥ 2 ≲ 2β t δd + δ 2 R 2 .Proof.E ∥X t -X s ∥ 2 ≲ E 2β t (B t -B s ) 2 + δ s E ∥X r ∥ 2 dr ≲ 2β t δd + δ 2 R 2 .■ Similar to Chen et al. [2023c] , we can also show that if we do forward process for time δ, q δ will be close to q 0 in W 2 distance. Lemma F.6. Suppose Assumption 4.1 holds. Let ϵ W2 > 0. If β 2 t = t 2 and τ = T 2 , we choose the early stopping parameter δ ≤ϵ 2/3 W 2 (d+R √ d) 1/3 . If β t = t and τ = T , we choose δ ≤ ϵ W 2 (d+R √ d) 1/2 . If consider pure VESDE (SMLD) (Equation (2)) with σ 2 t = t, we choose δ ≤ ϵ 2 W 2d . Then we have W 2 (q δ , q 0 ) ≤ ϵ W2 .Proof. For the forward process Equation (3), we know that X t := m t X 0 + σ t Z, where Z ∼ normal (0, I d ) is independent of X 0 and m t ≤ 1. Hence, for δ ≲ 1,W 2 2 (q 0 , q δ ) ≤ (1 -m t ) 2 E ∥X 0 ∥ 2 + E ∥σ δ Z∥ 2 .For β t = t 2 and τ = T 2 , we have thatW 2 2 (q 0 , q δ ) ≤ δ 3 d + R 2 δ 6 T 2Hence, we can take δ ≤ϵ 2/3 W 2 (d+R √ d) 1/3. For β t = t and τ = T , we have thatW 2 2 (q 0 , q δ ) ≤ δ 2 d + R 2 δ 4 T 2Hence, we can take δ ≤ ϵ W 2 (d+R √ d) 1/2 . For pure VESDE (Equation ( 2)) with σ t = t, we have W 2 2 (q 0 , q δ ) ≤ δd .In this section, we do synthetic experiments to show the power of our new forward process with small drift term in different setting.In this section, we do numerical experiments on 2-dimension Gaussian distribution to show the power of our new VESDE forward process in balancing different error sources.Experiment Setting. We set the mean of target distribution E[q 0 ] = [6, 8], the covariance matrix Cov[q 0 ] = 25 5 5 4 , the diffusion time T = 2, τ = T 2 and the reverse beginning distribution is N (0, T 2 I). We choose uniform stepsize γ k = h, ∀k ∈ [K] where h ∈ {0.005, 0.01, 0.02, 0.04}. For score functions, we directly calculate the ground truth score function instead of learning it by the score matching objective. We calculate the KL divergence between the generation distribution and target distribution q 0 as the experiments.The implementable algorithm. We choose three different VESDE forward processes in the experiments: (1) aggressive β t = t 2 with τ = T 2 ; (2) conservative β t = t with τ = T 2 and (3) VESDE without drift term Equation ( 2) with σ 2 t = t 2 . After determining the forward process, we run the reverse SDE with the above γ k , k ∈ [K]. For the discretization scheme, we choose two common method: exponential integrator (EI) [Zhang and Chen, 2022 ] and Euler-Maruyama (EM) discretization [Ho et al., 2020] .Observations. The experimental results are shown in Figure 2 . We note that the red line (EI, VESDE without drift, σ 2 t = t 2 ) and orange line (conservative drift VESDE, β t = t and τ = T 2 ) has a similar trend. Furthermore, the conservative drift VESDE has better performance compared to pure VESDE without drift term. Hence, our new forward process is representative enough to represent current VESDE, as discussed in Section 3.1.The experimental results also support our theoretical results and show the power of the new forward process in balancing different error terms. As shown in Figure 2 , the process with aggressive β t = t 2 with small drift term achieves the best and second performance in EI and EM discretization since it can balance the reverse beginning and discretization. The third best process is conservative β t = t with the small drift term. The reason is that though it can not achieve a exp (-T ) forward process guarantee, it also has a constant decay on prior information, as shown in Section 3.1. This decay slightly reduces the effect of the reverse beginning error. The worse process is VESDE without drift term since it is hard to balance different error sources. Our experimental results also show that EI discretization is better than EM discretization.In this section, instead of using an accurate score function, we train an approximated score function on the pure VESDE (Equation ( 2)) without drift term on two synthetic datasets: multiple Swiss rolls and 1-D GMM. Then, for the drift VESDE, we do not train the approximated score corresponding to Equation (3); we directly use the approximated score learned by pure VESDE and show that the drift VESDE can improve the generated distribution without the training process. For multiple Swiss rolls, we use a similar code compared to Listing 2 of Lai et al. [2023] , except Line 6. We change Line 6. to data /=10. to obtain a larger variance dataset. Each dataset contains 50000 datapoints.The implementable algorithm. In this subsection, we choose two forward processes: (1) conservative β t = 1 with τ = T ; (2) pure VESDE without drift term (Equation ( 2)) with σ 2 t = t. To match We note that although aggressive setting β t = t and τ = T has shown its power in theory (Lemma C.2) and the experiments with accurate score (Figure 2 ), other sampling issues may arise in practice. We leave the experimental exploration for drift VESDE with aggressive β t as a future work.The training detail. For each dataset, we train a score function with pure VESDE (Equation ( 2), σ 2 t = t). We train for 200 epochs with batch size 200 and learning rate 10 -4 . For both training and inference, the start time is δ = 10 -5 . For the conservative VESDE, we directly adapt the checkpoint learned by the pure VESDE since the conservative drift VESDE has a similar trend compared to pure VESDE, as shown in Figure 2 . The above experiments are runned over 5 random seed and we present the average over these seeds in Table 1 .The above experiments are conduct on a GeForce RTX 4090. It takes 25 minutes to train a score function of pure VESDE.Observation. We do experiments with T = 100 and lager T = 625 and these two choice show similar phenomenon. In this paragraph, we first use T = 100 as an example to discuss the results. As shown in Table 1 , the conservative drift VESDE has smaller KL divergence compared to pure VESDE under all sampling methods and datasets. From Figure 1 and Figure 4 , it is clear that pure VESDE has low density on the Swiss roll except the center one, which means that though pure VESDE can deal with small E[q 0 ], it is hard to deal with large dataset variance Cov[q 0 ], as we discuss in Section 4. For conservative drift VESDE (β t = 1 and τ = T ), as we discuss in Section 3.1, there is a constant decay on the prior information E[q 0 ] and Cov[q 0 ], which is helpful in deal with large dataset mean and variance. The experimental results support our augmentation. Figure 1 (c), Figure 4 (c) and Figure 5 (c) show that the density of the generated distribution is more uniform compared to pure VESDE, which means that the drift VESDE can deal with large dataset mean and variance.We also do experiments with larger T = 625. As we discuss in Section 4, larger T will reduce the influence of the prior data information and have greater generated distribution, as shown in Figure 4 (c) and Figure 4 (e) . The experiments of 1D-GMM (Figure 5 ) show a similar phenomenon compared to the multi Swiss rolls.After achieving great performance under the synthetic data, we show that our conservative drifted VESDE can improve the results of pure VESDE without training.Setting. In this experiment, we adapt well-known VESDE implementation [Song et al., 2020b] and do experiments on CelebA datasets (size: 256 * 256 * 3). More specifically, we use ve/celebahq_256_ncsnpp_continuous checkpoints provided by [Song et al., 2020b] and modify the sampling process strictly according to our drifted VESDE. To do a fair comparsion, we fix the random seed and use the reverse PFODE process. Then, we generate 10000 face images to calculate the metrics. We note that when using this checkpoint and pure VESDE pipeline provided by [Song et al., 2020b] , the models would generate almost pure noise with a certain probability. Hence, we use an aesthetic predictor [Schuhmann et al., 2022] (aesthetic score ≥ 5.5) to filter the generated images to ensure that the images are clear faces. We note that the goal of these experiments is to show that our conservative drifted VESDE is plugand-play without training instead of achieving a SOTA performance. Hence, we focus on the relative improvement compared to the baseline [Song et al., 2020b] . There are two interesting empirical future works. For the conservative drifted VESDE, we will do experiments on the SOTA pure VESDE models [Karras et al., 2022] and improve their results without training. For the aggressive drifted VESDE, since this process makes a larger modification compared with the conservative one, we need to train a new score function instead of directly using a pre-train one to achieve better results. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.We note that the choice τ ∈ [T, T 2 ] is used to guarantee the exploding variance of the forward process. In fact, our drifted VESDE is a general formula that covers the VP and VESDE by choosing τ ∈ [1, T 2 ]. More details are shown in Section 5.Now (Bt) t≥0 is the reversed Brownian motion, and we abuse this notation here for ease of notation.We note these results still hold for pure VESDE with σ 2 t = t and t 2 , and we use the general drifted VESDE for simplicity. Here, we only consider the dependence of ϵ. Readers can find detailed results in Appendix C.1. 


Decision: The paper received positive reviews from most reviewers and the several concerns have been clarified during the rebuttal period. After discussing with reviewer bHCt, I have come to the conclusion that the paper merits an acceptance, given all the promised changes/clarifications being implemented. Please make sure you carefully incorporate all the reviews/discussions into account.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose an analysis of the convergence of diffusion models under the manifold hypothesis in a similar setting as [1]. The main contribution is the extension of the analysis to the case of VESDE (Variance Exploding SDE) contrary to [1] which is limited to VPSDE (Variance Preserving). The rates obtained by the authors are better than the ones obtained in [1] (although in a different context). They also extend their analysis to ODE samplers which are notably more difficult to deal with than SDE samplers from a theoretical point of view. The decomposition of the error is the same as in [1] but with a more careful analysis of the tangent process (see Section 6 "The Tangent-based Analysis Framework"). Experiments in toy settings are presented. 

[1] De Bortoli -- Convergence of denoising diffusion models under the manifold hypothesis
strength_weakness: strength_weakness: * Experiments are only toyish. I actually don't think they benefit the paper. This is mostly theoretical work and I'm struggling to understand what point is made here. If this is to illustrate the validity of the samplers this is already well established. If the point of the experiment is to illustrate the benefit of drifted VESDE then I would have appreciated a more challenging setting (like CIFAR10 or Imagenet in image processing or larger models like Stable Diffusion). This does not require pretraining a large model since this is a modification of the sampler.

* There is actually not a lot of discussion on how drifted VESDE relate to VESDE. Can one obtain convergence results for  VESDE (classical) based on drifted VESDE?

* As of now, the analysis is limited to drifted VESDE. It would be interesting to analysis if the improved results can transfer to the VPSDE framework and improve on the results of [2].

* The paper is easy to follow and clearly presented. 

[1] Karras et  al. -- Elucidating the Design Space of Diffusion-Based Generative Models

[2] De Bortoli -- Convergence of denoising diffusion models under the manifold hypothesis
questions: questions: * l.34 "Furthermore, Karras et al. [2022] unify two processes and show that the optimal parameters of the general formula correspond to VESDE." --> Not clear what the authors are referring to here.

* l.48 "leads to a large reverse beginning error" --> I disagree as one could argue that most of the error in diffusion models arises from the approximation of the score. The "large reverse beginning error" is a strong statement here. 

* One relevant work that is not discussed is [1]

* l.296 "Furthermore, we emphasize that our tangent-based unified framework is not a simple extension of Bortoli [2022]." --> Can the authors provide more details here?

* Since there is a one-to-one mapping between VESDE and VPSDE could the authors have leveraged this connection? There is also a connection with Stochastic Localization as pointed out by [2]. Below we explicit the connection between VESDE and VPSDE.

Assume that $(X_t)_{t \geq 0}$ satisfies a VESDE $\mathrm{d} X_t = g(t) \mathrm{d} B_t$ then we have that $(Y_t)_{t \geq 0}$ given for any $t \geq 0$ by $Y_t = \exp[F(\phi(t))] X_{\phi(t)}$ satisfies $\mathrm{d} Y_t  = F(\phi)'(t) Y_t \mathrm{d}_t  + \exp[F(\phi(t))] \phi'(t)^{1/2} \mathrm{d} B_t$.

[1] Conforti et al. -- "Score diffusion models without early stopping: finite Fisher information is all you need"

[2] Montanari -- "Sampling, Diffusions, and Stochastic Localization"
limitations: limitations: Limitations are not really addressed in Section 8 ("Conclusion") in the paragraph "Future Work and Limitation". I think a more in depth discussion of the benefits of VESDE and VPSDE is needed.
paper_summary: paper_summary: This paper focuses on variance exploding (VE) based diffusion models and proposes a drifted VESDE forward process with an unbounded diffusion coefficient. This choice of coefficients allows an exponential-decay forward convergence rate, and the authors establish the first polynomial sample complexity for VE-based models with reverse PFODE. Moreover, the authors propose a tangent-based unified analysis framework with reverse SDE and PFODE and prove the first quantitative guarantee for SOTA VE-based models with reverse PFODE.
strength_weakness: strength_weakness: 1. The convergence guarantees for VE-based models with reverse PFODE are relatively weak. For example, in Assumption 3.1, the choice of $\beta_t$ is more conservative for reverse PFODE; In Theorem 6.2 part (2), the last term is $\bar{D}/\tau$ instead of $\bar{D} e^{-T/2} / \sqrt{\tau}$ in part (1). Does this mean that the $e^{-T}$ forward convergence rate can only achieved by the reverse SDE?

2. The numerical results only include synthetic experiments.
questions: questions: 1. In Assumption 3.1, why for reverse SDE and PFODE, the choices of $\beta_t$ are different? Could the authors provide some intuitive explanation?

2. In Theorem 5.2, the sample complexity has the same dependence on $\epsilon_{W_2}$ and $\epsilon_{TV}$ as that in Chen et al. (2023c). However, the dependence on $d$ is worse. Is it a consequence of a more aggressive $\beta_t$? 

3. Could the theoretical results reflect the superiority of VE-based diffusion models over VP-based diffusion models?
limitations: limitations: The authors discuss future work and limitations in Section 8.
paper_summary: paper_summary: The paper analyzes the Variance Exploding diffusion model under the manifold hypothesis. By a slight modification to the VESDE process, the authors propose a method whose convergence guarantees are better than prior best known rates in this regime.
strength_weakness: strength_weakness: The experiments in the main text and appendix are not particularly thorough. This is acceptable since the primary claimed contribution by this article is theoretical, but some kind of practical benchmark might also be beneficial.

The dimension and diameter dependence are quite severe. Is there any sense of how close these parameters are to their optimal values?
questions: questions: The final rates are only given in total variation. Is it possible to improve the metric to KL or to state W2 bounds with similar complexities? What is the analytical challenge otherwise?

Theorems 6.1 and Corollary 6.2 could have their exposition simplified in the main text; the intuition provided in the subsequent section is much clearer.

Minor:

The font used for "KL" is not consistent in the equations in the appendix.

97: about data -> about the data

102: score function -> the score function

107: with strong LSI -> with a strong LSI

108: assume the Lipschitz score -> assume the score is Lipschitz

108: first work focus -> first work to focus

138: introduces -> introduce

187: "reversing" does not make sense here. Perhaps "initial distribution of the reverse process" is meant

216: "does reverse" -> "which reverse the process"

345: "support" -> "supports"

364: "support our", "show that" -> "supports our", "shows that"
limitations: limitations: The authors have already addressed all major limitations of their work.
paper_summary: paper_summary: Diffusion models are powerful tools in generative modeling.  As the paper pointed out, very few theoretical works in the literature
consider variance exploding diffusion models. Among those works, forward convergence rate $1/\text{poly}(T)$ is achieved compared to $\exp(-T)$ from the variance preserving models.  This paper proposes a drifted variance exploding diffusion model that allows a faster $\exp(-T)$ forward convergence rate.  With this process, the polynomial sample complexity for a series of variance exploding models is achieved under the manifold hypothesis.  In addition to the reverse SDE, probability flow ODE is a popular alternative in the literature. The paper considers a more general setting and proves a convergence guarantee with probability flow ODE.
strength_weakness: strength_weakness: (1) The model seems to be an interplay between the VE-SDE model and VP-SDE model  in the literature in the sense that when $\tau=1$, it recovers the VP-SDE model and then $\tau=\infty$, it recovers the VE-SDE model. To better compare the model with the VP-SDE and VE-SDE models in the literature, I am wondering whether it is better to assume $\tau=[1,\infty]$ and for a fixed $\tau$, to see what is the convergence guarantees and how it depends on $\tau$? If you do that, will you find the optimal range of $\tau$ to be $[T,T^{2}]$ which is what proposed in the paper? My guess is that for given $\beta_{t}$, the optimal choice of $\tau$ should depend on $\beta_{t}$ instead of being in the range $[T,T^{2}]$.

(2) Since the model proposed is a new model, it would be even more convincing if the paper can include some experiments beyond synthetic experiments.
questions: questions: (1) You studied both total variation distance and 1-Wasserstein distance. Since your data is supported on a compact domain, would it be possible to study more general $p$-Wasserstein distance guarantees?

(2) After Assumption 3.1. you wrote that the choice $\tau\in[T,T^{2}]$ is used to guarantee the exploding variance of the forward process. This is not that accurate. I understand when $\tau$ is large, the mean-reverting effect of the forward SDE gets very weak, and the variance tends to grow. But that does not explain why you need $\tau\leq T^{2}$. I think you should also add some discussions on why you need $\tau\leq T^{2}$.
limitations: limitations: Limitations have been discussed in the conclusion section.

Comments: comment: We thank you once again for your careful reading of our paper and your constructive comments and suggestions. As the discussion period approaches its end,  we will appreciate it very much if you could let us know whether all your concerns are addressed. We are also more than happy to discuss our work and answer any further questions.
Thank you for your positive feedback and support! We will add the discussion and polish our presentation according to your comments. In case you have any other questions, please don't hesitate to let us know.
Thank you for your detailed response. In light of the numerous improvements shown both in your response to me and to the other reviewers, I will raise my score by 1.
Thank you for your positive feedback and support! In case you have any other questions, please don't hesitate to let us know.
Thank you for your detailed response again. I will raise my score to 6.
Thanks for your effort and time. We discuss each question in detail below.

(Question 1) The understanding in the response is correct. Section 6 provides a unified tangent-based framework for the VE-based models (including reverse SDE and PFODE). Hence, we achieve a slightly worse convergence guarantee compared to the results in Section 5 when considering the reverse SDE. In Section 5, we focus on the reverse SDE setting and give a more refined analysis. The more refined analysis shows that our drifted VESDE can balance different error terms and achieve better complexity results compared to pure VESDE. We will add the above discussion and polish our presentation according to your helpful comments.



(Question 2) In this part, we provide the sample complexity of Corollary 6.2. More specifically, by choosing  $\delta \leq \epsilon_{W_1}^2/d$, $T\ge \frac{\bar{D}\exp{(\Gamma)}\beta^{\Gamma/2}}{\delta^{\Gamma}\epsilon\_{W_1}}$ and $\bar{\gamma}\_K\leq \frac{\epsilon\_{W_1}^2\delta^{2\Gamma}}{C_2^2(\tau)\kappa_2^4(\tau)T^2\exp{(2\Gamma)}\beta^{\Gamma}}$, we have $W_1\left(R_K^{q_{\infty}^\tau}, q_0\right)\leq \epsilon_{W_1}$ with the sample complexity
$$
K\leq \frac{\bar{D}\exp{(3\Gamma)}\beta^{3\Gamma/2}C_2^2(\tau)\kappa_2^4(\tau)T^2}{\delta^{3\Gamma}\epsilon_{W_1}^3}.
$$
We note that compared with Thm. 6.1, the above complexity replace the exponential dependence on $\delta$ by a polynomial $\delta$ and an exponential $\Gamma$. Since $\delta$ is related to the $\epsilon_{W_1}$ and $\Gamma$ is only determined by the data structure, this result improves Thm.6.1. As discussed at the end of Section 6, an interesting future work is to introduce a suitable corrector to inject some small noise to the PFODE sampler and achieve a polynomial sample complexity (w.r.t all problem parameters) under the manifold hypothesis. We will add the above result and discuss it in Section 6.

We hope the above discussion can address your concerns. We are more than happy to discuss our work in detail and answer any further questions in the rebuttal phase.
Thank you for your detailed response. 

I have an additional concern. It seems that Theorem 6.1 implies an exponential dependence of sample complexity on the inverse of the target accuracy. Is it correct to understand Section 6 as providing a more general analysis of VE-based models to encompass the reverse PFODE, whereas the analysis in Section 5 is more refined for the reverse SDE? 
Additionally, could the authors elaborate on the dependence of the sample complexity on the target accuracy for the reverse PFODE under the extra assumption in Corollary 2?
Thanks again for your effort in reading our real-world experiments and rebuttal. We discuss each question in detail below.

(Question 1.) As shown at the end of W2 (a), we can obtain a polynomial sample complexity for the pure VESDE (with zero drift). We use pure VESDE ($\sigma_t^2=t^2$) as an example
$$
\mathrm{d} \mathbf{X}_t=\sqrt{2t} \mathrm{d} \mathbf{B}_t.
$$
In this case, the convergence guarantee is
$$
\begin{align}
    \frac{\bar{D} }{T}+\frac{R^2\sqrt{d}}{\delta^4}\sqrt{\bar{\gamma}_KT^4}+\epsilon\_{\text{score}}\sqrt{T^2},
\end{align}
$$
whose bound does not blow up. However, the above bound still has difficulty in balancing the reverse beginning and discretization error term. Hence, we propose our drifted VESDE to balance these two terms. 

For the drifted VESDE, as mentioned in your response,  the formula can not recover pure drifted VESDE by setting $\tau \rightarrow +\infty$, and we need the above independent theorem to give a guarantee for the pure VESDE. However, we also note that with a conservative $\beta_t$ (for example, $\beta_t=t$ when $\tau =T^2$), the performance of conservative VESDE is similar but better than pure VESDE (The red and brown line of our Sec. 7.1, Fig 2 and our real-world experiments). Hence, we present the sample of drifted VESDE with reverse SDE for the sake of coherence. We will add the above results of pure VESDE as an independent theorem and make our presentation clearer according to your comments in the next version.

(Question 2.) The learning process of the score function is an important part of the analysis of diffusion models. However, when considering the sample complexity, most current theoretical works assume a $L_2$ accuracy score function [1] [2] [3] [4] [5], and we follow this standard assumption in our work. In this work, we take the first step in analyzing the great performance of VE-based models from the sample complexity perspective. As mentioned in the response, the choice of different processes will influence the score learning process. Hence, an end-to-end analysis (considering the sampling and learning process simultaneously) for the VE-based models is a really interesting future work, and we will add a detailed discussion in our future paragraph.

(Question 3.) When discussing Karras et al., we want to show that the linear solution trajectory of VESDE ($\sigma_t^2=t^2$) is more friendly compared with the one of VPSDE when considering the sampling phase. The great performance has been shown in many areas, such as the one-step consistency model (including the follow-up works) [6]  and video generation models [7]. For the consistency models, they use the forward process proposed by Karras et al. due to the linear trajectory. For the Stable Video Diffusion, they also use the noise schedule to obtain a pre-trained base model (Sec. 4.1 of their paper). We will improve our presentation according to your comments and add the above discussion to our introduction paragraph.



[1] Chen et al. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. ICLR 2023.

[2] Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., & Salim, A. (2024). The probability flow ode is provably fast. *Advances in Neural Information Processing Systems*, *36*.

[3] De Bortoli, V. Convergence of denoising diffusion models under the manifold hypothesis. TMLR.

[4] Benton, J., Bortoli, V. D., Doucet, A., & Deligiannidis, G. (2024). Nearly d-linear convergence bounds for diffusion models via stochastic localization.

[5] Lee et al. Convergence for score-based generative modeling with polynomial complexity. NeurIPS 2022.

[6] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models. *arXiv preprint arXiv:2303.01469*.

[7] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., ... & Rombach, R. (2023). Stable video diffusion: Scaling latent video diffusion models to large datasets. *arXiv preprint arXiv:2311.15127*.
Thank you for your answer and the additional experiments. I would like to keep my score to  6.

I would also like to point out that I think there was a misunderstanding regarding W2. 
When I am talking about VESDE (Brownian motion) I am talking about the following dynamics.

$$ \mathrm{d} \mathbf{X}_t = g(t) \mathrm{d} \mathbf{B}_t . $$

To the best of my knowledge this is not described by (3). I have no doubt that VPSDE can be recovered. Regarding VESDE I am less sure. For example in the bound provided by the authors in the rebuttal to get to VESDE (I insist on having a zero drift), I need $\tau \to +\infty$. Unless, I am mistaken, this means that the bound provided by the authors blow up. 

This is the discussion I was asking for. 

I also think that the authors missed my point in Q2. My point was that improving on the discretization errors in SDE models might not be the right term to look at. Indeed, choosing VPSDE, VESDE (drifted or not), implies different sampling bounds (as illustrated by this work), but some of these choices also affect the learning of the score. It is hard to disentangle those parts. 

Regarding Q1, I don't think that Karras et al. "prove" the optimality of VESDE (and a reference to a Figure 3 is not sufficient).
We would like to thank the AC and reviewers again for their time and efforts! We appreciate that the reviewers highlight the contribution of this paper, such as the novel drifted VESDE, the non-trivial tangent process control lemma, and the rigorous analysis.

Also, reviewers provided valuable comments and suggestions to help us polish our work. The two common concerns are (1) the real-world experiments and (2) the extension of our general drifted VESDE. For the real-world experiments, we show that our drifted VESDE is a **plug-and-play** method without training and can generate more realistic images compared with the baseline from the quantitative and qualitative perspective. For the extension of drifted VESDE, we show that drifted VESDE is a general formula (including VP and VESDE) and can provide polynomial results for each setting. Furthermore, the drifted VESDE can go beyond the current VESDE, balances the reverse beginning and discretization error terms, and achieves better results for VE-based models. 

We have provided rebuttals for each reviewer's concern (including the above discussion and other concerns such as the general $W\_p$ guarantee and the technique novelty of the reverse PFODE). We are more than happy to discuss our work in detail and answer any further questions in the rebuttal phase.

Thank you very much for your helpful review.
Thanks again for your insightful suggestions and comments! According to your helpful comments, we improve our work from the empirical and theoretical perspectives. From the empirical perspective, we do experiments on the real-world CelebA 256 dataset and show that our drifted VESDE is a plug-and-play method without training. More specifically, the images generated by our drifted VESDE are more detailed than those of the pure VESDE baseline (shown in the PDF of the global rebuttal). From the theoretical perspective, we show that our drifted VESDE covers common diffusion models (including VP and VESDE) and goes beyond. More details and discussions are shown in the rebuttal part. We will add the above discussion to our next version and are more than happy to answer any further questions.