Title: Advancing Wasserstein Convergence Analysis of Score-Based Models: Insights from Discretization and Second-Order Acceleration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion model

Wasserstein convergence

Convergence Analysis
Abstract: Score-based diffusion models have emerged as powerful tools in generative modeling, yet their theoretical foundations remain underexplored. In this work, we focus on the Wasserstein convergence analysis of score-based diffusion models. Specifically, we investigate the impact of various discretization schemes, including Euler discretization, exponential integrators, and midpoint randomization methods. Our analysis provides the first quantitative comparison of these discrete approximations, emphasizing their influence on convergence behavior. Furthermore, we explore scenarios where Hessian information is available and propose an accelerated sampler based on the local linearization method. We establish the first Wasserstein convergence analysis for such a Hessian-based method, showing that it achieves an improved convergence rate of order $\widetilde{\mathcal{O}}\left(\frac{\sqrt{d}}{\varepsilon}\right)$, which significantly outperforms the standard rate $\widetilde{\mathcal{O}}\left(\frac{d}{\varepsilon^2}\right)$ of vanilla diffusion models. Numerical experiments on synthetic data and the MNIST dataset validate our theoretical insights.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the Wasserstein convergence of four different schemes for score-based diffusion models: Euler-Maruyama, exponential integrator, vanilla midpoint randomization, and exponential integrator with midpoint randomization. The paper observes that those two randomized midpoint methods do not give an improved convergence result because of the variance. The paper also proposed an accelerated sampler based on the Local Linearization Method. An approximation to the drift is designed according to Ito's formula, and Hessian estimation of the log density is employed. This leads to an improved dependence on the discretization $h$, so that only $O(\\frac{1}{\\varepsilon}\\log(\\frac{1}{\\varepsilon}))$ is required to be $\\varepsilon$ close to the target distribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper uses a clear coupling framework to study the convergence of four different practical schemes in a Wasserstein distance. \n2. An improved scheme using Hessian information is proposed, and the experiment on MNIST shows the effectiveness of the new scheme.\n\nWeaknesses:\n1. The analysis is kind of standard: using coupling technique and analyzing the contraction rate and the one-step discretization error. It is kind of incremental, but such a paper to illustrate and compare different discretization schemes could be helpful for people to refer to.\n2. I am not sure how practical the newly proposed scheme is. For example, there is no experiment on CIFAR-10. I understand that this is a theory-focused paper, but I just want to know whether there is some implementation difficulty on CIFAR-10.\n\nQUESTIONS:\nQuestions:\n- The theorems involve $m_{\\min} - 1/2$, which is supposed to be positive number. I understand that $m_{\\min} > 1/2$ is a simple condition for one-step contraction in the coupling argument. But I want to ask what if $m_{\\min} > 1/2$ is not satisfied. Do you need to consider other reverse SDEs by changing the coefficient in front of the score in eq.(3) so that the drift in (3) will lead to contraction during coupling? Currently there is no remark on the fact that $m_{\\min} > 1/2$ and I think maybe comments on this in the paper will be helpful to readers.\n- Currently the complexity result (eg. Corollary 2 and Corollary 5) only shows the dependence on $\\varepsilon$. Maybe the dependence on the dimension $d$ is also important especially for high dimensions. Ideally dependence on the condition number $L/m$ (or other counterparts such as $L(t)/m(t)$) is also interesting to people.\n- For $\\lVert X_0\\rVert_{\\mathbb{L}_2}$ in the bounds, if we assume $\\log p_0$ is minimized at $x=0$, then it has the upper bound $\\sqrt{d/m_0}$ (see for example, Lemma 4.0.1 in Sinho Chewi's book draft https://chewisinho.github.io/main.pdf). I want to ask why the authors left the term in the theorem rather than gave it a bound. I am also not sure whether such considerations could simplify/improve some bounds in the proof. (will the minimizer stays at $x=0$ for $p_t(x)$?)\n- How tight are the bounds? For example, the authors can illustrate this when the target is a Gaussian.\n- In Appendix G.2, implementation details are illustrated for the proposed scheme. How do you choose $k$ for the truncation of matrix exponentials?\n\n\nTypos:\n- page 8 footnote 2: the derivation is in Appendix E rather than F. Also, at the end of Appendix E, the matrix exponentials are addressed in Appendix G rather than F.\n- Appendix D.1 is the proof of Lemma 8 rather than Lemma 7.\n- References [13] and [14] are the same.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work first studies the convergence of various discretization methods for the diffusion backward process, measured by the Wasserstein-2 distance. This work also derives a new discretization method based on the higher-order derivatives.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The theoretical results are presented in good logic, and the new discretization method is also derived with sound reasoning.\n2. The assumptions are clearly stated and moderate.\n\n## Weaknesses\n\n1. As a whole, this work looks like a simple combination of two parts: one about the analysis of Wasserstein-2 distance, and the derivation of the new discretization. These two parts seem to be only loosely connected.\n2. For the first part, the work seems to lack **detailed** comparison against previous works.\n- 2a. Comparison against previous works on Wasserstein-2 distance: In the last paragraph of Section 3.1, results in [14] are mentioned. How do the results in this work compare against those in [14]?\n- 2b. The comparison against results based on KL and TV are mentioned but not explained in detail. It is unclear whether the results about Wasserstein-2 distance imply different insights.\n- 2c. The authors could also discuss the results in Li et al. (2024).\n\nLi, Runjia, Di, Qiwei, and Gu, Quanquan. Unified Convergence Analysis for Score-Based Diffusion Models with Deterministic Samplers. 2024\n \n3. For the second part, the experiments do not seem to be adequate.\n- 3a. The experiments are only conducted on MNIST, which is a relatively easy task. I would suggest that the authors include more challenging tasks so that the possible instability in the estimation of higher-order derivatives can be evaluated.\n- 3b. I would suggest that the authors include baselines based on ODE sampling, especially the Runge-Kutta discretization.\n- 3c. A possible drawback of the algorithm is that the estimation of high-order derivatives may cause higher computational complexity. However, the results does\n\nQUESTIONS:\n1. Continuing Weakness 2b, What is special about the Wasserstein-2 distance?\n2. In all theoretical results, a lot of coefficients are related to $m\\_{\\min}-0.5$. What is special about the $-0.5$ term? Does it pose an implicit restriction on $m_{\\min}$ that $m_{\\min}>0.5$?\n\nLIMITATIONS:\nAs is discussed in \"Weaknesses\", the experiments do not seem to be adequate. The paper does not seem to have any negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical convergence analysis of diffusion models in the Wasserstein-2 distance.\nSpecifically, the authors first conduct a comprehensive comparison of different discretization schemes, including EM method, exponential integrator (EI) and randomized midpoint methods. \nThese schemes are shown to have a similar convergence rate of $\\mathcal{O}(\\frac{1}{\\epsilon^2} \\log (\\frac{1}{\\epsilon}))$ in $W_2$ distance.\nIn addition to these first first-order discretization results, the authors introduce an accelerated sampler that leverages the Hessian estimation. \nTheoretically, they prove that this second-order discretization scheme achieves a significantly improved convergence rate of $\\mathcal{O}((\\frac{1}{\\epsilon} \\log (\\frac{1}{\\epsilon}))$.\n\nIn the experiments, the proposed second-order accelerated method is compared with the above first-order discretization schemes on both a synthetic posterior of a penalized logistic regression and the MNIST data distribution.\nThe numerical results in Figure 1 confirm that the proposed second-order method converges faster, which aligns with the theoretical analysis.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper provides a systematic comparison of multiple popular discretization schemes (EM, EI, REM, REI) in terms of $W_2$ distance.\n- The use of Hessian-vector products in the real-data experiment like MNIST makes the second-order approach computationally feasible.\n\n### Weaknesses\nThe theoretical results are comprehensive, but further analysis could help clarify their properties and underlying assumptions.\n- In contrast to prior work [1], which relies on the relatively mild assumption of a bounded second moment for total variation (TV) divergence analysis, Assumption 1 imposes a stronger condition—strongly log-concavity. This assumption implies the Log-Sobolev inequality, an assumption that is generally not adopted in most recent theoretical results of diffusion models. \n- For the data generation task, the paper would benefit from the inclusion of quantitative metrics. \nTo better align the experiments with the theoretical results, the authors could report the FID or $W_2$ distance on toy 2D tasks, adopting an approach similar to that shown in Table 1 and Table 2 of paper [2]. \n\n[1] Fan, Ying, and Kangwook Lee. \"Optimizing ddpm sampling with shortcut fine-tuning.\" ICML 2023.\n\n[2] Gupta, Shivam, Linda Cai, and Sitan Chen. \"Faster Diffusion Sampling with Randomized Midpoints: Sequential and Parallel.\" ICLR 2025.\n\nQUESTIONS:\n- My major question is about the relationship between the convergence rate and the dimension of the target distribution.\n- In the MNIST experiments, the running times for all five methods are reported to be close (within a 2-minute range for a >2 hour run) in Appendix G.1. \nCould the authors explain why the additional HVP computations in the second-order method did not lead to a noticeable increase in running time.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper established a Wasserstein convergence theory for the score-based diffusion model. It provided upper error bounds for several popular discretization methods: Euler-Maruyama, exponential integrator, and randomized midpoint methods. All discretization methods offer the same rate asymptotically and require $\\tilde{O}(\\epsilon^{-2})$ steps to reach $\\epsilon$ accuracy in $W_2$ distance, though midpoint randomization enables parallel computation and reduces computation time. The author also provides an acceleration method when the second-order information is available, and slashes the total sample steps to $\\tilde{O}(\\epsilon^{-1})$. The theoretical contributions were confirmed by the experiments on synthetic data and the MNIST dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper delivers a thorough Wasserstein-2 convergence analysis across multiple discretisation schemes, filling a gap left by prior work that centred on KL/TV metrics and relied almost exclusively on Euler steps.\n\n2. The author proposed a second-order acceleration method to improve the convergence rate. This is the first acceleration that only requires $\\tilde{O}(\\epsilon^{-1})$ sample steps to achieve $\\epsilon$ accuracy in $W_2$ distance.\n\n3. Overall, the paper is well written and easy to follow, and the result seems correct.\n\nWeaknesses:\n\n1. The theoretical guarantees hinge on strong assumptions—especially Assumptions 2 and 6—that may not hold for many non-Gaussian or multimodal data distributions. Recent TV-distance analyses (e.g., Li et al., 2024) achieve convergence without such restrictive conditions.\n\n2. The accelerated sampler needs accurate Hessian–vector products of $\\nabla^2\\log p_t(x)$ at every diffusion step. Computing these in high dimensions (e.g., $64\\times64$ ImageNet) may be impossible. Alternative accelerations that approximate Hessians via finite differences of score functions (again Li et al., 2024) to avoid this bottleneck. As a result, I think the assumptions for this paper can be further weakened if this strategy is applied (though Li et al., 2024, focus on the convergence of TV distance).\n\n3. Although the authors mention existing works focus on analyzing the convergence of $W_2$ distance, no comparison was given. It would be better to include a brief comparison of the setting/convergence rate between this paper and previous work.\n\nTypos:\n\n1. References [13] and [14] are the same.\n2. Page 16, Line 493, missing one right bracket \")\" in the first line.\n\nReferences:\n1. Li et al., 2024, \"Accelerating Convergence of Score-Based Diffusion Models, Provably\".\n\nQUESTIONS:\n1. As you mentioned, some other works have also focused on the convergence of diffusion in $W_2$ distance. Can you make a brief comparison of your result and theirs?\n\n2. As mentioned in Weakness 1, is there a way to drop the Lipshitz continuity of the score function when analyzing convergence in $W_2$ distance?\n\n3. Can you comment on the dimension dependence of the convergence result? Is it $\\widetilde{O}(d/\\epsilon^2)$ for non-accelerated version and $\\widetilde{O}(d/\\epsilon)$ for accelerated version?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze the effects of the Euler discretization, exponential integrator, midpoint randomization, and a novel exponential integrator with midpoint randomization scheme in the convergence of score-based diffusion models within the Wasserstein distance. They show that midpoint randomization introduces a variance term that cancels out the improved estimation, offering no benefit in convergence. The authors also propose an accelerated sampler that uses second-order information, achieving an improved convergence rate of $O(1/\\epsilon)$ iterations. A numerical study is offered, showing that the accelerated sampler allows for smaller error than the four previous discretizations (among which the randomized midpoint and Euler schemes perform the best).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n**Analysis of various discretization schemes.** \nThe authors study four different discretization schemes, and qualify the differences in performance therein from a theoretical standpoint. This analysis is novel, and welcome to the community, representing a step forward in understanding the behavior of discretization schemes within diffusion model sampling.  \nA nice numerical experiment (for a theory paper) is provided, allowing for an empirical comparison of these discretization schemes in practice that complements the theory provided. \n\n**Well-written.** The paper is very well-written. I commend the authors with respect to the clear amount of effort taken to present the results in a clear and coherent manner. Intuition with respect to the results and methodology is readily provided, and overall enhances the quality of the paper. \n \n## Weaknesses\n**Acceleration requires second-order acceleration.** Previous work suggests that this may not be necessary — Li et al. (2024) achieve a speedup for a deterministic sample in https://arxiv.org/pdf/2403.03852, though their guarantees are in the TV distance.\nIn fact, the guarantee provided by the “accelerated” stochastic sampler of that paper mirrors what I suspect might be the case here. The rate for the “accelerated” stochastic sampler within Li et al. (2024) was shown to be achieved by the vanilla DDPM sampler by another paper written by the same first author (Li et al. (2025)) a year later. It may be possible that the first-order sampler may be able to achieve the $O(1/\\epsilon)$ rate, though as Gao et al. (2025) remark, this may require additional assumptions.\nThe authors may have missed this remark, as seen in their remarks within Appendix A. The authors state that the “accelerated SDE sampler” of Li et al. (2024) “leverages higher-order expansions of the conditional density to enhance efficiency”. Yet, the rate achieved by the stochastic sampler of Li et al. (2024) is worse than the DDPM rate achieved by Li et al. (2025).\nThe authors do not quantify the dimensional dependence within the sample complexity explicitly. This hurts readability and clarity.\n \n## References\nLi et al. (2024), Accelerating Convergence of Score-Based Diffusion Models, Provably\n\nLi et al. (2025), O(d/T) Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions\n\nGao et al. (2025), Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models\n\nQUESTIONS:\nCan this requirement of exact second-order information be relaxed for a $O(1/\\epsilon)$ rate, as is the case for convergence results under the TV distance as in Li et al. (2024) and Li et al. (2025)?\n \nThis is a good paper. I am on the fence between a score of 4 and a score of 5. My concerns regarding whether the accelerated sampler is truly accelerated or not lead me to lean towards a score of 4, as this heavily affects the significancy of the paper. I will be happy to raise my score if the authors alleviate my concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-15"}