Title: Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete diffusion models

convergence analysis

tau-leaping

Euler method

Tweedie tau-leaping
Abstract: Discrete diffusion models have recently gained significant prominence in applications involving natural language and graph data. A key factor influencing their effectiveness is the efficiency of discretized samplers. Among these, $\tau$-leaping samplers have become particularly popular due to their theoretical and empirical success. However, existing theoretical analyses of $\tau$-leaping often rely on somewhat restrictive and difficult-to-verify regularity assumptions, and their convergence bounds contain quadratic dependence on the vocabulary size. In this work, we introduce a new analytical approach for discrete diffusion models that removes the need for such assumptions. For the standard $\tau$-leaping method, we establish convergence guarantees in KL divergence that scale linearly with vocabulary size, improving upon prior results with quadratic dependence. Our approach is also more broadly applicable: it provides the first convergence guarantees for other widely used samplers, including the Euler method and Tweedie $\tau$-leaping. Central to our approach is a novel technique based on differential inequalities, offering a more flexible alternative to the traditional Girsanov change-of-measure methods. This technique may also be of independent interest for the analysis of other stochastic processes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new analytical approach to analyze samplers for the backward process in discrete-state continuous-time diffusion models. With this new approach, the authors report convergence guarantees for the tau-leaping sampler in KL divergence that scale linearly with the state space size per dimension, in contrast to previous results that showed a quadratic dependence. \nAdditionally their method can be applied to the euler and tweedie tau leaping sampler, which are defined on a discrete sampling grid, \nproviding first convergence guarantees.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is an important theoretical contribution to the field of discrete diffusion models in continuous-time, providing valuable convergence guarantees that are valuable for the community.\n\nThe paper is well written, providing a good overview over the deterministic-step-size samplers used within the discrete diffusion models and the prior theoretical work on the convergence of the tau-leaping sampler.\n\nThe main strength of the paper is the proposed novel analysis based on the rate-of-change of the KL between posterior and sampling distribution as an alternative to the analysis based on the Girsanov change-of-measure framework. This brings two benefits: First, it allows application to samplers such as the Euler and tweedie tau-leaping for which the posterior rates are not defined on the whole continuous-time path. Second, the new method does not require regularity conditions.\n\nAnother strength of the paper are the convergence results, which either improve prior results or provide new guarantees.\n\nOne weakness of the paper is a missing discussion how this new results can be interpreted for practitioners, when choosing the sampler for their diffusion model.\n\nTypo\nLine 76 practivally\n\nQUESTIONS:\nCan these results point researchers in the right direction when choosing a sampler for their diffusion model?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThanks to suthors for submitting to NeuRIPs 2025. Their work addresses theoretical aspects of discrete diffusion models and derives novels framework for convergence bounds. Paper addresses two challenges present in current convergence bounds for $\\tau$-leaping sampler and its variants: 1. hard-to-verify assumptions (l 53-59) and 2. Better scaling - in terms of vocabulary size as well as for $\\tau$-leaping variants with deterministic step size, i.e., Euler and Tweedie, used in practise.\nBy carefully treating the discrete nature of the problem, the Kolmogorov forward eq. is used and upper bounded by bounding the leading term to replace a need of It\\^o calculus and Girsanov change of measure technique. Insuch doing, for specific parameterisation (5) the main result (Theorem 2) proves the linearly scaling bound in vocabulary size for $\\tau$-leaping sampler and its variants, improving existing quadratic bounds for $\\tau$-leaping and introducing novel results for Euler  and Tweedie.\n\nSTRENGTHS AND WEAKNESSES:\nWell motivated and timely topic. Introduction is well versed and provide context to problem addressed by the paper.\n\nThe paper is theoretical in nature and provides a rigorous treatment of the topic, referencing existing results and sources of inspiration (e.g., l 223) while emphasising the non-trivial novelties introduced by authors. So despite quite technical topic, the approach and assumptions needed are relatively straightforward to follow.\n\nContributions are both theoretical (novel theoretical results) as well as practical (new guarantess for popular algorithms) and thus paper is well suited for wider ML community.\n\nOn the improvemnt side:\nW1: Limitations: Presented limitations and wider discussion is very limited and perhaps too slim. For instance, the main results are derived for special CTMC rate matrix (5). To what extend is this limiting? If it is, it should be emphasized. Can it be extended? The same goes for Assumtions 1 and 2. One of the claims of the paper is it alleviates difficulties with hard-to-check assumptions. It would be worth discussing or ortherwise showing (experiments?) the new assumtions are easier to verify/meet.\n\nW2: Experiments/Ablation study. Despite its theoretical nature supporting experiment(s) could increase accessability and appeal fo the paper. For instance, Main result (Th 2) is an asymptotic result (in T) in its nature. Could authors show in the experiment (or otherwise) for what ranges of T asymptotic behaviour prevails? Is it data/task dependent?\n\nMinor:\n- numbering of equations is missing as of Section 3, included.\n- (l.140) define $\\mathbb{1}{y=x}$ and other symbiolics used\n\nQUESTIONS:\nQ1: Wider discussion or experiments concerning limitations/ablation study/asymptotics along the lines of Weakness W1 is suggested.\n\nQ2: See weakness W2.\n\nQ3: Could authors elaborate why $g_t$ is a Bregman divergence, whose non-negativity was used in proof (Appendix F)? Bregman div. is defined by $\\phi(y) - \\phi(x) - \\langle y-x, \\nabla\\phi(y)\\rangle$, where $\\phi$ is strictly convex function (here negative entropy), if I am not mistaken. Could authors rewrite presented relation to this form (or show it otherwise please?)\n\nLIMITATIONS:\nSee W1", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper presents a novel theoretical tool for convergence analysis of discrete diffusion models. 
The approach is based on differential-inequality, which circumvents the need for the regularity assumptions required by Girsanov theorem.

Theoretically, this new framework provides an improved convergence guarantee for $\tau$-leaping sampler.
The linear dependence on the vocabulary size $S$ improves upon the previously
results $\mathcal{O}(S^2)$ complexity.
Moreover, the authors provides the convergence order for the Euler method and Tweedie $\tau$-leaping samplers.
These two sampling schemes are shown to have the same convergence rate in terms of the number of steps, but with a lower per-step sampling complexity.

STRENGTHS AND WEAKNESSES:
### Strength
- The paper's contribution is clear: it provides an improved convergence bound for the $\tau$-leaping sampler using a novel analytical technique.  Notably, this approach also weakens standard regularity assumptions in Girsanov theorem without introducing any new ones.
- The paper is well-written and provides a dedicated section for its key notations (Section 2.4).


### Weaknesses
The theoretical analysis is strong. One question for the authors is whether they considered numerical validation for the convergence rate with respect to vocabulary size $S$.

QUESTIONS:
Please see the Weaknesses section.

LIMITATIONS:
The main text lacks a discussion of the limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel analytical framework for discrete diffusion models based on differential inequalities. This new technique circumvents the need for restrictive regularity assumptions previously required by methods relying on the Girsanov change-of-measure framework. The primary contributions are: (1) The development of a more flexible analysis tool for discrete diffusion models. (2) An improved convergence guarantee for the popular $\\tau$-leaping sampler, as well as the first-ever convergence guarantees for the empirically efficient Euler method and Tweedie $\\tau$-leaping samplers.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The novel differential inequality analysis removes restrictive Girsanov-based regularity assumptions, a significant theoretical advance.\n- The improved linear dependency on vocabulary size S for $\\tau$-leaping is a key result with practical importance.\n- This paper provides the first-ever convergence guarantees for the efficient Euler and Tweedie $\\tau$-leaping samplers.\n- The paper is clear and well-organized, and the included comparison table is very helpful for context.\n\n### Weaknesses\n- The work is entirely theoretical and lacks any empirical validation to support its findings.\n- The convergence rates maintain a quadratic dependence on data dimension $d$, which remains a practical limitation.\n\nQUESTIONS:\n- Your results in Theorems 2 and 3 show that the Euler and Tweedie $\\tau$-leaping samplers have the same asympotic convergence rate as $\\tau$-leaping, but with a significantly lower per-step sampling complexity. Does it suggest that there are larger hidden constants in the convergence bounds for these samplers that may impact their practical performance?\n- Can your analysis extend to the high-order samplers as in [1]?\n\n[1] Ren, Yinuo, et al. \"Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms.\" arXiv preprint arXiv:2502.00234 (2025).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}

comments: SUMMARY:
This paper extends the theoretical guarantee if discrete diffusion samplers with a novel framework. Specifically, the quadratic dependency on vocabulary size in the iteration guarantee is improved to linear and new convergence on two practically efficient samplers are further provided.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is well-structured and well-written with thorough proof. Assumptions are properly discussed. Detailed discussion and a comprehensive comparison with related works are included, which is very helpful.
- The theoretical contribution is clear and strong. The convergence guarantee is proved without regularity conditions. The iteration guarantee of $\tau$-leaping is improved and new convergence results on practical samplers are provided.

I am satisfied with this paper and I don't find any significant issues. I am familiar with diffusion models but not particularly familiar with this specific field. I will continue to refer to the opinions of other reviewers for a more comprehensive evaluation.

QUESTIONS:
I don't have further questions.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-09