Title: Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete diffusion model

fast solver

high-order scheme

Runge-Kutta method
Abstract: Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $\tau$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $\tau$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $\theta$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver

Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms

Abstract

Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as τ -leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, τ -leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the θ -Trapezoidal method in KL divergence. Empirical evaluations on GSM8Klevel math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver .

1 Introduction

Diffusion and flow-based models on discrete spaces [1-10] have emerged as a cornerstone of modern generative modeling for categorical data, offering unique advantages in domains where continuity assumptions fail. Unlike their continuous counterparts, discrete diffusion models inherently accommodate data with discrete structures, e.g. , language tokens, molecular sequences, tokenized images, and graphs, enabling principled generation and inference in combinatorially complex spaces. These models have exerted a large impact on numerous applications, from the design of molecules [11], proteins [12], and DNA sequences [13, 14] under biophysical constraints, to the generation of high-fidelity text [15] and images [16] via autoregressive or masked transitions, etc. . Beyond standalone tasks, discrete diffusion models also synergize with methodologies, ranging from tensor networks [17] to guidance mechanisms [18-20].

Discrete diffusion models, despite their broad applicability, face a critical bottleneck: inference inefficiency . Current inference methods include: (1) exact simulation methods [21], which ensure unbiased sampling from the pre-trained model but suffer from unpredictable inference time and redundant score evaluations, resulting in poor scaling w.r.t. dimensionality; and (2) approximate

methods such as τ -leaping [22], which offer simple and parallelizable implementation but, due to their first-order accuracy, requires small step sizes to control discretization error, forcing a stringent trade-off between speed and sample quality.

To address these limitations in possibly computationally constrained environments, we develop highorder numerical schemes tailored for discrete diffusion model inference. Drawing inspirations from acceleration techniques developed for ordinary differential equations (ODEs) [23], stochastic differential equations (SDEs) [24, 25], chemical reaction simulations [26], and most recently continuous diffusion [27-29], our work represents the first successful adaptation of high-order numerical schemes to the discrete diffusion domain . Through careful design, these high-order schemes provide unprecedented efficient and versatile solutions for discrete diffusion model inference.

Our Contributions. The main contributions of this paper are summarized as follows:

Weintroduce the first high-order numerical solvers for discrete diffusion model inference, namely the θ -Runge-Kutta-2 ( θ -RK-2) method and the θ -Trapezoidal method;

We rigorously establish the theoretical properties of both methods, proving second-order convergence of the θ -Trapezoidal method and conditional second-order convergence of the θ -RK-2 method;

We empirically validate our theoretical results and demonstrate the superior performance of the θ -Trapezoidal method through comprehensive evaluations on large-scale text and image generation benchmarks.

1.1 Related Works

Here we briefly review related works and defer a more detailed discussion to App. A.

Discrete Diffusion Models. Since their introduction, discrete diffusion models have undergone significant refinements, including the development of score-entropy loss [30] and flow-matching formulation [31, 32]. These models generally fall into two categories based on their noise distribution: uniform [30, 20] and masked (absorbing state) [33-35, 21], each offering unique advantages in modeling discrete distributions. Recent theoretical advances have emerged through studies [36-38].

High-Order Scheme for Continuous Diffusion Models. The development of high-order numerical schemes for solving ODEs and SDEs represents decades of research, as comprehensively reviewed in [23, 39, 40]. These schemes have recently been adapted to accelerate continuous diffusion model inference, encompassing approaches such as the exponential integrators [41-43], Adams-Bashforth methods [29, 44, 45], Taylor methods [27, 46] and (stochastic) Runge-Kutta methods [47, 28, 48-51].

High-Order Scheme for Chemical Reaction Systems. Regarding approximate methods for simulating compound Poisson processes and chemical reaction systems with state-dependent intensities, efforts have been made on the τ -leaping method [52], and its extensions [53, 54, 26, 55, 56]. For a quick review of the problem setting and these methods, one may refer to [57, 58]. The adaptation of these methods to discrete diffusion models presents unique challenges due to the presence of both time and state-inhomogeneous intensities in the underlying Poisson processes.

2 Preliminaries

In this subsection, we review several basic concepts and previous error analysis results of discrete diffusion models.

2.1 Discrete Diffusion Models

In discrete diffusion models, one considers a continuous-time Markov chain (CTMC) ( x t ) 0 ≤ t ≤ T on a finite space X as the forward process . We represent the distribution of x t by a vector p t ∈ ∆ | X | , where ∆ | X | denotes the probability simplex in R | X | . Given a target distribution p 0 , the CTMC

satisfies the following equation:

is the rate matrix at time t satisfying

̸

̸

Below we will use the notation Q 0 t = Q t -diag Q t . It can be shown that the corresponding backward process is of the same form but with a different rate matrix [59]:

̸

̸

is the rate matrix and ⃗ ∗ s denotes ∗ T -s . The rate matrix Q t is often chosen to possess certain sparse structures such that the forward process converges to a simple distribution that is easy to sample from. Popular choices include the uniform and absorbing state cases [30], where the forward process (2.1) converges to the uniform distribution on X and a Dirac distribution, respectively.

Common training practice is to define the score function (or the score vector) as s t ( x ) = ( s t ( x, y )) y ∈ X := p t p t ( x ) for any x ∈ X , t ∈ [0 , T ] and estimate it by a neural network ̂ s ϕ t ( x ) , where the parameters ϕ are trained by minimizing the score entropy [30, 60] for some weights ψ t ≥ 0 as:

̸

Similar to the continuous case, the backward process is approximated by another CTMC d q s d s = ̂ Q ϕ s q s , with q 0 = p ∞ and rate matrix ̂ Q ϕ s , where ̂ Q ϕ s ( y, x ) = ⃗ ̂ s ϕ s ( x, y ) ⃗ Q s ( x, y ) for any x = y ∈ X . The inference is done by first sampling from p ∞ and then evolving the CTMC accordingly. For simplicity, we drop the superscript ϕ hereafter.

2.2 Stochastic Integral Formulation of Discrete Diffusion Models

Discrete diffusion models can also be formulated as stochastic integrals, which is especially useful for their theoretical analysis [38]. In this section, we briefly recapitulate the relevant results and refer to App. B for the mathematical details. Below, we work on the probability space (Ω , B , P ) and denote the pairwise difference set of the state space X by D := { x -y : x = y ∈ X } . In this work, we focus on the case where X = [ S ] d with d data dimensions and S sites along each dimension.

̸

We first introduce the Poisson random measure, a key concept in the formulation.

Definition 2.1 (Informal Definition of Poisson Random Measure) . The random measure N [ λ ](d t, d ν ) on R + × D is called a Poisson random measure with evolving intensity λ w.r.t. a measure γ on D if, roughly speaking, the number of jumps of magnitude ν during the infinitesimal time interval ( t, t +d t ] is Poisson distributed with mean λ t ( ν ) γ (d ν )d t .

The forward process (2.1) can thus be represented by the following stochastic integral:

where the intensity λ is defined as λ t ( ν, ω ) = Q 0 t ( x t -( ω ) + ν, x t -( ω )) if x t -( ω ) + ν ∈ X and 0 otherwise. Here, the outcome ω ∈ Ω and x t -denotes the left limit of the càdlàg process x t at time t with x 0 -= x 0 . We will also omit the variable ω , should it be clear from context. The backward process in discrete diffusion models (2.2) can also be represented similarly as:

where the intensity µ is defined as µ s ( ν, ω ) = ⃗ s s ( y s -, y s -+ ν ) ⃗ Q 0 s ( y s -, y s -+ ν ) if y s -+ ν ∈ X and 0 otherwise. During inference, ̂ y s = ̂ y 0 + ∫ s 0 ∫ D νN [ ̂ µ ](d s, d ν ) is used instead of (2.4), where the estimated intensity ̂ µ is defined by replacing the true score s t with the neural network estimated score ̂ s t in µ s ( ν, ω ) . . In the following, we also denote the intensity µ s ( ν, ω ) at time s by µ s ( ν, y s -) with slight abuse of terminology to emphasize its dependency on ω through y s -( ω ) .

̸

+1

0.8

0.9

1.0

100

500

1000

2000

3000

Perplexity

Perplexity

NFEglyph[suppress]intensity

10

30

50

100

NFEglyph[suppress]intensity

θ

∆

n

(1

-

θ

)∆

n

y

s

n

y

∗

ρ

n

y

s

n

Inference Process

τ

-Leaping

̂

µ

s

n

θ

-RK-2

(i)

̂

µ

s

n

(ii)

(1

-

1

2

θ

)

̂

µ

s

n

+

1

2

θ

̂

µ

∗

ρ

n

θ

-Trapezoidal

(i)

̂

µ

s

n

(ii)

α

1

̂

µ

∗

ρ

n

-

α

2

̂

µ

s

n

3 Numerical Schemes for Discrete Diffusion Model Inference

Before introducing the proposed numerical schemes, we first review existing numerical schemes for discrete diffusion models, including exact simulation methods and the τ -leaping method, and discuss their merits and limitations.

3.1 Exact Simulation Methods

Unlike in continuous diffusion models, where exact simulation is infeasible, discrete diffusion models permit inference without discretization error. Notable examples of unbiased samplers include uniformization [36] for the uniform state case and the First-Hitting Sampler (FHS) [21] for the absorbing state case. The main idea behind these methods is to first sample the next jump time and then the jump itself. Theoretical analysis [38] reveals that such schemes lack guarantees with finite computation budget , since the number of required jumps (and thus the inference time) follows a random distribution with expectation Ω( d ) . This computational restriction may be less favorable for high-dimensional applications, such as generative modeling of DNA or protein sequences.

Furthermore, the absence of discretization error does not necessarily translate to superior sample quality , given the inherent estimation errors in neural network-based score functions. This limitation is further amplified by the highly skewed distribution of jumps, with a concentration occurring during the terminal phase of the backward process, when the neural network-based score function exhibits the highest estimation error. This phenomenon stems from the potential singularity of the target distribution p 0 , which induces singularities in the score function, making accurate neural network estimation particularly challenging during that phase ( cf. Assump. 4.4 [38]).

The left figure in Fig. 1 illustrates an application of the uniformization algorithm to discrete diffusion inference for text generation, with detailed experimental parameters presented in Sec. 6.3 and App. D.3. As the process approaches the target distribution ( t → T ), the number of jumps (in terms of the number of score function evaluations, NFE) grows unbounded, while perplexity improvements become negligible. This skew in computational effort leads to redundant function evaluations . Although early stopping is commonly adopted at T -δ for some small δ ≪ 1 to alleviate this inefficiency, this approach introduces challenges in its selection, particularly under computational constraints or when efficiency-accuracy trade-offs are desired. Moreover, the variable jump schedules across batch samples complicate parallelization efforts in exact methods, highlighting the need for more adaptable and efficient algorithmic solutions.

3.2 Approximate Method: τ -Leaping Method

The τ -leaping method [52, 22] is a widely adopted scheme that effectively addresses both dimensionality scaling and inference-time control challenges. This Euler-type scheme approximates the backward process with time-dependent intensity µ t via the following updates:

where ∆ denotes the time step and P ( · ) denotes a Poisson random variable. In general, one may design different discretization schemes for τ -leaping, and the summation in (3.1) is parallelizable, underscoring the method's flexibility and efficiency. We refer to Alg. 3 and App. B.2 for a detailed description of the τ -leaping method for discrete diffusion model inference. Regarding convergence properties as the time discretization becomes increasingly refined, theoretical analyses by [22, 38] have established the error bounds of the τ -leaping method, the results of which are summarized in the following theorem. Further discussion can be found in App. B.2.

Theorem 3.1 (Thm. 4.7 in [38]) . Under a certain discretization scheme and technical assumptions, and given an ϵ -accurate score function, the following error bound holds:

D KL ( p δ ∥ ̂ q T -δ ) ≲ exp( -T ) + ϵ + κT, (3.2) where δ ≪ 1 is the early stopping time, κ controls the step size, and T is the time horizon. The notation ≲ indicates the inequality holds up to a constant factor as κ → 0 .

The error bound (3.2) decouples three error sources of the τ -leaping scheme: the truncation error O ( e -T ) , the score estimation error ϵ , and the discretization error O ( κT ) . Similar to the case for the Euler method for ODEs and the Euler-Maruyama scheme for SDEs, the τ -leaping method is a first-order scheme in terms of the discretization error O ( κT ) .

4 Algorithms: High-Order Inference Schemes

A natural improvement of τ -leaping is to develop high-order schemes for discrete diffusion models. As a foundational example, consider the second-order Runge-Kutta (RK-2) method with two stages [23] for solving the ODE d x t = f t ( x t )d t . This method represents one of the simplest highorder numerical schemes:

̂ ̂ ̂ ̂ ̂ ̂ ̂ This scheme reduces to the exact midpoint method when θ = 1 2 and Heun's method when θ = 1 . The underlying intuition stems from the observation that for f ∈ C 2 ( R ) , [( 1 -1 2 θ ) f (0) + 1 2 θ f ( θ ∆) ] ∆

offers a second-order approximation of ∫ ∆ 0 f ( x )d x in contrast to f (0)∆ , which is only first-order. This approach has been successfully adapted for SDE simulation [24] and continuous diffusion model inference [48, 28, 29, 49, 51]. Notably, these methods enhance sample quality and computational efficiency without requiring additional model training, making the development of high-order schemes for discrete diffusion inference both theoretically appealing and practically viable.

In this section, we propose two high-order solvers for inference in the discrete diffusion model. We will primarily focus on two-stage algorithms aiming for second-order accuracy. Specifically, we will introduce the θ -RK-2 and θ -Trapezoidal methods. Throughout this section, we assume a time discretization scheme ( s i ) i ∈ [0: N ] with 0 = s 0 < · · · < s N = T -δ , where δ is the early stopping time and use the shorthand notations ∗ + = max { 0 , ∗} . For any s ∈ ( s n , s n +1 ] and n ∈ [0 : N -1] , we define ⌊ s ⌋ = s n , ρ s = (1 -θ ) s n + θs n +1 , ∆ n = s n +1 -s n , and θ -section points as ρ n = (1 -θ ) s n + θs n +1 . We choose γ (d ν ) to be the counting measure on D .

4.1 θ -RK-2 Method

We first present the θ -RK-2 method , which is simple in design and serves as a natural analog of the second-order RK method for ODEs (4.1) in terms of time and state-dependent Poisson random measures, as a warm-up for the θ -Trapezoidal method. We note that similar methods have been proposed for simulating SDEs driven by

Algorithm 1: θ -RK-2 Method

Brownian motions or Poisson processes, such as the stochastic [24] and the Poisson [54] RK methods. A summary of this method is given in Alg. 1.

Intuitively, the θ -RK-2 method is a two-stage algorithm that:

(i) Firstly, it runs τ -leaping with step size θ ∆ n , obtains an intermediate state ̂ y ∗ ρ n at the θ -section point ρ n , and evaluates the intensity µ ∗ ρ n there;

̂ (ii) Then another step of τ -leaping for a full step ∆ n is run using a weighted sum of the intensities at the current time point s n and the θ -section point ρ n .

We emphasize that our method differs from the midpoint method proposed in [52] for simulating chemical reactions, in which the Poisson random variable in the first step is replaced by its expected value. Such modification is in light of the lack of continuity and orderliness of the state space.

4.2 θ -Trapezoidal Method

As to be shown theoretically and empirically, the conceptually simple θ -RK-2 method may have limitations in terms of both accuracy and efficiency. To this end, we propose the following θ -Trapezoidal method , which is developed based on existing methods proposed for simulating SDEs [25] and chemical reactions [26]. Below, we introduce two parameters that will be used extensively later:

The θ -Trapezoidal method is summarized in Alg. 2. Intuitively, it separates each interval ( s n , s n +1 ] into two sub-intervals ( s n , ρ n ] and ( ρ n , s n +1 ] , on which simulations are detached with different intensities designed in a balanced way.

Algorithm 2: θ -Trapezoidal Method

Input:

0

,

q

θ

(0

y

Output:

1

for

0

A sample

n

2

3

∼

= 0

̂

y

to

N

∗

ρ

y

n

s

n

+1

Compared to the θ -RK-2 method, the θ -Trapezoidal method is also twostage with an identical first step. The

∑

̂

end

4

̂

second step, however, differs in two major aspects:

,

1]

∼

(

q

∈

s

N

y

-

̂

+

n

s

y

ν

∈

y

D

ν

ρ

∗

P

n

+

α

∑

s

n

, ρ

trap

t

do

ν

̂

∈

1

µ

D

∗

ρ

̂

.

N

ν

P

-

(

µ

α

)

n

s

n

2

µ

∈

[0:

(

ν

s

n

N

)

)

-

1]

θ

∆

+

(

,

µ,

̂

n

)

ν

̂

µ

;

)(1

-

̂

(1) The second step starts from the intermediate state ̂ y ∗ ρ n instead of ̂ y s n and only runs for a fractional step (1 -θ )∆ n rather than a full step ∆ n ;

(2) The weighted sum is comprised of an altered pair of coefficients ( α 1 , -α 2 ) , performing an extrapolation instead of interpolation with coefficients (1 -1 2 θ , 1 2 θ ) as in the θ -RK-2 method with θ ∈ [ 1 2 , 1] . This feature will be shown to render the algorithm unconditionally second-order.

Following the common practice in the literature [22], we reject updates with multiple jumps along one dimension in both algorithms, ensuring their well-posedness. A simple analysis shows that rejection only happens with probability O ( κ ) , and we refer to further details in Rmk. C.4. We refer to Props. C.2 and C.3 for the stochastic integral formulations of these two algorithms. We provide a visual comparison between the θ -RK-2 and the θ -Trapezoidal method in the right figure of Fig. 1.

5 Theoretical Analysis

In this section, we provide the theoretical results of the θ -Trapezoidal and θ -RK-2 methods. The goal of this section is to show that under certain conditions, both methods are second-order accurate, improving from the first-order accuracy of the τ -leaping method ( cf. Thm. 3.1). Our theoretical analysis also reveals that the θ -Trapezoidal method is more robust to the choice of θ than θ -RK-2, to be confirmed by our empirical results in Sec. 6.

5.1 Assumptions

For simplicity, we impose a periodic boundary condition on the state space X = [ S ] d , i.e. , embed the state space in the d -dimensional torus T d , to streamline the proofs ( cf. Rmk. C.4).

Assumption 5.1 (Convergence of Forward Process) . The forward process converges to the stationary distribution exponentially fast, i.e. , D KL ( p T ∥ p ∞ ) ≲ exp( -T ) .

This assumption ensures rapid convergence of the forward process, controlling error when terminated at a sufficiently large time horizon T , and is automatically satisfied in the masked state case

n

̂

n

̂

( (

1

̂

,

∗

θ

.

)∆

n

)

;

and the uniform state case, given sufficient connectivity of the graph ( cf. [38]). The exponential rate aligns with continuous diffusion models ( cf. [61]).

Assumption 5.2 (Regularity of Intensity) . For the true intensity µ s ( ν, y s -) and the estimated intensity ̂ µ s ( ν, y s -) , it holds almost everywhere w.r.t. µ s ( ν, y s -) γ (d ν ) ⃗ p s -(d y s -) that: (1) Both intensities belong to C 2 ([0 , T -δ ]) ; (2) Both intensities are upper and lower bounded on [0 , T -δ ] .

This assumes two key requirements of the scores: (1) the forward process maintains sufficient smoothness, which is achievable through appropriate time reparametrization; and (2) if and only if a state y ∈ X is achievable by the forward process and ν is a permissible jump therefrom, then both its true and estimated intensity are bounded, corresponding to Assumps. 4.3(i), 4.4, and 4.5 [38].

Assumption 5.3 (Estimation Error) . For all grid points and θ -section points, the estimation error of the neural network-based score is small, i.e. , for any s ∈ ∪ n ∈ [0: N -1] { s n , ρ n } , we have

This assumption quantifies the proximity of the estimated intensity ̂ µ to the true intensity µ after sufficient training. Compared with [38], the additional L ∞ part in (2) is required for technical reasons, which is similar to [62, 51]. In practice, such additional assumptions may be realized by adding extra penalty terms to the objective function during training.

5.2 Convergence Guarantees

The following theorem summarizes our theoretical guarantees for the θ -Trapezoidal method:

Theorem 5.4 (Second Order Convergence of θ -Trapezoidal Method) . Suppose θ ∈ (0 , 1] and α 1 ̂ µ ∗ ρ s -α 2 ̂ µ ⌊ s ⌋ ≥ 0 for all s ∈ [0 , T -δ ] , then the following error bound holds for Alg. 2 under Assumps. 5.1 to 5.3:

The complete proof is presented in App. C.2. The outline is to first bound D KL ( p δ ∥ ̂ q trap T -δ ) by the KL divergence between the corresponding path measures, as established in Thm. C.5, and then decompose the integral in the log-likelihood and bound respectively, where the primary technique used is Dynkin's formula (Thm. C.10). With a term-by-term comparison with Thm. 3.1, we observe a significant improvement in the discretization error term from O ( κT ) to O ( κ 2 T ) . This confirms that the θ -Trapezoidal method achieves second-order accuracy given a sufficient time horizon T and accurate score estimation, with empirical validation presented in Sec. 6.

D KL ( p δ ∥ ̂ q trap T -δ ) ≲ exp( -T ) + ( ϵ I + ϵ II ) T + κ 2 T, where δ is the early stopping time, κ = max n ∈ [0: N -1] ∆ n , i.e. , the largest stepsize, and ̂ q trap T -δ is the distribution obtained by Alg. 1 as defined in Prop. C.2.

Theorem 5.5 (Conditional Second-Order Convergence of θ -RK-2 Method) . Suppose θ ∈ (0 , 1 2 ] and (1 -1 2 θ ) ̂ µ ⌊ s ⌋ + 1 2 θ ̂ µ ∗ ρ s ≥ 0 for all s ∈ [0 , T -δ ] , then the following error bound holds for Alg. 1 under Assumps. 5.1 to 5.3:

The proof of the theorem above is provided in App. C.3. The restricted range of θ is caused by one specific error term (III . 4) (C.9) that permits bounding with Jensen's inequality only when θ ∈ (0 , 1 2 ] , similar to its counterpart (II . 4) (C.11) in the θ -Trapezoidal method. The limitation arises partially because the weighted sum with coefficients (1 -1 2 θ , 1 2 θ ) becomes an extrapolation only if 1 -1 2 θ < 0 , a feature that naturally holds for all θ ∈ (0 , 1] in the θ -Trapezoidal method. These theoretical findings are consistent with the empirical observations in Fig. 6 of App. D.3, where the performance of θ -RK-2 method clearly peaks when θ ∈ (0 , 1 2 ] .

D KL ( p δ ∥ ̂ q RK T -δ ) ≲ exp( -T ) + ( ϵ I + ϵ II ) T + κ 2 T, where δ is the early stopping time, κ = max n ∈ [0: N -1] ∆ n , i.e. , the largest stepsize, and ̂ q RK T -δ is the distribution obtained by Alg. 2 as defined in Prop. C.3.

Remark 5.6 (Comparison between Trapezoidal and RK-2 Methods) . Trapezoidal methods were originally proposed by [25] as a minimal second-order scheme in the weak sense for simulating SDEs. In simulating chemical reaction contexts, [26] claimed that trapezoidal methods also achieve second-order convergence for covariance error apart from the weak error, a property not shared by midpoint (RK-2) methods. Our empirical results partly reflect these findings, while we defer theoretical investigation of covariance error convergence in discrete diffusion models to future work.

Remark 5.7 (Remark on the Positivity of Extrapolated Intensity) . Due to the nature of extrapolation, both our theorems require an additional assumption on the positivity of the extrapolated intensity, which is classically assumed in [25, 26], and resolving this issue is a long-standing open problem. The best result so far is Prop. 5 [26], claiming clamping the intensity above 0 only causes an error of order O ( κ p ) , for any large integer p . We empirically demonstrate the validity of this assumption Tab. 6 in practice through the text generation task (Sec. 6.2) and find that positivity occurs for both methods with high probability over 95%, approaching 100% with increasing NFE. We refer to further discussion in Rmk. C.6.

6 Experiments

Based on the theoretical analysis, we expect the θ -Trapezoidal method to outperform the τ -leaping method and the θ -RK-2 method in terms of sample quality, given the same number of function evaluations. This section empirically validates the anticipated effectiveness of our proposed θ -Trapezoidal method (Alg. 2) through comprehensive evaluations across text and image generation tasks. Our comparative analysis includes established discrete diffusion samplers as baselines, e.g. , the Euler method [33], τ -leaping [22], Tweedie τ -leaping [30], First-Hitting Sampler (FHS) [21], Parallel Decoding [63], and Semi-Autoregressive (Semi-AR) sampler [64]. We benchmark on both uniform and masked discrete diffusion models, with experiment details provided in App. D.

6.1 15-State Toy Model

We first evaluate the performance of the θ -Trapezoidal method using a 15 -state toy model ( d = 1 , S = 15 ). The target distribution is uniformly generated from ∆ 15 , with rate matrix Q = 1 15 E -I , where E is the all-one and I is the identity matrix. This setup provides analytically available score functions, allowing isolation and quantification of numerical errors introduced by inference algorithms. We apply both the θ -Trapezoidal and the θ -RK-2 method to generate 10 6 samples and estimate the KL divergence between the true ground truth p 0 and the generated distribution q T .

16

32

64

128

256

512

1024

Numberglyph[suppress]ofSteps

10

4

10

3

10

2

10

1

(

p

0- -

T

)

slopeglyph[suppress]=glyph[suppress]-1.543

slopeglyph[suppress]=glyph[suppress]-1.937

slopeglyph[suppress]=glyph[suppress]-1.819

- RK- 2

rape

zo

i dal

RK- 2glyph[suppress]

i tte d

rape

zo

dal glyph[suppress]

i tte d

RK- 2glyph[suppress]

dglyph[suppress]

(l as

tglyph[suppress]

4glyph[suppress]

po

ts

)

-

F

i

i

n

̂ For a fair comparison, we choose θ = 1 2 for both methods, and the results are presented in Fig. 2. While both methods exhibit super-linear convergence as the total number of steps grows, the θ -Trapezoidal method outperforms the θ -RK-2 method in terms of both absolute value and convergence rate, while the θ -RK-2 method takes longer to enter the asymptotic regime. Moreover, the fitted line indicates that the θ -Trapezoidal method approximately converges quadratically with respect to the step count, confirming our theoretical results.

6.2 Text Generation

For the text generation task, we employ the pre-trained score function from RADD [33] as our base model for benchmarking inference algorithms. RADD is a masked discrete diffusion model with GPT-2-level text generation capabilities [65] and is trained on the OpenWebText dataset [66] with d = 1024 and S = 50258 . Our comparative analysis maintains consistent computational resources across methods, quantified through the number of score function evaluations (NFE), and evaluates the sample quality produced by FHS, the Euler method, τ -leaping, Tweedie τ -leaping, Semi-AR,

q

L

K

D

-

-

-

T

T

F

i

F

4

8

16

32

64

Number of Function Evaluations (NFE)

6.00

7.85

10.26

13.42

17.54

22.94

30.00

Frechet Inception Distance (FID)

-leaping

FHS

Euler

Parallel decoding

-Trapezoidal

16

32

64

6.50

7.29

8.17

9.15

10.26

11.50

Zoom-in

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

6.60

7.08

7.57

8.05

8.53

9.02

9.50

FID

-Trapezoidal (NFE = 32)

-Trapezoidal (NFE = 64)

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

25.00

32.50

40.00

47.50

55.00

62.50

70.00

Generative Perplexity

-Trapezoidal (NFE = 128)

-Trapezoidal (NFE = 1024)

and our proposed θ -Trapezoidal method. We generate text sequences of 1024 tokens and measure their generative perplexity (computed with GPT-2 Large [65]) following the evaluation protocol established in [33].

Tab. 1 presents the results for both low ( 128 ) and high ( 1024 ) NFEs, with comprehensive results across additional NFE values in Tab. 3. The empirical results demonstrate that the θ -Trapezoidal method consistently produces better samples within a fixed computation budget than existing popular inference algorithms. Notably, it outperforms Euler and Tweedie τ -leaping, two of the best-performing samplers adopted by RADD, by a large margin. It also consistently prevails over FHS, which performs exact simulation at high NFE ( 1024 ), supporting again our observations that being free of

discretization error does not necessarily imply better sampling quality. These results validate the practical efficiency and accuracy of Alg. 2. Additional evaluation results, including unigram entropy and generative perplexities evaluated under LLaMA 3 [67], are detailed in App. D.2.

6.3 Image Generation

Our experiments on image generation utilize the pre-trained score function from MaskGIT [63, 68] as the base model, which can be converted into a masked discrete diffusion model by introducing a noise schedule (see App. D.3). MaskGIT employs a masked image transformer architecture trained on ImageNet [69] of 256 × 256 resolution, where each image amounts to a sequence of 256 discrete image tokens following VQ-GAN tokenization [70] ( d = 256 , S = 1025 ). We evaluate the θ -Trapezoidal method against FHS, the Euler method, τ -leaping, and parallel decoding under equivalent NFE budgets ranging from 4 to 64. Following the setting in [63], we generate 5 × 10 4 images and compute their Fréchet Inception Distance (FID) against the ImageNet validation split.

Fig. 3 reveals that θ -Trapezoidal method (Alg. 2) consistently achieves lower (and thus better) FID values compared to both the Euler method and τ -leaping across all NFE values. While FHS and parallel decoding show advantages at extremely low NFE ( ≤ 8 ), their performance saturates with increased computational resources, making them less favorable compared to our rapidly converging method. Additional results, including generated image samples (Fig. 7), are detailed in App. D.

Remark 6.1 (Algorithm Hyperparameters) . We evaluate the performance of the θ -Trapezoidal method across various θ and NFE values for both text and image generation tasks. As illustrated in Fig. 4, we observe that the θ -Trapezoidal method demonstrates robustness to θ , with a flat landscape near the optimal choice. Our empirical analysis suggests that θ ∈ [0 . 3 , 0 . 5] consistently yields competitive performance across different tasks.

6.4 Diffusion Large Language Model and Math Reasoning

To verify the effectiveness of the proposed method on a scale, we additionally benchmark its performance on LLaDA-Instruct [64], an 8B masked diffusion LLM with one of the best language modeling performances among discrete diffusionbased LLMs. We examine its perfor-

mance on GSM8K [71], a math-reasoning dataset consisting of grade-school-level problems. We compare θ -Trapezoidal to the semi-autoregressive (Semi-AR) sampler therein, with both confidencebased (Conf.) and purely random (Rand.) remasking strategies. For each method, we generate a response of 256 tokens in a zero-shot prompting manner, with NFE ranging from 64 to 256 , and report the accuracy in Tab. 2. θ -Trapezoidal outperforms the Semi-AR sampler in the low NFE regime, where NFE is strictly smaller than the sequence length. At high NFE regime, θ -Trapezoidal exhibits a similarly competitive performance as other solvers. This observation accords with our claim that high-order samplers perform better with lower NFE budgets, and that these advantages persist even when the model size is large. Further implementation details are available at App. D.4.

7 Conclusion and Future Works

In this work, we introduce the θ -RK-2 and θ -Trapezoidal methods as pioneering high-order numerical schemes tailored for discrete diffusion model inference. Through rigorous analysis based on their stochastic integral formulations, we establish second-order convergence of the θ -Trapezoidal method and that of the θ -RK-2 method under specified conditions. Our analysis indicates that the θ -Trapezoidal method generally provides superior robustness and computational efficiency compared to the θ -RK-2 method. Our empirical evaluations, spanning both a 15-dimensional model with precise score functions and large-scale text and image generation tasks, validate our theoretical findings and demonstrate the superiority performance of our proposed θ -Trapezoidal method over existing samplers in terms of sample quality under equivalent computational constraints. Additionally, we provide a comprehensive analysis of the method's robustness by examining the optimal choice of the parameter θ in our schemes.

Future research directions include comparative analysis of these schemes and development of more sophisticated numerical approaches for discrete diffusion model inference, potentially developing inference methods of higher order [56] or incorporating adaptive step sizes and parallel sampling methodologies. From the perspective of applications, these methods may also show promise for tasks in computational chemistry and biology, particularly in the design of molecules, proteins, and DNA sequences. Moreover, it would also be interesting to explore the usage of higher-order numerical solvers in other problem settings involving the inference of diffusion models, such as sampling via discrete diffusion models [72-75], inference-time scaling of diffusion models [76-91], improving the reasoning ability of diffusion large language models [92, 64, 93-99], etc.

Acknowledgments and Disclosure of Funding

YC is supported by the National Science Foundation under Grants No. ECCS-1942523, DMS2206576, and CMMI-2450378. GMR is supported by a Google Research Scholar Award. YZ and MT are grateful for partial supports by NSF Grants DMS-1847802, DMS-2513699, DOE Grants NA0004261, SC0026274, and Richard Duke Fellowship. YR, HC and LY acknowledge support of the National Science Foundation under Award No. DMS-2208163.

References

Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202 , 2022.

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089 , 2022.

Griffin Floto, Thorsteinn Jonsson, Mihai Nica, Scott Sanner, and Eric Zhengyu Zhu. Diffusion on the probability simplex. arXiv preprint arXiv:2309.02530 , 2023.

Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Lm8T39vLDTE .

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems , 34:12454-12465, 2021.

Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems , 35:34532-34545, 2022.

Pierre H Richemond, Sander Dieleman, and Arnaud Doucet. Categorical sdes with simplex diffusion. arXiv preprint arXiv:2210.14784 , 2022.

Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuoustime discrete diffusion models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=BYWWwSY2G5s .

Javier E Santos, Zachary R Fox, Nicholas Lubbers, and Yen Ting Lin. Blackout diffusion: generative diffusion models in discrete-state spaces. In International Conference on Machine Learning , pages 9034-9059. PMLR, 2023.

Thomas J Kerby and Kevin R Moon. Training-free guidance for discrete diffusion models for molecular generation. arXiv preprint arXiv:2409.07359 , 2024.

Nathan C Frey, Daniel Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien LafranceVanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, et al. Protein discovery with discrete walk-jump sampling. arXiv preprint arXiv:2306.12360 , 2023.

Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning , pages 1276-1301. PMLR, 2023.

Wei Guo, Yuchen Zhu, Molei Tao, and Yongxin Chen. Plug-and-play controllable generation for discrete masked models. arXiv preprint arXiv:2410.02143 , 2024.

Do Huu Dat, Do Duc Anh, Anh Tuan Luu, and Wray Buntine. Discrete diffusion language model for long text summarization. arXiv preprint arXiv:2407.10998 , 2024.

Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11502-11511, 2022.

Luke Causer, Grant M Rotskoff, and Juan P Garrahan. Discrete generative diffusion models without stochastic differential equations: a tensor network approach. arXiv preprint arXiv:2407.11133 , 2024.

Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572 , 2024.

Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Masatoshi Uehara. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08252 , 2024.

Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dallatorre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10193 , 2024.

Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908 , 2024.

Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems , 35:28266-28279, 2022.

John Charles Butcher. The numerical analysis of ordinary differential equations: Runge-Kutta and general linear methods . Wiley-Interscience, 1987.

Kevin Burrage and Pamela Marion Burrage. High strong order explicit runge-kutta methods for stochastic ordinary differential equations. Applied Numerical Mathematics , 22(1-3):81101, 1996.

David F Anderson and Jonathan C Mattingly. A weak trapezoidal method for a class of stochastic differential equations. Communications in Mathematical Sciences , 9(1):301-318, 2011.

Yucheng Hu, Tiejun Li, and Bin Min. A weak second order tau-leaping method for chemical kinetic systems. The Journal of chemical physics , 135(2), 2011.

Hideyuki Tachibana, Mocho Go, Muneyoshi Inahara, Yotaro Katayama, and Yotaro Watanabe. Quasi-taylor samplers for diffusion generative models based on ideal derivatives. arXiv preprint arXiv:2112.13339 , 2021.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems , 35:5775-5787, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning , 2024.

Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04997 , 2024.

Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum? id=GTDKo3Sv9p .

Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736 , 2024.

Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329 , 2024.

Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524 , 2024.

Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact implementation through uniformization. arXiv preprint arXiv:2402.08095 , 2024.

Zikun Zhang, Zixiang Chen, and Quanquan Gu. Convergence of score-based discrete diffusion models: A discrete-time analysis. arXiv preprint arXiv:2410.02321 , 2024.

Yinuo Ren, Haoxuan Chen, Grant M Rotskoff, and Lexing Ying. How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. arXiv preprint arXiv:2410.03601 , 2024.

Peter Eris Kloeden and Eckhard Platen. Numerical solution of stochastic differential equations . Stochastic Modelling and Applied Probability, Applications of Mathematics, Springer, 1992.

Peter Eris Kloeden, Eckhard Platen, and Henri Schurz. Numerical solution of SDE through computer experiments . Springer Science & Business Media, 2012.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Loek7hfb46P .

Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=1hKE9qjvz-.

Martin Gonzalez, Nelson Fernandez Pinto, Thuy Tran, Hatem Hajri, Nader Masmoudi, et al. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. Sa-solver: Stochastic adams solver for fast sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Qinsheng Zhang, Jiaming Song, and Yongxin Chen. Improved order analysis and design of exponential integrator for diffusion models sampling. arXiv preprint arXiv:2308.02157 , 2023.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems , 35:30150-30166, 2022.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems , 35: 26565-26577, 2022.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems , 36:55502-55542, 2023.

Gen Li, Yu Huang, Timofey Efimov, Yuting Wei, Yuejie Chi, and Yuxin Chen. Accelerating convergence of score-based diffusion models, provably. arXiv preprint arXiv:2403.03852 , 2024.

Yuchen Wu, Yuxin Chen, and Yuting Wei. Stochastic Runge-Kutta methods: Provable acceleration of diffusion models. arXiv preprint arXiv:2410.04760 , 2024.

Daniel T Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. The Journal of chemical physics , 115(4):1716-1733, 2001.

Yang Cao, Linda R Petzold, Muruhan Rathinam, and Daniel T Gillespie. The numerical stability of leaping methods for stochastic simulation of chemically reacting systems. The Journal of chemical physics , 121(24):12169-12178, 2004.

K Burrage and T Tian. Poisson runge-kutta methods for chemical reaction systems in advances in scientific computing and applications, 2004.

Yucheng Hu and Tiejun Li. Highly accurate tau-leaping methods with random corrections. The Journal of chemical physics , 130(12), 2009.

Markus Arns, Peter Buchholz, and Andriy Panchenko. On the numerical analysis of inhomogeneous continuous-time markov chains. INFORMS Journal on Computing , 22(3):416-432, 2010.

Desmond J Higham. Modeling and simulating chemical reactions. SIAM review , 50(2):347368, 2008.

Weinan E, Tiejun Li, and Eric Vanden-Eijnden. Applied stochastic analysis , volume 199. American Mathematical Soc., 2021.

Frank P Kelly. Reversibility and stochastic networks . Cambridge University Press, 2011.

Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models. Journal of the Royal Statistical Society Series B: Statistical Methodology , 86(2):286-301, 2024.

Joe Benton, Valentin De Bortoli, Arnaud Doucet, and George Deligiannidis. Nearly d -linear convergence bounds for diffusion models via stochastic localization. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/ forum?id=r5njV3BsuD .

Sitan Chen, Sinho Chewi, Holden Lee, Yuanzhi Li, Jianfeng Lu, and Adil Salim. The probability flow ode is provably fast. Advances in Neural Information Processing Systems , 36, 2024.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11315-11325, 2022.

Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992 , 2025.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github. io/OpenWebTextCorpus , 2019.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. arXiv preprint arXiv:2310.14400 , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Peter Holderrieth, Michael S Albergo, and Tommi Jaakkola. Leaps: A discrete neural sampler via locally equivariant networks. arXiv preprint arXiv:2502.10843 , 2025.

Yuchen Zhu, Wei Guo, Jaemoo Choi, Guan-Horng Liu, Yongxin Chen, and Molei Tao. Mdns: Masked diffusion neural sampler via stochastic optimal control. arXiv preprint arXiv:2508.10684 , 2025.

Zijing Ou, Ruixiang Zhang, and Yingzhen Li. Discrete neural flow samplers with locally equivariant transformer. arXiv preprint arXiv:2505.17741 , 2025.

Wei Guo, Jaemoo Choi, Yuchen Zhu, Molei Tao, and Yongxin Chen. Proximal diffusion neural sampler. arXiv preprint arXiv:2510.03824 , 2025.

Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, and Tommaso Biancalani. Inference-time alignment in diffusion models with reward-guided generation: Tutorial and review. arXiv preprint arXiv:2501.09685 , 2025.

Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Scaling inference time compute for diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 2523-2534, 2025.

Raghav Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. arXiv preprint arXiv:2501.06848 , 2025.

Marta Skreta, Tara Akhound-Sadegh, Viktor Ohanesian, Roberto Bondesan, Alán AspuruGuzik, Arnaud Doucet, Rob Brekelmans, Alexander Tong, and Kirill Neklyudov. Feynmankac correctors in diffusion: Annealing, guidance, and product of experts. arXiv preprint arXiv:2503.02819 , 2025.

Haoxuan Chen, Yinuo Ren, Martin Renqiang Min, Lexing Ying, and Zachary Izzo. Solving inverse problems via diffusion-based priors: An approximation-free ensemble sampling approach. arXiv preprint arXiv:2506.03979 , 2025.

Yinuo Ren, Wenhao Gao, Lexing Ying, Grant M Rotskoff, and Jiequn Han. Driftlite: Lightweight drift control for inference-time scaling of diffusion models. arXiv preprint arXiv:2509.21655 , 2025.

Sophia Tang, Yuchen Zhu, Molei Tao, and Pranam Chatterjee. Tr2-d2: Tree search guided trajectory-aware fine-tuning for discrete diffusion. arXiv preprint arXiv:2509.25171 , 2025.

Meihua Dang, Jiaqi Han, Minkai Xu, Kai Xu, Akash Srivastava, and Stefano Ermon. Inference-time scaling of diffusion language models with particle gibbs sampling. arXiv preprint arXiv:2507.08390 , 2025.

Vignav Ramesh and Morteza Mardani. Test-time scaling of diffusion models via noise trajectory search. arXiv preprint arXiv:2506.03164 , 2025.

Vineet Jain, Kusha Sareen, Mohammad Pedramfar, and Siamak Ravanbakhsh. Diffusion tree sampling: Scalable inference-time alignment of diffusion models. arXiv preprint arXiv:2506.20701 , 2025.

Tianlang Chen, Minkai Xu, Jure Leskovec, and Stefano Ermon. Rfg: Test-time scaling for diffusion large language model reasoning with reward-free guidance. arXiv preprint arXiv:2509.25604 , 2025.

Xiangcheng Zhang, Haowei Lin, Haotian Ye, James Zou, Jianzhu Ma, Yitao Liang, and Yilun Du. Inference-time scaling of diffusion models through classical search. arXiv preprint arXiv:2505.23614 , 2025.

Mohsin Hasan, Marta Skreta, Alan Aspuru-Guzik, Yoshua Bengio, and Kirill Neklyudov. Discrete feynman-kac correctors. In 2nd AI for Math Workshop@ ICML 2025 , 2025.

Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307 , 2025.

Cheuk Kit Lee, Paul Jeha, Jes Frellsen, Pietro Lio, Michael Samuel Albergo, and Francisco Vargas. Debiasing guidance for discrete diffusion with sequential monte carlo. arXiv preprint arXiv:2502.06079 , 2025.

Zijing Ou, Chinmay Pani, and Yingzhen Li. Inference-time scaling of discrete diffusion models via importance weighting and optimal proposal design. arXiv e-prints , pages arXiv2505, 2025.

Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, et al. Diffusion of thought: Chain-of-thought reasoning in diffusion language models. Advances in Neural Information Processing Systems , 37:105345105374, 2024.

Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15487 , 2025.

Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223 , 2025.

Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216 , 2025.

Chengyu Wang, Paria Rashidinejad, DiJia Su, Song Jiang, Sid Wang, Siyan Zhao, Cai Zhou, Shannon Zejiang Shen, Feiyu Chen, Tommi Jaakkola, et al. Spg: Sandwiched policy gradient for masked diffusion language models. arXiv preprint arXiv:2510.09541 , 2025.

Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, et al. Inpainting-guided policy optimization for diffusion large language models. arXiv preprint arXiv:2509.10396 , 2025.

Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang. Coevolutionary continuous discrete diffusion: Make your diffusion language model a latent reasoner. arXiv preprint arXiv:2510.03206 , 2025.

Yuchen Zhu, Wei Guo, Jaemoo Choi, Petr Molodyk, Bo Yuan, Molei Tao, and Yongxin Chen. Enhancing reasoning for diffusion llms via distribution matching policy optimization. arXiv preprint arXiv:2510.08233 , 2025.

Ilia Igashov, Arne Schneuing, Marwin Segler, Michael Bronstein, and Bruno Correia. Retrobridge: Modeling retrosynthesis with markov bridges. arXiv preprint arXiv:2308.16212 , 2023.

Yang Li, Jinpei Guo, Runzhong Wang, and Junchi Yan. From distribution learning in training to gradient search in testing for combinatorial optimization. Advances in Neural Information Processing Systems , 36, 2024.

Zhiqing Sun and Yiming Yang. Difusco: Graph-based diffusion solvers for combinatorial optimization. Advances in Neural Information Processing Systems , 36:3706-3731, 2023.

Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Bac Nguyen, Stefano Ermon, and Yuki Mitsufuji. G2d2: Gradient-guided discrete diffusion for image inverse problem solving. arXiv preprint arXiv:2410.14710 , 2024.

Wenda Chu, Yang Song, and Yisong Yue. Split gibbs discrete diffusion posterior sampling. arXiv preprint arXiv:2503.01161 , 2025.

Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X Lu, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. BioRxiv , pages 2023-09, 2023.

Patrick Emami, Aidan Perreault, Jeffrey Law, David Biagioni, and Peter St John. Plug & play directed evolution of proteins with gradient-based discrete mcmc. Machine Learning: Science and Technology , 4(2):025014, 2023.

Dmitry Penzar, Daria Nogina, Elizaveta Noskova, Arsenii Zinkevich, Georgy Meshcheryakov, Andrey Lando, Abdul Muntakim Rafi, Carl De Boer, and Ivan V Kulakovskiy. Legnet: a bestin-class deep learning model for short dna regulatory regions. Bioinformatics , 39(8):btad457, 2023.

Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature , 620(7976):1089-1100, 2023.

John J Yang, Jason Yim, Regina Barzilay, and Tommi Jaakkola. Fast non-autoregressive inverse folding with discrete diffusion. arXiv preprint arXiv:2312.02447 , 2023.

Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05841 , 2024.

Kai Yi, Bingxin Zhou, Yiqing Shen, Pietro Liò, and Yuguang Wang. Graph denoising diffusion for inverse protein folding. Advances in Neural Information Processing Systems , 36, 2024.

Yiheng Zhu, Jialu Wu, Qiuyi Li, Jiahuan Yan, Mingze Yin, Wei Wu, Mingyang Li, Jieping Ye, Zheng Wang, and Jian Wu. Bridge-if: Learning inverse protein folding with markov bridges. arXiv preprint arXiv:2411.02120 , 2024.

Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in neural information processing systems , 34:3518-3532, 2021.

Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Discrete predictor-corrector diffusion models for image synthesis. In The Eleventh International Conference on Learning Representations , 2022.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

Ari Seff, Wenda Zhou, Farhan Damani, Abigail Doyle, and Ryan P Adams. Discrete object generation with reversible inductive construction. Advances in neural information processing systems , 32, 2019.

Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics , pages 4474-4484. PMLR, 2020.

Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation. arXiv preprint arXiv:2001.09382 , 2020.

Yiming Qin, Clement Vignac, and Pascal Frossard. Sparse training of discrete diffusion models for graph generation. arXiv preprint arXiv:2311.02142 , 2023.

Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

Kilian Konstantin Haefeli, Karolis Martinkus, Nathanaël Perraudin, and Roger Wattenhofer. Diffusion models for graphs benefit from discrete state spaces. arXiv preprint arXiv:2210.01549 , 2022.

Yiming Qin, Manuel Madeira, Dorina Thanou, and Pascal Frossard. Defog: Discrete flow matching for graph generation. arXiv preprint arXiv:2410.04263 , 2024.

Jun Hyeong Kim, Seonghwan Kim, Seokhyun Moon, Hyeongwoo Kim, Jeheon Woo, and Woo Youn Kim. Discrete diffusion schrödinger bridge matching for graph transformation. arXiv preprint arXiv:2410.01500 , 2024.

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10167-10176, 2023.

Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and Dongmei Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7226-7236, 2023.

Seunggeun Chi, Hyung-gun Chi, Hengbo Ma, Nakul Agarwal, Faizan Siddiqui, Karthik Ramani, and Kwonjoon Lee. M2d2m: Multi-motion generation from text with discrete diffusion models. arXiv preprint arXiv:2407.14502 , 2024.

Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, and Yi Yang. Diversemotion: Towards diverse human motion generation via discrete diffusion. arXiv preprint arXiv:2309.01372 , 2023.

Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 31:1720-1733, 2023.

Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P Breckon, and Chris G Willcocks. Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes. In European Conference on Computer Vision , pages 170-188. Springer, 2022.

Zhicong Tang, Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint arXiv:2205.16007 , 2022.

Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete contrastive diffusion for cross-modal music and image generation. arXiv preprint arXiv:2206.07771 , 2022.

Zhichao Wu, Qiulin Li, Sixing Liu, and Qun Yang. Dctts: Discrete diffusion model with contrastive learning for text-to-speech generation. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 11336-11340. IEEE, 2024.

Jun Han, Zixiang Chen, Yongqian Li, Yiwen Kou, Eran Halperin, Robert E Tillman, and Quanquan Gu. Guided discrete diffusion for electronic health record generation. arXiv preprint arXiv:2404.12314 , 2024.

Juntong Shi, Minkai Xu, Harper Hua, Hengrui Zhang, Stefano Ermon, and Jure Leskovec. Tabdiff: a multi-modal diffusion model for tabular data generation. arXiv preprint arXiv:2410.20626 , 2024.

Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029 , 2022.

Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. arXiv preprint arXiv:2112.06749 , 2021.

Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems , 36:39957-39974, 2023.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq-v2: Bridging discrete and continuous text spaces for accelerated seq2seq diffusion models. arXiv preprint arXiv:2310.05793 , 2023.

Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737 , 2023.

Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion-nat: Self-prompting discrete diffusion for non-autoregressive text generation. arXiv preprint arXiv:2305.04044 , 2023.

Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21357 , 2024.

Yinuo Ren, Grant M Rotskoff, and Lexing Ying. A unified approach to analysis and design of denoising markov models. arXiv preprint arXiv:2504.01938 , 2025.

Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. arXiv preprint arXiv:2410.07761 , 2024.

Yixiu Zhao, Jiaxin Shi, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243 , 2024.

Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via discrete non-markov diffusion models with predetermined transition time. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discrete&continuous-time discrete denoising diffusion. arXiv preprint arXiv:2402.03701 , 2024.

Machel Reid, Vincent Josua Hellendoorn, and Graham Neubig. Diffuser: Diffusion via editbased reconstruction. In The Eleventh International Conference on Learning Representations , 2023.

Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. Distillation of discrete diffusion through dimensional correlations. arXiv preprint arXiv:2410.08709 , 2024.

Ludwig Winkler, Lorenz Richter, and Manfred Opper. Bridging discrete and continuous state spaces: Exploring the ehrenfest process in time-continuous diffusion models. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/ forum?id=8GYclcxQXB .

Harshit Varma, Dheeraj Nagaraj, and Karthikeyan Shanmugam. Glauber generative model: Discrete diffusion models via binary classification. arXiv preprint arXiv:2405.17035 , 2024.

Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien LafranceVanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems , 36, 2024.

Severi Rissanen, Markus Heinonen, and Arno Solin. Improving discrete diffusion models via structured preferential generation. arXiv preprint arXiv:2405.17889 , 2024.

Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael Gómez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. arXiv preprint arXiv:2410.06264 , 2024.

Oscar Davis, Samuel Kessler, Mircea Petrache, Avishek Joey Bose, et al. Fisher flow matching for generative modeling over discrete data. arXiv preprint arXiv:2405.14664 , 2024.

GN Mil'shtejn. Approximate integration of stochastic differential equations. Theory of Probability & Its Applications , 19(3):557-562, 1975.

Assyr Abdulle and Stephane Cirilli. S-rock: Chebyshev methods for stiff stochastic differential equations. SIAM Journal on Scientific Computing , 30(2):997-1014, 2008.

Evelyn Buckwar and Renate Winkler. Multistep methods for sdes and their application to problems with small noise. SIAM journal on numerical analysis , 44(2):779-803, 2006.

Denis Talay and Luciano Tubaro. Expansion of the global error for numerical schemes solving stochastic differential equations. Stochastic analysis and applications , 8(4):483-509, 1990.

Kevin Burrage and Pamela M Burrage. Order conditions of stochastic runge-kutta methods by b-series. SIAM Journal on Numerical Analysis , 38(5):1626-1646, 2000.

Kevin Burrage and Tianhai Tian. Predictor-corrector methods of runge-kutta type for stochastic differential equations. SIAM Journal on Numerical Analysis , 40(4):1516-1537, 2002.

Andreas Rössler. Runge-kutta methods for the numerical solution of stochastic differential equations. Shaker-Verlag, Aachen , 2003.

Andreas Rößler. Runge-kutta methods for the strong approximation of solutions of stochastic differential equations. SIAM Journal on Numerical Analysis , 48(3):922-952, 2010.

James M Foster, Goncalo Dos Reis, and Calum Strange. High order splitting methods for sdes satisfying a commutativity condition. SIAM Journal on Numerical Analysis , 62(1):500-532, 2024.

Lei Li, Jianfeng Lu, Jonathan Mattingly, and Lihan Wang. Numerical methods for stochastic differential equations based on gaussian mixtures. Communications in Mathematical Sciences , 19(6):1549-1577, 2021.

Ruoqi Shen and Yin Tat Lee. The randomized midpoint method for log-concave sampling. Advances in Neural Information Processing Systems , 32, 2019.

Nima Anari, Sinho Chewi, and Thuy-Duong Vuong. Fast parallel sampling under isoperimetry. arXiv preprint arXiv:2401.09016 , 2024.

Lu Yu and Arnak Dalalyana. Parallelized midpoint randomization for langevin monte carlo. arXiv preprint arXiv:2402.14434 , 2024.

Changyou Chen, Nan Ding, and Lawrence Carin. On the convergence of stochastic gradient mcmc algorithms with high-order integrators. Advances in neural information processing systems , 28, 2015.

Alain Durmus, Umut Simsekli, Eric Moulines, Roland Badeau, and Gaël Richard. Stochastic gradient richardson-romberg markov chain monte carlo. Advances in neural information processing systems , 29, 2016.

Xuechen Li, Yi Wu, Lester Mackey, and Murat A Erdogdu. Stochastic runge-kutta accelerates langevin monte carlo and beyond. Advances in neural information processing systems , 32, 2019.

Sotirios Sabanis and Ying Zhang. Higher order langevin monte carlo algorithm. Electron. J. Statist , 13(2):3805-3850, 2019.

Wenlong Mou, Yi-An Ma, Martin J Wainwright, Peter L Bartlett, and Michael I Jordan. Highorder langevin diffusion yields an accelerated mcmc algorithm. Journal of Machine Learning Research , 22(42):1-41, 2021.

Pierre Monmarché. High-dimensional mcmc with a standard splitting scheme for the underdamped langevin diffusion. Electronic Journal of Statistics , 15(2):4117-4166, 2021.

James Foster, Terry Lyons, and Harald Oberhauser. The shifted ode method for underdamped langevin mcmc. arXiv preprint arXiv:2101.03446 , 2021.

Kevin Burrage, PM Burrage, and Tianhai Tian. Numerical methods for strong solutions of stochastic differential equations: an overview. Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences , 460(2041):373-402, 2004.

Grigori N Milstein and Michael V Tretyakov. Stochastic numerics for mathematical physics , volume 39. Springer, 2004.

Alfred B Bortz, Malvin H Kalos, and Joel L Lebowitz. A new algorithm for monte carlo simulation of ising spin systems. Journal of Computational physics , 17(1):10-18, 1975.

Daniel T Gillespie. A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. Journal of computational physics , 22(4):403-434, 1976.

Daniel T Gillespie. Exact stochastic simulation of coupled chemical reactions. The journal of physical chemistry , 81(25):2340-2361, 1977.

Yang Cao, Dan Gillespie, and Linda Petzold. Multiscale stochastic simulation algorithm with stochastic partial equilibrium assumption for chemically reacting systems. Journal of Computational Physics , 206(2):395-411, 2005.

Yang Cao, Daniel T Gillespie, and Linda R Petzold. The slow-scale stochastic simulation algorithm. The Journal of chemical physics , 122(1), 2005.

Weinan E, Di Liu, Eric Vanden-Eijnden, et al. Nested stochastic simulation algorithm for chemical kinetic systems with disparate rates. The Journal of chemical physics , 123(19), 2005.

Weinan E, Di Liu, and Eric Vanden-Eijnden. Nested stochastic simulation algorithms for chemical kinetic systems with multiple time scales. Journal of computational physics , 221 (1):158-180, 2007.

Michael A Gibson and Jehoshua Bruck. Efficient exact stochastic simulation of chemical systems with many species and many channels. The journal of physical chemistry A , 104(9): 1876-1889, 2000.

David F Anderson. A modified next reaction method for simulating chemical systems with time dependent propensities and delays. The Journal of chemical physics , 127(21), 2007.

Casper HL Beentjes and Ruth E Baker. Uniformization techniques for stochastic simulation of chemical reaction networks. The Journal of Chemical Physics , 150(15), 2019.

Muruhan Rathinam, Linda R Petzold, Yang Cao, and Daniel T Gillespie. Stiffness in stochastic chemically reacting systems: The implicit tau-leaping method. The Journal of Chemical Physics , 119(24):12784-12794, 2003.

Daniel T Gillespie and Linda R Petzold. Improved leap-size selection for accelerated stochastic simulation. The journal of chemical physics , 119(16):8229-8234, 2003.

Kevin Burrage, Tianhai Tian, and Pamela Burrage. A multi-scaled approach for simulating chemical reaction systems. Progress in biophysics and molecular biology , 85(2-3):217-234, 2004.

Yang Cao, Daniel T Gillespie, and Linda R Petzold. Avoiding negative populations in explicit poisson tau-leaping. The Journal of chemical physics , 123(5), 2005.

Anne Auger, Philippe Chatelain, and Petros Koumoutsakos. R-leaping: Accelerating the stochastic simulation algorithm by reaction leaps. The Journal of chemical physics , 125(8), 2006.

Yang Cao, Daniel T Gillespie, and Linda R Petzold. Adaptive explicit-implicit tau-leaping method with automatic tau selection. The Journal of chemical physics , 126(22), 2007.

Basil Bayati, Philippe Chatelain, and Petros Koumoutsakos. D-leaping: Accelerating stochastic simulation algorithms for reactions with delays. Journal of Computational Physics , 228 (16):5908-5916, 2009.

Yang Cao and Linda Petzold. Slow-scale tau-leaping method. Computer methods in applied mechanics and engineering , 197(43-44):3472-3479, 2008.

Zhouyi Xu and Xiaodong Cai. Unbiased τ -leap methods for stochastic simulation of chemically reacting systems. The Journal of chemical physics , 128(15), 2008.

Mary Sehl, Alexander V Alekseyenko, and Kenneth L Lange. Accurate stochastic simulation via the step anticipation τ -leaping (sal) algorithm. Journal of Computational Biology , 16(9): 1195-1208, 2009.

Krishna A Iyengar, Leonard A Harris, and Paulette Clancy. Accurate implementation of leaping in space: The spatial partitioned-leaping algorithm. The Journal of chemical physics , 132(9), 2010.

David F Anderson and Desmond J Higham. Multilevel monte carlo for continuous time markov chains, with applications in biochemical kinetics. Multiscale Modeling & Simulation , 10(1):146-179, 2012.

Alvaro Moraes, Raúl Tempone, and Pedro Vilanova. Hybrid chernoff tau-leap. Multiscale Modeling & Simulation , 12(2):581-615, 2014.

Jill Padgett and Silvana Ilie. An adaptive tau-leaping method for stochastic simulations of reaction-diffusion systems. AIP Advances , 6(3), 2016.

Jana Lipková, Georgios Arampatzis, Philippe Chatelain, Bjoern Menze, and Petros Koumoutsakos. S-leaping: an adaptive, accelerated stochastic simulation algorithm, bridging τ -leaping and r-leaping. Bulletin of mathematical biology , 81(8):3074-3096, 2019.

Muruhan Rathinam, Linda R Petzold, Yang Cao, and Daniel T Gillespie. Consistency and stability of tau-leaping schemes for chemical reaction systems. Multiscale Modeling & Simulation , 4(3):867-895, 2005.

Tiejun Li. Analysis of explicit tau-leaping schemes for simulating chemically reacting systems. Multiscale Modeling & Simulation , 6(2):417-436, 2007.

Yucheng Hu, Tiejun Li, and Bin Min. The weak convergence analysis of tau-leaping methods: revisited. Communications in Mathematical Sciences , 9(4):965-996, 2011.

David F Anderson, Desmond J Higham, and Yu Sun. Complexity of multilevel monte carlo tau-leaping. SIAM Journal on Numerical Analysis , 52(6):3106-3127, 2014.

Chuchu Chen and Di Liu. Error analysis for d-leaping scheme of chemical reaction system with delay. Multiscale Modeling & Simulation , 15(4):1797-1829, 2017.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Linfeng Zhang, Weinan E, and Lei Wang. Monge-ampère flow for generative modeling. arXiv preprint arXiv:1809.10188 , 2018.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems , 34: 1415-1428, 2021.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 , 2022.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571 , 2022.

Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797 , 2023.

Haoxuan Chen, Yinuo Ren, Lexing Ying, and Grant M Rotskoff. Accelerating diffusion models with parallel sampling: Inference at sub-linear time complexity. arXiv preprint arXiv:2405.15986 , 2024.

Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7777-7786, 2024.

Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems , 36:76806-76838, 2023.

Hanzhong Guo, Cheng Lu, Fan Bao, Tianyu Pang, Shuicheng Yan, Chao Du, and Chongxuan Li. Gaussian mixture solvers for diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arXiv:2403.06807 , 2024.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189 , 2023.

Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081 , 2024.

Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 , 2021.

Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14297-14306, 2023.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Vinh Tong, Trung-Dung Hoang, Anji Liu, Guy Van den Broeck, and Mathias Niepert. Learning to discretize denoising diffusion odes. arXiv preprint arXiv:2405.15506 , 2024.

Eric Frankel, Sitan Chen, Jerry Li, Pang Wei Koh, Lillian J Ratliff, and Sewoong Oh. S4s: Solving for a diffusion model solver. arXiv preprint arXiv:2502.17423 , 2025.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068 , 2021.

Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning , pages 42390-42402. PMLR, 2023.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15762-15772, 2024.

Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, and Arnaud Doucet. Accelerated diffusion models via speculative sampling. arXiv preprint arXiv:2501.05370 , 2025.

Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 , 2021.

Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Andy Shih, Suneel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems , 36, 2024.

Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6059-6069, 2023.

Zhiwei Tang, Jiasheng Tang, Hao Luo, Fan Wang, and Tsung-Hui Chang. Accelerating parallel sampling of diffusion models. In Forty-first International Conference on Machine Learning , 2024.

Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, and Luc Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2824-2834, 2024.

Nikil Roashan Selvam, Amil Merchant, and Stefano Ermon. Self-refining diffusion samplers: Enabling parallelization via parareal iterations. arXiv preprint arXiv:2412.08292 , 2024.

Saravanan Kandasamy and Dheeraj Nagaraj. The poisson midpoint method for langevin dynamics: Provably efficient discretization for diffusion models. arXiv preprint arXiv:2405.17068 , 2024.

Shivam Gupta, Linda Cai, and Sitan Chen. Faster diffusion-based sampling with randomized midpoints: Sequential and parallel. arXiv preprint arXiv:2406.00924 , 2024.

Philip Protter. Point process differentials with evolving intensities. In Nonlinear stochastic problems , pages 467-472. Springer, 1983.

Bernt Øksendal and Agnes Sulem. Applied Stochastic Control of Jump Diffusions . Springer, 2019.

A Further Discussion on Related Works

In this section, we provide a more detailed literature review of both continuous and discrete diffusion models, as well as several studies on the numerical methods for SDEs and chemical reaction systems, which are highly related to our work.

Discrete Diffusion Models: Methodology, Theory, and Applications. Discrete diffusion and flow-based models [1-4, 6-10, 22] have recently been proposed as generalizations of continuous diffusion models to model discrete distributions.

Such models have been widely used in various areas of science and engineering, including but not limited to modeling retrosynthesis [100], combinatorial optimization [101, 102], solving inverse problems [103, 104] and sampling high-dimensional discrete distributions [90, 72], designing molecules, proteins, and DNA sequences [105, 13, 106, 12, 107-109, 31, 110, 11, 111, 112], image synthesis [113-115], text summarization [15], as well as the generation of graph [116-123], layout [124, 125], motion [126, 127], sound [22, 128], image [16, 129-131], speech [132], electronic health record [133], tabular data [134] and text [135-140, 34, 35, 141, 14]. Inspired by the huge success achieved by discrete diffusion models in practice, researchers have also conducted some studies on the theoretical properties of these models, such as [36-38, 142].

An extensive amount of work has also explored the possibility of making discrete diffusion models more effective from many aspects, such as optimizing the sampling schedule [143], adding correctors [144], developing fast samplers [145], designing correctors based on information learnt by the model [144], simplifying the loss function for training [146], adding editing-based refinements [147], synergizing these models with other techniques and methodologies like distillation [148], Ehrenfest processes [149], Glauber dynamics [150], tensor networks [17], enhanced guidance mechanisms [151, 18-20], structured preferential generation [152], the plan-and-denoise framework [153] and alternative metrics, e.g. , the Fisher information metric [154]. However, to the best of our knowledge, existing work on accelerating the inference of discrete diffusion models is relatively sparse compared to the ones we listed above, which makes it a direction worth exploring and serves as one of the main motivations behind this work.

Numerical Methods for SDEs and Chemical Reaction Systems. Below, we review advanced numerical methods proposed for simulating SDEs and chemical reaction systems, which are the main techniques adopted in our work. For the simulation of SDEs driven by Brownian motions, many studies have been performed to design more accurate numerical schemes, which have been widely applied to tackle problems in computational physics, optimization, and Monte Carlo sampling. Examples of such work include the Milstein method [155], explicit methods [156], multistep methods [157], extrapolation-type methods [158, 25], stochastic Runge Kutta methods [24, 159162], splitting methods [163], methods based on gaussian mixtures [164], randomized midpoint method [165], parallel sampling methods [166, 167] as well as high-order methods for stochastic gradient Markov Chain Monte Carlo [168, 169], underdamped and overdamped Langevin Monte Carlo [170-174]. For a more comprehensive list of related numerical methods, one may refer to [39, 175, 176, 40, 58].

Regarding the simulation of chemical reaction systems, numerical methods can be categorized into two classes. The first class consists of exact simulation methods, which are similar to the Kinetic Monte Carlo (KMC) method [177] developed for simulating spin dynamics and crystal growth in condensed matter physics. Examples of such methods include the Gillespie algorithm (or the Stochastic Simulation Algorithm, a.k.a. SSA) [178, 179] and its variants for multiscale modeling [180-183], the next reaction method and its variants [184, 185], uniformizationbased methods [186], etc. The second class of methods are approximate simulation methods, including but not limited to the τ -leaping method [52] and its variants [187, 188, 53, 54, 189195, 55, 196, 56, 197, 26, 198-201]. For a subset of the methods listed above, numerical analysis has also been performed in many works [202-206] to justify their validity.

Continuous Diffusion Models: Methodology, Theory, and Acceleration. Continuous diffusion and probability flow-based models [207-216] have also been the most popular methods in generative modeling, with a wide range of applications in science and engineering. For a list of related work on the theoretical studies and applications of these models, one may refer to the literature review

conducted in [217, 38]. Here we will only review studies on accelerating the inference of continuous diffusion models, which motivates our work.

An incomplete list of accelerating methods includes approximate mean direction solver [218], restart sampling [219], gaussian mixture solvers [220], self-consistency [221-224], knowledge distillation [225-229], combination with underdamped Langevin dynamics [230], operator learning [231] and more recently ideas from accelerating large language models (LLMs) like caching [232] and speculative decoding [233]. Among all the proposed accelerating methods, one major class of methods are developed based on techniques from numerical analysis like adaptive step sizes [234], exponential integrators [41-43], predictor-corrector solver [235], Adams-Bashforth methods [29, 44, 45], Taylor methods [27, 46], Picard iteration and parallel sampling [236-240, 217], (stochastic) RungeKutta methods [47, 28, 48-51] and randomized midpoint method [241, 242]. In contrast, there have been fewer studies on the acceleration of discrete diffusion models via techniques from numerical analysis, which inspires the study undertaken in this paper.

B Mathematical Background

In this section, we provide the mathematical background for the stochastic integral formulation of discrete diffusion models, the error analysis of the τ -leaping method, and useful lemmas for the theoretical analysis of high-order schemes for discrete diffusion models.

B.1 Stochastic Integral Formulation of Discrete Diffusion Models

Throughout this section, we will assume that (Ω , F , P ) is a probability space, X is a finite-state space, and denote the pairwise difference set of the state space by D := { x -y : x = y ∈ X } . We also assume that the pairwise difference set X is equipped with a metric ∥· ∥ , a finite measure γ , and a σ -algebra B .

̸

As a warm-up, we introduce the definition of the Poisson random measure for a time-homogeneous counting process.

Definition B.1 (Poisson Random Measure [38, Definition A.1]) . The random measure N (d t, d ν ) on R + × D is called a Poisson random measure w.r.t. measure γ if it is a random counting measure satisfying the following properties:

(i) For any B and 0 s < t

∈ B ≤ , )) ;

(ii) For any t 0 and pairwise disjoint sets B

are independent stochastic processes.

Then we define the Poisson random measure with evolving intensities. The term 'evolving' refers to that the intensity is both time and state-dependent.

Definition B.2 (Poisson Random Measure with Evolving Intensity [38, Definition A.3]) . Suppose λ t ( y ) is a non-negative predictable process on R + × D × Ω satisfying that for any 0 ≤ T < T , ∫ T 0 λ t ( ν )d t < ∞ , a.s..

The random measure N [ λ ](d t, d ν ) on R + × D is called a Poisson random measure with evolving intensity λ t ( ν ) w.r.t. measure γ if it is a random counting measure satisfying the following properties:

(i) For any B ∈ B and 0 ≤ s < t ,

(ii) For any t 0 and pairwise disjoint sets B i

are independent stochastic processes.

Remark B.3 (Construction of Poisson Random Measure with Evolving Intensity) . As discussed in Thm. A.4 in [38] and originally proposed by [243], the Poisson random measure with evolving intensity can be constructed in the following way.

One first augments the ( X , B , ν ) measure space to a product space ( D × R , B × B ( R ) , γ × m ) , where m is the Lebesgue measure on R , and B ( R ) is the Borel σ -algebra on R . The Poisson random measure with evolving intensity λ t ( ν ) can be defined in the augmented measure space as

where N (d τ, d ν, d ξ ) is the Poisson random measure on R + × D × R w.r.t. measure ν (d y )d ξ .

The following theorem provides the change of measure theorem for Poisson random measure with evolving intensity, which is crucial for the theoretical analysis of numerical schemes for discrete diffusion models.

Theorem B.4 (Change of Measure for Poisson Random Measure with Evolving Density [38, Thm. 3.3]) . Let N [ λ ](d t, d ν ) be a Poisson random measure with evolving intensity λ t ( ν ) , and h t ( ν ) a positive predictable process on R + × D × Ω . Suppose the following exponential process is a local F t -martingale:

and Q is another probability measure on (Ω , F ) such that Q ≪ P with Radon-Nikodym derivative d Q / d P | F t = Z t [ h ] .

Then the Poisson random measure N [ λ ](d t, d ν ) under the measure Q is a Poisson random measure with evolving intensity λ t ( ν ) h t ( ν ) .

B.2 Error Analysis of τ -leaping

The τ -leaping method was originally proposed by [52] and adopted for the inference of discrete diffusion models by [22]. A summary of the algorithm is given in Alg. 3. In this subsection, we provide a sketch for the error analysis of the τ -leaping method when applied to discrete diffusion models, which will be compared with that of high-order schemes later on.

Algorithm 3: τ -Leaping Method for Discrete Diffusion Model Inference

Input: ̂ y 0 ∼ q 0 , θ ∈ [0 , 1] , time discretization ( s n , ρ n ) n ∈ [0: N -1] , ̂ µ , ̂ µ ∗ as defined in Prop. C.2. Output: A sample y s N ∼ q RK t N .

Proof of Thm. 3.1. As we are considering the case where X = [ S ] d , i.e. the state space is a d -dimensional grid with S states along each dimension, we have log | X | = d log S . Then we consider a simple time-homogeneous transition matrix Q t ≡ Q that allows jumps between neighboring states with equal probability. Specifically, we have

which can be verified to satisfy Assumption 4.3(i) in [38] with C = 1 and D = D = 2 d . Assumption 4.3(ii) is also satisfied, as shown in Example B.10 of [38].

Then we may apply Thm. 4.7 in [38] by using the required time discretization scheme according to the properties of the target distribution and plugging in the corresponding values of C, D, D . The result follows by scaling the transition matrix Q by 1 d , equivalent to scaling the time by d .

C Proofs

In this section, we provide the missing proofs in the main text. We will first provide the proofs of the stochastic integral formulations of high-order schemes for discrete diffusion models in App. C.1. Then we will provide the proofs of the main results for the θ -Trapezoidal method in App. C.2 and the θ -RK-2 method in App. C.3. We remark that the proof for the θ -Trapezoidal method requires more techniques and is more involved, to which the proof for the θ -RK-2 method is analogous. In App. C.4, we provide the detailed lemmas and computations omitted in the proofs of Thms. 5.4 and 5.5.

C.1 Stochastic Integral Formulations of High-Order Schemes

In order to rigorously analyze the θ -RK-2 method, we need the following definition:

Definition C.1 (Intermediate Process) . We define the intermediate process ̂ y ∗ s piecewisely on ( s n , s n +1 ] as follows:

where the intensity ̂ µ s n is given by ̂ µ s n ( ν, ̂ y s n ) = ⃗ ̂ s s n ( ̂ y s n , ̂ y s n + ν ) ⃗ Q 0 s n ( ̂ y s n , ̂ y s n + ν ) , i.e. , ̂ y ∗ s is the process obtained by performing τ -leaping from time s n to s with intensity ̂ µ . The following proposition provides the stochastic integral formulation of this method.

Proposition C.2 (Stochastic Integral Formulation of θ -RK-2 Method) . The θ -RK-2 method (Alg. 1) is equivalent to solving the following stochastic integral:

in which the intensity ̂ µ RK is defined as a weighted sum and the intermediate intensity µ ∗ is defined piecewisely as

̂ ̂ ̂ ̂ ̂ ̂ ̂ with the intermediate process ̂ y ∗ s defined in (C.1) for the corresponding interval. We will call ̂ y RK s the interpolating process of the θ -RK-2 method and denote the distribution of y RK s by q RK s .

̂ ̂ The following proposition establishes the stochastic integral formulation of the θ -Trapezoidal method, whose proof can be found in App. C.1.

Proposition C.3 (Stochastic Integral Formulation of θ -Trapezoidal Method) . The θ -Trapezoidal method (Alg. 2) is equivalent to solving the following stochastic integral:

Above, 1 ( · ) denotes the indicator function and the intermediate process ̂ y ∗ s is defined in (C.1) for the corresponding interval. We will call the process ̂ y trap s the interpolating process of the θ -Trapezoidal method and denote the distribution of y trap s by q trap s .

where the intensity ̂ µ trap is defined piecewisely as ̂ µ trap s ( ν ) = 1 s<ρ s ̂ µ ⌊ s ⌋ ( ν, ̂ y trap ⌊ s ⌋ ) + 1 s ≥ ρ s ( α 1 ̂ µ ∗ ρ s ( ν, ̂ y ∗ ρ s ) -α 2 ̂ µ ⌊ s ⌋ ( ν, ̂ y trap ⌊ s ⌋ ) ) + . (C.6)

̂ ̂ Proof of Props. C.2 and C.3. Without loss of generality, we give the proof on the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , and the generalization to the whole interval [0 , T ] is straightforward.

Notice that once we condition on the filtration F s n and construct the intermediate process ̂ y ∗ s as specified in (C.1) along the interval ( s n , s n +1 ] , the intermediate intensity ̂ µ ∗ and the piecewise intensity ̂ µ ⌊ s ⌋ do not evolve with time s or the interpolating processes ̂ y RK s (or ̂ y trap s , respectively) since it

only depends on the state, the intensity at the beginning of the interval s n and other randomness that is independent of the interpolating process.

Therefore, the stochastic integral on this interval can be rewritten as for the θ -RK-2 scheme that and for the θ -Trapezoidal scheme that

and the statement follows by taking γ (d ν ) as the counting measure.

Remark C.4 (Remark on Rejection Sampling and Periodicity Assumption) . The rejection sampling procedure in both algorithms (Algs. 1 and 2) guarantees well-posedness in the rare scenarios where a large drawn value of Poisson random variables or multiple simultaneous jumps in one coordinate would result in an update out of the state space X = [ S ] d . To enforce this, we simply allow at most one jump per update across the summation, for example, in the update as the standard practice in the literature [22, 38]. The indicator function 1 ̂ µ sn > 0 in Alg. 1 is also used to ensure that only valid jumps from the current state ̂ y s n are considered, while in Alg. 2, this is implicitly guaranteed by taking the positive part of α 1 ̂ µ ∗ ρ n -α 2 ̂ µ s n , which implies the positivity of α 1 ̂ µ ∗ ρ n and thus the validity of the jumps ̂ y ∗ ρ n . We point out that the single-jump rule is only a convenient sufficient condition, one should notice that this condition is not necessary for the wellposedness of our algorithms, since our setting of the state space X carries both orderliness and algebraic structure, and thus one could in principle admit multiple simultaneous jumps without ambiguity.

Over the full inference process, the total probability of rejection is at most O ( κ ) . Below, we give a brief justification and we refer to Proposition A.14 in [38] for a complete proof of this claim. During the update aforementioned, the probability of at least two jumps occurring is bounded by

Summing O (∆ 2 n ) over N steps gives ∑ N -1 n =0 ∆ 2 n ≲ κT , and an identical argument applies to the second update in each iteration. Hence, the overall rejection rate is at most O ( κ ) .

When we impose periodic boundary conditions, X = [ S ] d is equipped with a convenient algebraic structure: addition and scalar multiplication are globally well-defined. In that case, Algs. 1 and 2 match exactly the stochastic integral formulations in Props. C.2 and C.3. This alignment removes the need for per-step rejection, streamlines the application of the change-of-measure argument, and greatly simplifies the convergence proofs of Thms. 5.4 and 5.5. Even without periodicity, those theorems hold with probability at least 1 -O ( κ ) , as shown above.

C.2 Convergence Analysis of the θ -Trapezoidal Method

Theorem C.5. Let ⃗ p 0: T -δ and ̂ q trap 0: T -δ be the path measures of the backward process with the stochastic integral formulation (2.4) and the interpolating process (C.5) of the θ -Trapezoidal method (Alg. 2), then it holds that

̂ where the intensity ̂ µ trap is defined in (C.5), and the expectation is taken w.r.t. both paths generated by the backward process (2.4) and the randomness of the Poisson random measure used in the first step of each iteration of the algorithm, i.e. , the construction of the intermediate process (C.1), which is assumed to be independent of that of the backward process.

Proof. First, we will handle the randomness introduced by the Poisson random measure in the first step of each iteration of the θ -Trapezoidal method. For the ease of presentation, we encode the aforementioned randomness as a random variable ζ and suppose it is still supported on the probability space (Ω , F , P ) while being independent of the backward process. Then for each realization of ζ , the intermediate process ̂ y ∗ s is constructed as in (C.1) and the corresponding intensity ̂ µ ∗ s is defined in (C.4).

Given the stochastic integral formulation of the backward process (2.4) and the interpolating process of the θ -Trapezoidal method (C.5), we have by Thm. B.4 that this particular realization of the path measure ̂ q trap 0: T -δ can be obtained by changing the path measure ⃗ p 0: T -δ with the Radon-Nikodym derivative

Then it is easy to see by the data processing inequality and the chain rule of KL divergence that

and the proof is complete.

In the following, we will provide the outline of the proof of Thm. 5.4, where we leave the proof of several lemmas and detailed calculations to App. C.4 for the clarity of presentation.

Proof of Thm. 5.4. Throughout this proof, including the subsequent lemmas and propositions that will be detailed in App. C.4, we will assume that ( y s ) s ∈ [0 ,T ] is a process generated by the path measure ⃗ p 0: T of the backward process with the stochastic integral formulation (2.4) and set it as the underlying paths of the expectation in (C.7) as required by Thm. C.5. Especially, y s ∼ ⃗ p s holds for any s ∈ [0 , T ] . For simplicity, we will assume that the process y s is left-continuous at each grid point s i for i ∈ [0 : N ] , which happens with probability one.

We first consider the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , and thus we have ⌊ s ⌋ = s n and ρ s = ρ n . Within this interval, we will denote its intermediate process as appeared in (C.1) as y ∗ s , and the corresponding intermediate intensity as appeared in (C.4) as ̂ µ ∗ s . In the following discussion, we will assume implicitly that the processes are conditioned on the filtration F s n .

By the definition of the intensity µ trap ( ν ) as specified in (C.6)

we can rewrite the corresponding part of the integral in (C.7) as

where the assumption that α 1 ̂ µ ∗ ρ s -α 2 ̂ µ ⌊ s ⌋ ≥ 0 for all s ∈ [0 , T -δ ] is applied here for the second term (II) above.

Decomposition of the Integral. Next, we decompose the integral (I) and (II) into several terms, the magnitudes of which or combinations of which are to be bounded.

(i) The first term is decomposed as

where each term is defined as

(ii) The second term is decomposed as

where each term is defined as

Bounding the Error Terms. Then we briefly summarize the intuitions and related techniques used in the bounds of the terms above, and the detailed calculations and proofs of the lemmas and propositions are deferred to App. C.4.

(i) Error due to estimation error associated with the intensity: The terms (I . 1) and (II . 1) are bounded by the assumption on the estimation error of the intensity µ s (Assump. 5.3), as

for any θ ∈ (0 , 1] .

The term (II . 4) is bounded by Prop. C.9, as

where Jensen's inequality is applied here based on the convexity of the loss.

(ii) Error related to the smoothness of intensity: By Cor. C.13, the terms (I . 2) and (II . 2) are bounded by

By Cor. C.14, the terms (I . 4) and (II . 6) are bounded by

Intuitively, the bounds on these terms closely relate to the properties of the jump process and quantify the smoothness assumption on the intensity µ s (Assump. 5.2), especially when the intensity does not vary significantly within the interval ( s n , s n +1 ] . The main technique used for bounding these terms is Dynkin's Formula (Thm. C.10). The third-order accuracy here directly follows from the intuition provided in Sec. 4 based on numerical quadrature.

(iii) Error involving the intermediate process: The terms (II . 3) and (II . 5) are bounded by Prop. C.18 and Cor. C.19 respectively as follows

The term (I . 3) is bounded by Prop. C.20 as below

The three terms above all involve the intermediate process y ∗ s and the corresponding intermediate density ̂ µ ∗ s .

In conclusion, by summing up all these terms, we have

Therefore, the overall error is bounded by first applying Thm. C.5 and then the upper bound derived above to each interval ( s n , s n +1 ] , which yields

as desired.

Remark C.6 (Discussion on the Positivity Assumption) . In the following, we will take the positivity assumption in Thm. 5.4 as an example, and the case of the θ -RK-2 method is similar. In the statement of Thm. 5.4, we have assumed that

α 1 ̂ µ ∗ ρ s ( ν ) -α 2 ̂ µ ⌊ s ⌋ ( ν ) ≥ 0 in (C.6) for all s ∈ [0 , T -δ ] , which allows us to replace ( α 1 ̂ µ ∗ ρ s ( ν ) -α 2 ̂ µ ⌊ s ⌋ ( ν ) ) + by the difference itself. [25] showed that this approximation is at most of O (∆ 3 n ) within the corresponding interval, and [26] further proved that for any order p ≥ 1 , there exists a sufficiently small step size ∆ such that this approximation is at least p -th order, i.e. , of order O (∆ p ) for that step.

We give a brief justification of this assumption here. We consider the expectation of the difference itself, which is given by

where we used E [ | ̂ µ ∗ ρ s ( ν ) -̂ µ ρ s ( ν ) | ] ≲ κϵ II , as established in (C.17) and E [ | ̂ µ ρ s ( ν ) -̂ µ ⌊ s ⌋ ( ν ) | ] ≲ κ , as shown in (C.18) . Therefore, as long as the step sizes ∆ n are sufficiently small, the positivity assumption is valid in the sense that the expectation of the difference is at least 1 -O ( κ ) .

C.3 Convergence Analysis of the θ -RK-2 Method

Here we may again apply the data processing inequality and the chain rule of KL divergence to upper bound the error associated with the θ -RK-2 method. A statement of the upper bound is provided in Thm. C.7 below, whose proof is omitted here since it is similar to that of Thm. C.5 above.

Theorem C.7. Let ⃗ p 0: T -δ and ̂ q RK 0: T -δ be the path measures of the backward process with the stochastic integral formulation (2.4) and the interpolating process (C.2) of the θ -RK-2 method ( Alg. 1), then it holds that

̂ where the intensity ̂ µ RK is defined in (C.2), and the expectation is taken w.r.t. both paths generated by the backward process (2.4) and the randomness of the Poisson random measure used in the first step of each iteration of the algorithm, i.e. , the construction of the intermediate process (C.1), which is assumed to be independent of that of the backward process.

Following the same flow as in the proof of Thm. 5.4, we will first provide an outline of the proof of Thm. 5.5, and defer the proof of several key lemmas and detailed calculations to App. C.4 for the clarity of presentation. We will also comment on the differences that may lead to the less desirable numerical properties of the θ -RK-2 method.

Proof of Thm. 5.5. In the following proof sketch, we will be using the same notation as in the proof of Thm. 5.4, and we will assume that the process y s is left-continuous at each grid point s i for i ∈ [0 : N ] . We also start by taking a closer look at the integral within each interval ( s n , s n +1 ] for n ∈ [0 : N -1] , and denote the intermediate process as appeared in (C.1) as y ∗ s and the corresponding intermediate intensity as appeared in (C.4) as µ ∗ s .

̂ As defined in (C.3), the intensity µ RK ( ν ) is given by which helps us rewrite the corresponding part of the integral in (C.8) as

Above we again use the positivity assumption that (1 -1 2 θ ) ̂ µ ⌊ s ⌋ + 1 2 θ ̂ µ ∗ ρ s ≥ 0 for the term (III) above, just as what we have done in the proof and discussion of Thm. 5.4 above.

Decomposition of the Integral. Then we perform a similar decomposition of the integral as in the proof of Thm. 5.4 as follows:

where each term is defined as

Bounding the Error Terms. Then we briefly summarize the intuitions and related techniques used in the bound of the terms above,. Detailed calculations and proofs of the lemmas and propositions used here are deferred to App. C.4.

(i) Error due to the intensity estimation: The terms in (III . 1) are bounded by the assumption on the estimation error of the intensity µ s (Assump. 5.3) as follows

for any θ ∈ (0 , 1] .

(ii) Error related to the smoothness of intensity: By Cors. C.16 and C.17, the terms (III . 2) and (III . 6) are bounded by

respectively.

(iii) Error involving the intermediate process: The term (III . 3) and (III . 5) are bounded in almost the same way as that of Prop. C.18 and Cor. C.19. By simply altering the integral upper limits, we obtain that

The only term that cannot be directly bounded based on results in App. C.4 is (III . 4) , which is given by

To be specific, by the assumption on the estimation error (Assump. 5.3), we can reduce (C.9) to

Recall that in the proof of its counterpart (Prop. C.9), we utilized the convexity of the loss function and the extrapolation nature of the second step in the θ -Trapezoidal method (C.6) to bound the error term. However, the same technique cannot be directly applied to the θ -RK-2 method for any θ ∈ [0 , 1] , as the intensity ̂ µ RK s is an interpolation of the intensity ̂ µ s when θ ∈ ( 1 2 , 1] . Therefore, below we will first focus on the case when θ ∈ (0 , 1 2 ] .

which can then be upper bounded based on Jensen's inequality and the convexity of the loss function for θ ∈ (0 , 1 2 ] .

Summing up the bounds of the terms above, we have

Consequently, the overall error of the θ -RK-2 method is bounded by

which suggests that the θ -RK-2 is also of second order when θ ∈ (0 , 1 2 ] . For the other case when θ ∈ ( 1 2 , 1] , we will provide a brief discussion in the remark below.

Remark C.8 (Discussions on the case when θ ∈ ( 1 2 , 1] ) . For θ ∈ ( 1 2 , 1] , the term (C.10) is positive and thus not necessarily bounded. One may wonder if, despite being positive, this term is still of at least second order. However, the answer seems negative. By applying the Dynkin's formula (Thm. C.10 and Cor. C.11) to µ s log ̂ µ s in the term (III . 4) , we have that the first integral in (C.9) can be expanded as follows

Similarly, applying Dynkin's formula to the following function

with G 0 ( ν, y s n ) = µ s n ( ν, y s n ) log ̂ µ s n ( ν, y s n ) allows us to expand the second integral in (C.9) as below

where

Comparing the first and second order terms in the two expansions of the two integrals in (C.9) above then implies that the term (III . 4) is of at most second order.

C.4 Lemmas and Propositions

In this section, we provide the detailed proofs of the lemmas and propositions omitted in the proof of Thms. 5.4 and 5.5.

Error due to the Intensity Estimation. Apart from the terms (I . 1) and (II . 1) in the proof of Thm. 5.4 and the term (III . 1) in the proof of Thm. 5.5, we also need to bound the error terms (II . 4) in terms of the intensity estimation error, which is given by the following proposition. Notably, the following bound also utilizes the convexity of the loss function and the extrapolation nature of the second step in the θ -Trapezoidal method (C.6).

Proposition C.9. For the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , we have the following error bound:

Proof. We first define and bound three error terms (II . 4 . 1) , (II . 4 . 2) , and (II . 4 . 3) with score estimation error (Assump. 5.3) as follows:

Similarly, we also have

and

The remaining term (II . 4 . 4) = (II . 4) -(II . 4 . 1) -(II . 4 . 2) -(II . 4 . 3) is then given by

for α 1 , α 2 ≥ 0 and α 1 -α 2 = 1 . Therefore, by summing up the terms above, we have

and the proof is complete.

Error Related to the Smoothness of Intensity. Below we first present the Dynkin's formula, which is the most essential tool for the proof of the error related to the smoothness of the intensity.

Theorem C.10 (Dynkin's Formula) . Let ( y t ) t ∈ [0 ,τ ] be the following process:

where N [ µ ](d s, d ν ) is a Poisson random measure with intensity µ of the form µ s ( ν, y s -) . For any f ∈ C 1 ([0 , τ ] × X ) , we define the generator of the process ( y t ) t ∈ [0 ,τ ] as below

Proof. The definition and the form of the generator L , as well as the Dynkin's formula are all wellknown in the literature of jump processes. We refer readers to detailed discussions on these topics in [244].

Here we take X ( t ) = ( t, y t ) , z = ( ν, ξ ) , α ( t, X ( t )) = 0 , σ ( t, X ( t )) = 0 , γ ( t, X ( t -) , z ) = ν 1 0 ≤ ξ ≤ µ t ( ν,y t -) in the statement of Thm. 1.19 in [244] and replace the compensated Poisson random measure ˜ N ( dt, dz ) with the Poisson random measure N (d s, d ν, d ξ ) defined as Rmk. B.3. Then we are allowed to use the ordinary Poisson random measure instead of the compensated one since we are working with a finite measure γ (d ν ) .

From Thm. 1.22 in [244], we have that

and the proof is complete.

In many cases below, we will need the following first-order expansion of the expectation of the function f t ( y t ) by assuming the second-order smoothness of the function f .

Corollary C.11. Suppose that the process ( y t ) t ∈ [0 ,τ ] and the generator L are defined as in Thm. C.10. If we further assume that f ∈ C 2 ([0 , τ ] × X ) , then it holds that

Proof. We expand the function f s ( y s ) from t = 0 as follows

where L 2 is the second-order generator of the process ( y t ) t ∈ [0 ,τ ] defined as follows which is bounded uniformly by a constant based on the assumption on the smoothness of the function f up to the second order and the boundedness of the measure γ (d ν ) . Therefore, the second-order term above is of magnitude O ( t 2 ) , and the proof is complete.

The following lemma provides a general recipe for bounding a combination of errors, which resembles standard analysis performed for numerical quadratures. In fact, the following lemma can be easily proved by Taylor expansion when the process ( y t ) t ∈ [0 ,τ ] is constant, i.e. , y t ≡ y . Cor. C.11 offers an analogous approach to perform the expansion when the process ( y t ) t ∈ [0 ,τ ] is not constant.

Lemma C.12. For any function f ∈ C 2 ([0 , τ ] × X ) and the true backward process ( y t ) t ∈ [0 ,τ ] defined in (2.4), it holds that

Proof. Let L be the generator defined in Thm. C.10. By applying the Dynkin's formula (Thm. C.10 and Cor. C.11) to the function f t ( y t ) and plugging in the expression of the generator L , we have that

which is of the order O ( τ 3 ) by noticing that and the proof is complete.

We remark that in Thm. C.10, Cor. C.11, and Lem. C.12, the smoothness of the function f implies that its derivatives up to the relevant order are bounded by constants independent of the time step τ . This condition is verified in the subsequent proofs.

Then we are ready to bound some of the error terms in the proof of Thm. 5.4 with Lem. C.12.

Corollary C.13. For the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , we have the following error bound: E [(I . 2) + (II . 2)]

Proof. The bound is obtained by applying Lem. C.12 with f being the function

Strictly speaking, f s ( y s ) is actually in the form of f s ( y s -) , but the argument can be easily extended to this case by assuming time continuity of the function f .

Corollary C.14. For the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , we have the following error bound:

Proof. Note that the intermediate process y ∗ s defined in (C.1) is driven by a Poisson random measure that is independent of the Poisson random measure driving the process y s within the interval ( s n , s n +1 ] . Therefore, the error bound is obtained by

Taking the expectation w.r.t. the intermediate process y ∗ s and thus the intermediate intensity ̂ µ ∗ s , and (2) Then applying Lem. C.12 with f being the following function

The result follows directly.

Now we turn to the error term (III . 6) in Thm. 5.5, for which we need the following variant of Lem. C.12.

Lemma C.15. For any function f ∈ C 2 ([0 , τ ] × X ) and the true backward process ( y t ) t ∈ [0 ,τ ] defined in (2.4), it holds that

Proof. The proof is similar to that of Lem. C.12. Specifically, we let L be the generator defined in Thm. C.10, apply the Dynkin's formula (Thm. C.10 and Cor. C.11) to the function f t ( y t ) and plug in the expression of the generator L , which yields

as desired.

Corollary C.16. For the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , we have the following error bound:

Proof. By applying Lem. C.15 with f being the function

we have that the result follows directly.

Corollary C.17. For any n ∈ [0 : N -1] and the corresponding interval ( s n , s n +1 ] , we have the following error bound:

Proof. Following the arguments in the proof of Cor. C.14, the error bound is obtained by first taking the expectation w.r.t. the intermediate process y ∗ s and thus the intermediate intensity ̂ µ ∗ s , and then applying Lem. C.15 with f being the function as desired.

Error involving the Intermediate Process.

Proposition C.18. For the interval ( s n , s n +1 ] with n ∈ [0 : N -1] , we have the following error bound:

Proof. First, we rewrite the error term (II . 3) as

Then we expand the integrand by applying the Dynkin's formula (Thm. C.10 and Cor. C.11) to the function ̂ µ s ( ν ) w.r.t. the intermediate process ( y ∗ s ) s ∈ [ s n ,ρ n ] and the process ( y s ) s ∈ [ s n ,ρ n ] respectively as follows

̂ where the generators L ∗ and L are defined as in (C.12) w.r.t. the processes ( y ∗ s ) s ∈ [ s n ,ρ n ] and ( y s ) s ∈ [ s n ,ρ n ] , respectively, i.e. , for any function f ∈ C 1 ([ s n , ρ n ] × X ) , we have

where we used the assumption on the estimation error (Assump. 5.3) in the last inequality. Then we can further reduce (C.13) to

Corollary C.19. For the interval ( s n , s n +1 ] for n ∈ [0 : N -1] , we have the following error bound:

Proof. Since the two integrands in (II . 5) only differ by replacing ̂ µ ∗ ρ n ( ν ) with ̂ µ ρ n ( ν ) , we have the following upper bound by using the assumption on the boundedness of the intensities (Assump. 5.2 (II))

Applying the same arguments as in Prop. C.18, which uses the generators L and L ∗ defined in (C.14), we can bound the RHS above as follows

̂ where the last inequality follows from (C.15). Substituting (C.17) into (C.16) then yields the desired upper bound.

Proposition C.20. For the interval ( s n , s n +1 ] with n ∈ [0 : N -1] , we have the following error bound:

Proof. First, we observe by Dynkin's formula (Thm. C.10) that and also

∣ ∣ Secondly, applying the given assumption (Assump. 5.2 (II)) on the boundedness of the intensities yields

where the last inequality follows from (C.17) proved above. Therefore, we may further deduce that E [(I . 3)]

where the first inequality is due to the independency of y s and y ∗ s for s ∈ [ s n , ρ n ] , and the proof is complete.

D Details of Numerical Experiments

In Apps. D.1 to D.4, we present additional numerical results for the 15-dimensional toy model, text generation, image generation, and diffusion LLM, respectively.

D.1 15-Dimensional Toy Model

We first derive the closed-form formula of the marginal distributions p t in this model. Recall that the state space X = { 1 , 2 , ..., d } with d = 15 , and the initial distribution is p 0 ∈ ∆ d . The rate matrix at any time is Q = 1 d E -I . By solving (2.1), we see that

and therefore p t converges to the uniform distribution p ∞ = 1 d 1 as t → ∞ . The formula of p t directly yields the scores s t ( x ) = p t p t ( x ) .

During inference, we initialize at the uniform distribution q 0 = p ∞ and run from time 0 to T = 12 . The truncation error of this choice of time horizon is of the magnitude of 10 -12 reflected by D KL ( p T ∥ p ∞ ) , and therefore negligible. The discrete time points form an arithmetic sequence.

̂ We also perform bootstrapping for 1000 times to obtain the 95% confidence interval of the KL divergence, the results are shown by the shaded area in Fig. 2. The fitted lines are obtained by standard linear regression on the log-log scale with the slopes marked beside each line in Fig. 2.

We generate 10 6 samples for each algorithm and use np.bincount to obtain the empirical distribution ̂ q T as the output distribution. Finally, the KL divergence is computed by

D.2 Text Generation

For text generation, we use the small version of RADD [33] checkpoint 1 trained with λ -DCE loss. We choose an early stopping time δ = 10 -3 for a stable numerical simulation. Since RADD is a masked discrete diffusion model, we can freely choose the noise schedule σ ( t ) used in the inference process. We consider the following log-linear noise schedule used in the model training,

where we choose ϵ = 10 -3 .

The score function s θ ( x t , t ) used for computing the transition rate matrix can be computed from the RADD score model p θ using the following formula from [33],

where the model p θ is trained to approximate the conditional distribution of the masked positions given all unmasked positions. More specifically, let d be the length of the sequence and { 1 , 2 , ..., S } be the vocabulary set (not including the mask token). Then given a partially masked sequence x = ( x 1 , ..., x d ) , the model p θ ( x ) outputs a d × S matrix whose ( ℓ, s ) element approximates P X ∼ p data ( x ℓ = s | X UM = x UM ) when x ℓ is mask, and is 1 X ℓ ,s if otherwise. Here, x UM represents the unmasked portion of the sequence x .

We adopt a uniform discretization of the time interval ( δ, 1] . For θ -RK-2 and θ -Trapezoidal, we pick θ = 1 2 . We compare our proposed θ -RK-2 and θ -Trapezoidal with the Euler method, Tweedie τ -leaping, τ -leaping, and we present full results across all NFEs ranging from 16 to 1024 in Tabs. 4 and 5. For each method, we generate 1024 samples and compute the average perplexities and unigram entropy on both GPT-2 large 2 and LLaMA 3. 3 All the experiments are run on a single NVIDIA A100 GPU.

From the results in Tabs. 3 to 5, we observe that θ -Trapezoidal almost outperforms all other approaches and generates samplers with better perplexities across almost all NFEs. We also noticed that both the Euler method and Tweedie τ -leaping perform similarly, and are beaten by a large margin by θ -RK-2 and τ -leaping.

In Tab. 6, we present the percentage of positive extrapolated intensities for different algorithms across NFE values. This partially validates the assumption in our theoretical analysis (Thms. 5.4 and 5.5) that the intensity remains positive throughout the sampling process.

D.3 Image Generation

For the image generation, we use the checkpoint of MaskGIT [63, 68] reproduced in Pytorch 4 . Recall that the MaskGIT is a masked image model which, given a partially masked sequence, outputs the conditional distributions of the masked positions given the unmasked portion, just like the model p θ ( · ) in the aforementioned masked text model, RADD. Therefore, by similarly introducing a time noise schedule σ ( t ) (for which we adopt the same log-linear schedule (D.1) in our experiment), we obtain a masked discrete diffusion model akin to the RADD. The score function can be computed accordingly using the model output as in (D.2).

We choose an early stopping time δ = 10 -3 , and adopt a uniform discretization of the time interval ( δ, 1] for θ -RK-2, θ -Trapezoidal, τ -leaping and the Euler method. For parallel decoding, we use a linear randomization strategy in the re-masking step and an arccos masking scheduler, as recommended in [63]. For each method, we generate 50 k samples in a class-conditioned way and compute its FID against the validation split of ImageNet. We use classifier-free guidance to enhance generation quality and set the guidance strength to w = 3 .

We present the full results for NFE ranging from 4 to 64 in Fig. 5. All the experiments are run on 1 NVIDIA A100. Notably, θ -Trapezoidal with θ = 1 3 is the best-performing method except for extremely low NFE budgets. While θ -Trapezoidal with θ = 1 2 in general demonstrates a less competitive performance, it converges to the same generation quality as θ = 1 3 in the high NFE regime. We also noticed that when using extrapolation with θ = 1 3 , θ -RK-2 beats τ -leaping for NFE larger than 8 , which again accords with our theoretical prediction of its competitive performance in θ ∈ (0 , 1 2 ] regime.

4

8

16

32

64

Number of Function Evaluations (NFE)

6.00

7.71

9.91

12.73

16.35

21.01

27.00

Frechet Inception Distance (FID)

-leaping

FHS

Euler

-RK-2 (

=1/3)

-Trapezoidal (

=1/3)

-Trapezoidal (

=1/2)

16

32

64

6.50

7.29

8.17

9.15

10.26

11.50

Zoom-in

To investigate the robustness of θ -RK-2 with respect to the choice of θ , we also benchmark its performance across multiple choices at NFE 32 and 64 , and we present the results in Fig. 6. We observe that the performance of θ -RK-2 has a flat landscape around the optimal θ choices, which fall in the range [0 . 15 , 0 . 4] . In general, as shown by the curve, the method performs better when extrapolation is used to compute the transition rate matrix, confirming the correctness of our theory (Thm. 5.5) and our discussions. Similar to the behavior of θ -Trapezoidal method in Fig. 4, the performance of θ -RK-2 has a flat landscape around the optimal θ choices, which typically falls in the range [0 . 3 , 0 . 5] . In general, as shown by the curve, both methods exhibit better performances when extrapolations are deployed, which, once again, certifies the validity of our theoretical results.

Finally, we visualize some images generated with θ -Trapezoidal on 6 different classes in Fig. 7. θ -Trapezoidal consistently generates high-fidelity images that are visually similar to the ground truth ones and well aligned with the concept.

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

7.4

7.6

7.8

8.0

8.2

8.4

8.6

8.8

9.0

FID

-RK-2 (NFE = 32)

-RK-2 (NFE = 64)

D.4 Diffusion Large Language Model and Math Reasoning

For the evaluation of diffusion LLMs on math reasoning datasets, we use the checkpoint of LLaDA [64] after instruction tuning, LLaDA-Instruct 8B 5 , one of the state-of-the-art diffusion-based LLMs trained natively based on masked discrete diffusion models [33]. For the math reasoning datasets, we consider GSM8K, 6 a standard benchmark for math reasoning consisting of elementary-school-level word problems. Similar to RADD, since LLaDA is also a masked discrete diffusion model, we can freely choose the noise schedule σ ( t ) used in the inference process. We adopt the same log-linear schedule described in (D.1), and we choose ϵ = 10 -3 . The computation of the score function using LLaDA follows the same procedure as is depicted in App. D.2.

We choose an early stopping time δ = 10 -3 , and adopt a uniform discretization of the time interval ( δ, 1] for θ -Trapezoidal, with θ = 1 2 . For Semi-AR with random remasking or confidence-based remasking strategies, we set the block length to 256 , the same as the generation length, to avoid exploiting LLaDA's inherent preference for auto-regressive generation order and ensure a fair comparison with our proposed method. However, we note that our proposed method can also be generalized to a blockwise sampling setting, and we include additional results here for a proof of concept and to demonstrate that LLaDA indeed prefers a left-to-right generation order. We set the sampling temperature to 0 . 5 for all evaluated inference methods. The evaluation of generated responses follows a standard pipeline with lm-evaluation harness, 7 a standard LLM evaluation kit, following the instruction in the LLaDA repository. 8 Full results are presented in Tab. 7. All experiments are run on 1 NVIDIA A100.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work performed by the authors.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provides the full set of assumptions and a complete (and correct) proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper opens access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets, used in the paper, are properly credited and the license and terms of use are explicitly mentioned and properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new assets introduced in the paper are well documented and the documentation is provided alongside the assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The paper does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces high-order numerical solvers (θ-RK-2 and θ-trapezoidal methods) for discrete diffusion models, aiming to accelerate sampling and inference compared to standard τ-leaping and exact simulation. Theoretical analysis demonstrates second-order convergence in KL divergence under certain conditions. Empirical results on text and image generation show better sample quality for the proposed methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths and Weaknesses:  \n\nStrengths\n- The paper provides a rigorous theoretical analysis, with detailed stochastic integral formulations and error bounds.\n- The introduction of high-order schemes to discrete diffusion inference is novel and addresses an under-explored topic.\n- Empirical validation is thorough, with experiments on both text and image benchmarks.\n- The proposed methods are conceptually simple and easy to incorporate into existing diffusion frameworks.\n\nWeaknesses  \n- The experimental validation is limited: most results are on toy or small-scale models, with no evaluation on large-scale or widely recognized datasets (e.g., Geneval [1], MMMU [2]). The base models are relatively simple, and only basic baselines are considered.\n- Evaluation metrics are primarily FID and NFE, which may not be sufficient to fully support the claimed effectiveness of the methods.\n- Theoretical guarantees depend on strong assumptions (regularity, boundedness, positivity, accurate score estimation) that are difficult to ensure in practical scenarios. The robustness of the proposed methods when these conditions are not met is not addressed.\n\n[1] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment\n\n[2] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI\n\nQUESTIONS:\n- Can the authors provide evidence that their methods remain effective on larger, more complex benchmarks and with a broader set of evaluation metrics?\n- How do the methods perform when the theoretical assumptions are not strictly satisfied? If the authors can address such problem, I am willing to improve my rating.\n\nLIMITATIONS:\nThe main limitations regarding assumptions and empirical scope are discussed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of slow sampling in discrete diffusion models. The authors propose and analyze several high-order sampling algorithms, which are novel extensions of the tau-leaping method. The primary goal is to accelerate the generation process, drawing a parallel to the successful application of high-order solvers for continuous diffusion models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles a conceptually important and underexplored problem. While accelerating sampling for continuous diffusion models is a mature field, this work is one of the first to provide a rigorous exploration of high-order methods for the discrete domain.\n2. The theoretical analysis is a core strength. The proposed high-order variants of the tau-leaping method are well-motivated, and their derivation and convergence analysis are presented with mathematical rigor.\n3. The experiments are comprehensive and sufficiently validate the efficacy of the proposed algorithms. The results clearly demonstrate an acceleration in the sampling process when compared to baseline first-order methods across several tasks.\nWeakness: \nI think the paper misses a direct comparison to conceptually simpler methods like the confidence-ranked remasking technique proposed in LLaDA (Arxiv: 2502.09992). Such methods, while perhaps less theoretically grounded in diffusion theory, are highly practical and serve as a crucial baseline. A thorough analysis, particularly in the important inference scenario where the Number of Function Evaluations (NFE) equals the sequence length, is needed to demonstrate the practical advantages of this work's more complex approach over the strong, simple heuristics in LLaDA. See also my question below.\n\nQUESTIONS:\n1. Regarding Theorem 5.4, the error bound seems to suggest that performance could be further improved by setting the NFE higher than the sequence length (L). Is this a correct interpretation? If so, does it make practical sense to use more sampling steps than the number of tokens to be generated?\n2. A critical point of comparison arises at NFE = L. For a simple masking-based sampler like the one in LLaDA, setting NFE = L (i.e., unmasking one token per step) is analogous to autoregressive decoding. This process should, in principle, allow for exact sampling from the model's conditional distributions, resulting in zero sampling error. In this specific and practical regime: What are the theoretical and empirical advantages of using the proposed high-order methods over such a simple, exact-sampling baseline?\n\nLIMITATIONS:\nA key limitation is the lack of comparison to simpler, heuristic-based sampling algorithms (e.g., LLaDA's remasking). The practical benefits of the proposed high-order methods are not clearly established in regimes where these simpler methods might perform optimally (such as when NFE = Sequence Length), which could limit the adoption of this more complex approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces higher order methods for discrete diffusion models, enabling time stepping schemes with higher order convergence when compared to standard schemes. Explicit examples of higher order methods, such as RK-2 and Trapezoidal are given. Computational examples are provided.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper is well motivated, it is true in the continuous setting that higher order methods have traditionally given a computational gain. As the discrete time diffusion is still continuous in its temporal component, this is a natural extension that should be explored. \n\nThe paper is well layed out and easy to follow. The paper starts with defining discrete diffusion models, shows current stepping methods and presents Theorem 3.1 revealing standard methods are first order. The authors then present their new schemes and Theorem 5.4 and 5.5, natural extensions of Theorem 3.1. \n\nThe paper well states its assumptions, with Assumptions 5.1,5.2 and 5.3.  In particular, it is nice that Assumption 5.2 is directly stated, showing that for their proposed higher order scheme to work, they need 2nd order regularity in the rate matrix. \n\nWeaknesses:\nI do not see any major weaknesses. It would be nice if some standard techniques from higher order schemes were also presented, such as Buchner tables for the construction of these methods if applicable.\n\nQUESTIONS:\nCan the techniques presented here be used to create even higher order methods, such as 3rd or fourth order methods? \n\nCan the authors comment on the influence that the noise schedule has on their method? Usually the constant on the order of a high order scheme has some dependence on the gradient of the function to be integrated. By choice of a clever schedule, would one be able to also help diminish the size of this constant.\n\nDo classical time stepping stability analysis apply? For standard Runge-Kutta methods, we can look at regions of stability, can we do anything similar here?\n\nLIMITATIONS:\nThere is no limitations section of this paper, I would recommend that the authors add this to their submission.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses  computational bottleneck in discrete diffusion models by developing higher-order solvers that significantly reduce the number of function evaluations (NFE) required for sampling. The authors introduce theoretical foundations for higher-order methods in the discrete setting and demonstrate substantial improvements in sampling efficiency for text generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper provides mathematical foundations for higher-order solvers in discrete diffusion models, extending beyond the first-order methods commonly used in practice.\n* The paper demonstrates meaningful reductions in NFE while maintaining or improving generation quality, addressing a key limitation of discrete diffusion models.\n* The paper is well-written, clearly organized, and accessible to readers.\n\n\n**Weakness**\nLimited Scope of Experimental Evaluation\n\nThe paper lacks comprehensive analysis of how different solver configurations perform under varying conditions, limiting understanding of when and why higher-order methods excel. This limits drawing insights and adoption guidelines for readers and broader community \n\n- Currently this work does not compare against several important recent methods and how effective higher order solvers are across these settings.\n- While the appendix mentions predictor-corrector solvers, the authors neither evaluate them experimentally nor discuss how higher-order methods could be integrated within predictor-corrector frameworks. \n- Remasking during sampling shows enhanced generation quality. Would be interesting to see higher order solvers with Remasking like in works like ReMDM\n- Absence of semantic quality metrics: No evaluation using ModernBERT-based MAUVE scores or similar metrics that assess semantic coherence and diversity (entropy)\n- Minimal Performance gain in case of Image Generation (but didn't consider this much for rating)\n\nQUESTIONS:\nFollowing Weakness\n\nIt would be informative for readers and broader community to see integration of higher order solvers with\n- Effect of Temperature/Confidence\n- Multi-token unmasking heuristics like in FastDLLMs, etc.\n- ReMasking\n- Predictor-Corrector Solvers\n- How well higher order solvers work with distilled discrete diffusion models based on e.g., SDTT Checkpoints\n\nAlso additional metrics like use state-of-the-art language models (LLaMA-3 or alternatives) to evaluate generated text quality independently of the diffusion model.\n\n**References** \n\n1. G. Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling (https://arxiv.org/abs/2503.00307)\n2. J. Deschenaux et al. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (https://arxiv.org/html/2410.21035v1)\n3. C. Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (https://arxiv.org/html/2505.22618v1)\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}