Non-Markovian Discrete Diffusion with Causal Language Models
Authors: Yangtian Zhang
Sizhuang He
Daniel Levine
Lawrence Zhao
David Zhang
Syed A Rizvi
Shiyang Zhang
Emanuele Zappala
Rex Ying
David van Dijk
Venue: NeurIPS 2025 poster ()
Keywords: Generative Model
Causal Language Model
Primary area: deep_learning

Abstract: Discrete diffusion models offer a flexible, controllable approach to structured sequence generation, yet they still lag behind causal language models in expressive power. A key limitation lies in their reliance on the Markovian assumption, which restricts each step to condition only on the current state, leading to potential uncorrectable error accumulation.
In this paper, We introduce CaDDi, a discrete diffusion model that conditions on the entire generative trajectory, thereby lifting the Markov constraint and allowing the model to revisit and improve past states. By unifying sequential (causal) and temporal (diffusion) reasoning in a single non‑Markovian transformer, CaDDi also treats standard causal language models as a special case and permits the direct reuse of pretrained LLM weights with no architectural changes. Empirically, CaDDi outperforms state‑of‑the‑art discrete diffusion baselines on natural‑language benchmarks, substantially narrowing the remaining gap to large autoregressive transformers.

Non-Markovian Discrete Diffusion with Causal Language Models
Yangtian Zhang"
Sizhuang He* Daniel Levine
Lawrence Zhao
David Zhang
Syed Asad Rizvi
Shiyang Zhang Emanuele Zappala Rex Ying!
David van Dijk"
Yale University, New Haven, CT, USA
{yangtian. zhang, sizhuang. he, levine, lawrence (eerence.zhaa]$$pe fdavid. zhang, syed.rizvi. shiyang- zhanssaaaaaaaaaeeeeea edu emanuele. zappala@isu.c edu frex.ying david. vandijkk edu
edu
Abstract
Discrete diffusion models offer a flexible, controllable approach to structured sequence generation, yet they still lag behind causal language models in expres- sive power. A key limitation lies in their reliance on the Markovian assumption, which restricts each step to condition only on the current state, leading to potential uncorrectable errоr accumulation. In this paper, we introduce CaDDi (Causal Discrete Diffusion Model), a discrete diffusion model that conditions the entire generative trajectory, thereby lifting the Markov constraint and allowing the model to revisit and improve past states. By unifying sequential (causal) and temporal (diffusion) reasoning in a single non- Markovinn transformer, CaDDi also treats standard causal language models pretrained LLM weights with no architectural changes. Empirically, CaDDi outper- forms state-of-the-ee discrete diffusion baselines natural-language benchmarks, substantially narrowing the remaining gap to large autoregressive transformers.
special case and permits the direct reuse of
Introduction
Autoregressive transformers have become dominant approach for sequence modeling [60 12 56], achieving state-of-the-art performance in many natural language and biological tasks. Their left-to- right decoding paradigm simplifies training via next-token prediction and is supported by large-scale pretraining, unlocking broad linguistic (or domain) knowledge. However, these models can be less flexible for bidirectional partially specified generation, such text infilling prompting from arbitrary locations.
By contrast, discrete diffusion models 17 2, 24 21 naturally accommodate controllable genera- tion scenarios where tokens can be iteratively refined and sampled in a bidirectional manner [47]. Recent advances have extended discrete diffusion to continuous time [5] 48) improved its training objectives 48, 37 44 (66,, and accelerated inference [4] 36). Yet, these models often lag behind autoregressive approaches in generation quality [66], due in part to their reliance on a single latent state for denoising leading to fragile inference where small decoding errors can accumulate over
time [63] 66 32).
To address this limitation, in this work, we explore Man-Maaovv diffusion framework discrete sequences by allowing each denoising step condition the entire generative trajectory, rather than single latent state. While prior work such as DART 23 has investigated non-Markovian diffusion for continuous data, its formulation relies on continuous kernels and smooth transitions that do not directly
"qqual contribution Corresponneecee to: rex. www.yinn@yale..oo edu, david,
edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

translate to the discrete setting. Notably, when instantiated for discrete sequences, non- Mar-vaao diffusion naturally mirrors the structure of token-level autoregressive models highlightigg close connection between the two.
Leveraging this insight, propose CaDDi, causal discrete diffusion model that unifies sequential (left-to-right) and temporal (multi-step) dimensions in a single decoder-only transformer architec- ture. As result, CaDDi can be trained efficiently via simple next-block/nexx-tokee prediction loss-similar to a causal language model-while preserving the bidirectional control and iterative refinement of diffusion.
Additionally, we show that its token-level variation CaDDi-AR can be viewed as a generalization of traditional autoregressive models (T is the special case), making it straightforward to fine-tune pretrained L.IM for discrete diffusion. Such adaptation unlocks flexible generation modes (e.g., text infilling) without sacrificing the rich knowledge encoded by large-scale pretraining.
In summary, our key contributions are follows.
We introduce non-Markovian discrete diffusion framework where each denoising step incorpo- rates the full generative trajectory, improving inference robustness.
We propose CaDDi, a causal discrete diffusion model that unifies sequential and temporal model- ing within Markkovan diffusion framework. Its further variation CaDDi-AR generalizes traditional causal language models as a special case and can seamlessly adopt pretrained L.MMs for discrete diffusion, enabling more controllable and structured generation.
Quantitative results show that CaDDi outperforms recent discrete diffusion models, achiev- ing lower generative perplexity on language datasets and stronger reasoning capabilities when leveraging pretrained LLM.
General Discrete Diffusion
Recently, discrete diffusion models have emerged as powerful framework. In contrast their continuous counterparts, which corrupt data by adding Gaussian noise in real-valued space, discrete diffusion models operate on categorical variables, gradually corrupting tokens before reconstructing them through learned denoising process.
x be sequence of discrete tokens from a vocabulary of size | The forward (noising) process produces latent variables X1, X Xy by iteratively corrupting each token according to a time-dependent transition matrix Q
Forward Process. Let Xg = (x], xi
4x, xx- Cat X Xt-1 Q:).
(1)
where Cat(:; т) denotes the categorical distribution with parameter ग,, and [Qr]i gives the probability of transitioning from state state j time Due the Markovian assumption, the marginal distribution at any timestep can be computed in closed form:
q(x,
x Cat(x;:0 where
Q - - Q2 Q
(2)
Common choices for the transition kernel Q include uniform kernel Q (1 B.)I + 3,11 A /
of size [V] x V and absorbing kernel Q, = (1 - 3c) I B,1e,T Te of size (|V) + 1) x (D 1), where
1 is an all-one vector and e,T represents a one-hot vector where element at index m = V + 1
Reverse Process. The reverse (denoising) model pe (x X²) is commonly parameterized in form of posterior distribution, which is learned reverse the corruption process
Xx X O p& (Xt, Q.-1
Po (x,-1 xx) = g(xx-1 xx, X = 14g (xr, ()) Cat
Xt-1;
(3)
H4 (Xe, t) Qix,
where Mg (xr, = px( x₂) is time-dependent denoiser (xo-parameterization) mapping x, back to the mean distribution of clean data.

---

())
Taken Positioo
(b)-
Large Corpus
Nest token prediction training
Pretrained Causal Language Model
Time
Fly
w/ww
ba
Finetune under CaDDi sbjective
Fositoonal Rataee Endedd
Controllable Generation
Netie welee
Temppool
Via Discruta Diffusion
Mesting!
Figure 1: (a). Inference paradigm for standard causal language model versus CaDDi-AR. In CaDDi-AR, each timestep first autoregressively denoises the tokens into Xo. then re-applies noise via the diffusion kernel to obtain X A traditional autoregressive model emerges as the special case of T 1, which can be adapted to discrete diffusion by fine-tuning. (b). Extending 1D to 2D Rotary Positional Encoding. Standard rotary encodings for token positions are seamlessly generalized to also encode diffusion timesteps, remaining fully bockward-compatien with existing language model architectures.
Error During Inference. During inference, discrete diffusion models can potentially encounter
variety of sources of error, including inaccuracies arising from heavy-tailed per-token sampling dis- tributions, and independent factorization in denoising distribution po (Xo xx) TT,o (x6 xx)
11 (x,; t), which prevents the model from fully capturing the true posterior q (Xo xx) that inher- ently exhibits dependencies across token dimensions [63]
Although each individual source of error may be moderate, they can propagate and accumulate across reverse steps. Specifically, incorrectly predicted x, will not only impact the immediate denoiser output p4g (x,, f), but also affect subsequent predictions, leading further deviations the trajectory progresses. Even when denoiser predictions are accurate, the constrained form of the approximated posterior can limit the model's ability to self-correct earlier mistakes
3
Non-Markovian Discrete Diffusion
3.1 Is the Markovian Assumption Necessary?
Diffusion models can be viewed as special class of hierarchical variational autoencoders (HVAEs) with Markovian assumption, where the latent variables consist of progressively corrupted versions of the original data. Let xo be the original data and x1.7 be the latent variables at timesteps 1.... T. The ELBO objective of HVAE can written as:
T
ELBO max 0,00
Exp (x (11))
log po (x-11 X+TT + (ogpe((x)) - log qol (x1.T xxo)
where 4a(x1; (X1:T Xo) is the variational posterior distribution, and po (x Xxr) ) is the generative (reverse) model. Most of the existing discrete diffusion models make Markovian assumptions
by specifying a fixed, non-leamable forward process 96 (X1:T x,) = I (xx, x(-1), , and
modeling the reverse process as a time-localized transition pe (x,-1 X..T =p (x,
(x,1 xr), which
is approximated by posterior estimator such x(x,-1 computational efficiency, the Markovian assumption compresses all relevant information into single state x;, making the model more susceptible to error accumulation through the generative trajectory, which is shown in prior work [63] and our experiments in Figure 4
X ng (x,, 1)) While beneficial for
In this work, we explore more flexible alternative: treating discrete diffusion models general HVAEs without requiring the Markovian assumption. Inspired by recent studies 23 we con- sider non Markovian generative process, where the reverse model pe (x,-1 X can access the
One illustrative case of such limited self celf-correction is the failure-ton remaskk phenomenon under absorbing diffusion kernels, See Appendix

---

Pe(x;
[xxr)
(9)
X
X1-1
Xo
Algorithm Inference for Non Markkoian Discrete Diffusion
o
gf
Input: Prior distribution g(xx), model parameters 0
g(0y Xx)
Rello [MASKI
Woeld
Output: Sampled data Xo
Initialize Xy g(xy) the noisy input
Two Optinn
(b).
the final timestep
for T down to do
CaDoi
CaDoi-AR
Predict Xo Py (Xo X f)
Xo
Sample X (x,-1 Xb)
Pe (Xa Kzz)
return X
Telt [MASK]]
Figure 2: Illustration of the Non-Markovian discrete diffusion inference algorithm (right) and its forward/reverse process visualization (left).
entire future trajectory X denoise X;-1 for denoising in an autoregressive manner. Assum- ing an independently parameterized forward cortuption process, the reverse model simplifies to: pe (x,-1 Xx.T q (x X pe (X,T) t)), relaxing the structural constraints imposed by the Markovian factorization in both the forward and reverse directions.
In the following, will describe how the non-Markovian discrete diffusion process is constructed. Crucially, we will see that the resulting non-Markovian autoregressive inference mechanism es- sentially aligns with causal language model plus an additional temporal dimension- the groundwork for our unified spatial-temporal framework (Section 4).
3.2 Non-Markovian Forward Process
Following 23 we adopt simple yet expressive independent noising process to ensure that the latent trajectory carry more complementary information across timesteps. Instead of relying on stepwise Markov chain, inject independent noise into the original data xo at each timestep, as illustrated in Fig. 2 Formally, the forward process is defined as:
T
T
(xxor)) = g(X,, X1, X2. .x) = ((x)) II
qx,, (x:tx)) = {(xo) II (x( (x)).
(5)
E=1
1=1
where the final equality follows from our assumption that noise is added independently at each timestep, conditioned only on Xg. While the per-step marginals q (x; Xu=-1) = 9 (x, xx) appear similar to those in Markovian diffusion models, the conditional independence of x, from x,-1 (given Xy ) introduces fundamentally different structure into the forward trajectory.
In this non Non-Markovian formulation, the forward process depends solely on the marginal corruption kernel q (x; Xg), rather than the usual Markovian kernel q (x, x,-1). In practice,
can still leverage standard discrete diffusion kernels-such as the absorbing or uniform kernels-or even mixtures thereof, without enforcing a temporal Markov constraint. Additionally, we can establish the following correspondence:
Proposition 3.1. An abworbing--oaaa
non-Markovian discrete diffasion process with marginal transi- Markovian discrete diffusion process with marginal transition kernel Q; = (1 - -0; ) + of le such that the two pro- cesses exhibit identical mutual information decay between FeT and zu, provided that the coefficients
tion kernel Q = (1 a,) ) I a,le,, admits bijection to an absorbing-stie
satisfy of II_ Oy.. Proofs is provided Appendix A.1
3.3 Non-Markovian Inference Process
Similar to conventional diffusion models, our non-Markovian discrete diffusion model employs posterior-inspired parameterization for the reverse process pe (x,-1 xx) However, due to the
assumption of an independent corruption dernel-whee the forward noise depends only on x we can further simplify the posterior form:
p XxT) qxx-11 Xr., Xo p4 (X.T, t)) 9(xx-1 xo 4gg ()).
(6)

---

Notice that it lift the constraint introduced by XeT in the posterior form and delegating all the flexibility for the denoiser model Hg. As illustrated in Fig. 2 (right), inference proceeds in an autoregressive manner: at each timestep, we first predict a clean estimate Xy ~ μa (XtT: (), then sample x;-1 using the forward corruption kernel q (x,-1 Xo). This loop continues backward through time until the final data sample is recovered.
3.4 Evidence Lower Bound
As with traditional discrete diffusion models, the non-Markovian variant can be trained using a variational objective. Specifically, we optimize the Evidence Lower Bound (ELBO) as:
Liann markov
E. (x. (xo) log Po (Xo
X1:T) KT. (g(xr)
Xo) |p (xx)) Lr
where Cr Di=y Eyxxor xo)
( xo)l (xt-1 Xtrr). The second term is constant for
most forward kernels. For specific kernel choices (e.g. absorbing or uniform), the ELBO simplifies further:
Proposition 3.2. Suppose the non-Markovian diffusion process adopt an absorbing marginal kernel
q (az, zn) = Cat (x,; X Qt), where Q: (1 a+) I ar le and on is increasing function with ag Rs 0 and or xe 1. The ELBO loss in Equation 7 can be further simplified to (see Appendix for derivation):
Labuorh E
on- xx] log pi (Xt:T, ()]
(8)
X1 (x (x0)
The above proposition shows that for absorbing-type forward kernels. training reduces to a weighted cross-entropy between the model's prediction and the ground-truth clean input Xu, with time- dependent weights 04-1 reflecting the degree of corruption.
4 CaDDi: Causal Discrete Diffusion Model
In natural language and image modeling, decoder-only causal models have demonstrated strong perfor- mance autoregressive modeling, due to their efficient parallel training and scalability 42
4
Meanwhile, the reverse process in non-Marke Markovann diffusion models is inherently autoregressive: we decode the entire sequence of latent states Xt.T to retrieve x,-1- Building on this observation, we introduce CaDDi, a causal discrete diffusion model that unifies the sequential dimension (i.e., token order) and the temporal dimension (i.e., discrete diffusion timesteps) within a single Transformer. Specifically, CaDDi employs standard left-to-right architecture while conditioning on multiple timesteps from the diffusion chain, enabling it model Markovoovi discrete diffusion a unified framework.
4.1 Unified Sequential and Temporal Modeling
To accommodate both sequential and temporal dependencies in single model, straightforward choice is to construct a data instance as non- Man-MManovian forward trajectory:
(0)
(L)
(0)
(L)
where the upper index (i) denotes the token position in the original sequence of length and the subscript denotes the diffusion timestep. As illutrated in Fig. 2 block-wise causal mask is then applied, where we can use the logits from the last block (i.e. block corresponding to the position of x,) as the prediction Xn of the denoiser, assuming a Xo parameterization In inference, we do the block-wise autoregression where predicted clean data Xp is used for constructing the next latent variable X,-1-
2D Rotary Positional Encoding. While it's straightforward use decoder-only causal model for modeling the sequential and temporal dependency of latent chains. One of the issue is that modern causal language model typically encodes only the sequential dimension, via rotary positional encodings [53] However, for non- non-Markovian discrete diffusion, we must capture not just the standard token-level sequence but also a temporal dimension corresponding to diffusion timesteps. To address this, extend the original ID rotary scheme to 2D variant.
5

---

Specifically, standard RoPE in modern language models [3] rotates a subset of the query/key dimen- sions according to the token position i. If R (1) denotes the rotation matrix parameterized by i, the attention weight between positions and j becomes (() (R (R)))) where q8)) k( are the query/key vectors, and R ( applies 2D rotations defined by position index i to corresponding pairs of dimensions. To incorporate the timestep i, we introduce an additional rotation along a disjoint
subspace of the embedding. This results in a block-diagonal like rotation matrix:
(6) R
R
0
time
where Racg is the valid part of position- based rotation matrix as before, and R R applies the same rotational principle to separate set of dimensions using the timestep t. By interleaving temporal based rotation in these additional dimensions, it's easy observe that when two tokens share the same timepoints t:
(8) R qi
R
= q: R 0 1
(2)
(R() qi (.))
R (OOR
which means that in the same timepoint the sequential attention pattern is identical that of conventional causal model, and R." reduces to the usual (1D) rotation i.
In practice, feeding all timesteps into the model can be prohibitively large. Thus we have proposed several optional strategies such as latent truncation and trajectory re-composition to compress latent Xx.T into fixed context window size. Details can be found in Appendix
4.2 CaDDi-AR: Factorization over Token Space
independent factorization over dimensions in reverse step make pe (Xt-1 Xx:T) neglecting the token dependence and fail to match the true posterior g (x:-1 X:T).. straightforward solution following (23 is to further factorizing over token space leading to token- level autoregression po (Xt-1 X T Po [x-_1 x X.T) and next-token prediction loss. We denote this varient as CaDDi-AR. Notice it naturally fits our decoder-only causal language model, where the sampling process essentially correpond to autoregressive generation with historical trajectory X.P serving as "prompt". The token-level autoregression decomposing enables CaDDi-AR a more fine-gre -en-igr granularity in per-step generation.
As dicussed in
23
Semi-Speculative Decoding CaDDi-AR usu- ally gives stronger performance compared with vanilla CaDDi. However, naive generation can
Algorithm 2 Semi-Speculative Decoding for CaDDi-AR
1: Input: model parameters 0, prior distribution g(xy)
be much slower as it requires O(Lx XT) function evaluations for sequence of length L over T timesteps. By leveraging the unique properties of decoder-only causal language modeling, we propose a semi-speculative decoding strategy that substantially reduces inference time while maintaining generation quality.
2: Output: Sampled data Xo
3: for T down to do
4: e--0
5:
if sxpee available then
6:
+- VERIFY X Xpeer)
7:
Se.ppre +- CORRECCTTP.-- XET, s
8:
50 Xg
-0:i.prev
Specifically, although causal language models generate tokens sequentially, they can verify the probabilities of any pre-drafted sequence 11:
9:
while < L do
10:
% t pe (Xo Xeer 58
i i
in parallel. Since CaDDi-AR shares the same denoising target X across all timesteps, this suggests a natural procedure: reuse the previ- ous timestep's 's predictions Xgrev as draft for the current timestep (see Algorithm 2). The model then verifies these drafted tokens in parallel, ac- cepting those that meet a specified confidence threshold (e.g., high probability).
12:
Xt-1
~ q (x1-1 xx)
13:
xprev X0
return Xo
Figure 3: Semi-Speculative Decoding with CaDDi- AR: The model verifies all tokens in parallel to identify the first rejection index i, then resumes sampling from that point.
This approach closely resembles speculative decoding
34 g, with one key difference: we not rely on separate, smaller model to propose the draft sequence. Instead, CaDDi-AR's own predictions

---

from the preceding timestep serve as the draft. Like speculative decoding, various verification and correction strategies (e.g., greedy, nucleus sampling) can be employed, ensuring either a comparable or identical sampling distribution while significantly reducing the total number of sampling steps.
4.3 Leveraging Pretrained LLM for Discrete Diffusion
One key observation is that standard causal language modeling can be seen special case of CaDDi-AR under particular settings: specifically, when using a single-step diffusion process (T = 1) without any latent trajectory conditioning as "prompt', i.., pe (xg x =0 2). Furthermore, as shown in Section 4.1 the 2D rotary positional encoding language models, enabling a unified treatment of both sequential and temporal dimensions..
be seamlessly integrated into existing
Given these equivalences, one can take pretrained LLM (trained in a standard causal manner) and further fine-tune it with the CaDDi-AR diffusion objective By allowing the model to condition on historical latent variables I:T. we endow it with iterative denoising and bidirectional modeling capabilities. This straightforward adaptation expands the model's generation modes (e.g., text infilling) while fully leveraging the pretrained language model's existing knowledge.
While several recent works have fine-tuned autoregressive LLMs as discrete diffusion models 66 [2]] our approach preserves the natural causal masking across both temporal and token dimensions. This design choice significantly mitigates the model drift issues observed in prior methods that replaced causal masks with full attention, which often required careful learning rate tuning to stabilize training 64
5
Experiments
Baseline. For most of our benchmark we compare CaDDi against several well-established discrete diffusion models: D3PM (2). SEDD (37), MDLM 44). UDLM [46), and Discrete Flow Matching [11] We adopt the official MDLM and UDLM codebases, which also include reference implemen- tations of D3PM and SEDD. All models use 12-layer Transformer (hidden size 768, 12 attention heads), trained with learning rate of 3e-4, 2500 warm-up steps, and linear learning rate annealing. For evaluation, we sample the same number of sequences across all models. On the Text8 benchmark, we additionally include flow-based methods IAF/SCF [67), Argmax Flow [31. Discrete Flow [58), Any-order Autoregressive Models (30] MAC 49 and Mult. Diffusion BT. We denote our block- level autoregressive variant as CaDDi and the token-level variant as CaDDi-AR. Unless otherwise noted, all models are trained from scratch and experiments by default use absorbing-margee kernels, corresponding to the linear noise schedule of D3PM (2).
Table 1: Evaluation on the LMIB dataset. We report guided Generative Perplexity (PPL) under three pretrained causal language models (GPT-2, Llama-2-7B, and Llama-3.2-3BB different sampling temperatures of each baseline model and CaDDi (T' = 1, T 0.7, T = 0.5). The best performance bolded, and the second-best underlined.
GPT2
LLAMA-2
LLAMA-3
ENTROPY
Model
T=l
T=0.7
T=0.5
T=l
T=0.7
T=0.5
T=1
T=0.7
T=0.5
Data
114.79
22.97
42.20
UDLM
313.24
318.80
328.99
110.37
111.86
119.21
207.91
219.86
231.05
6.0
D3PM
232.37
134.55
133.38
76.00
55.54
59.02
145.96
98.26
110.86
5.7
SEDD
201.19
148.43
81.44
77.54
66.54
46.91
144.23
84,20
60.00
5.6
MDLM
199.45
135.85
106.20
67.86
62.34
60.09
126.35
113.19
104.71
5.6
DFM
182,21
94.46
106.03
67.02
38.93
61.91
120.09
66,22
102.89
5.7
CaDDi
142.51
64.27
45,96
70.27
35.40
23.81
124.51
58.26
36.79
5.7
CaDDi-AR
139.80
76.02
67.59
65.93
35.38
27.76
121.25
59.66
44.54
5.7
One Billion Words Dataset We evaluate CaDDi's generative capabilities on the One Billion Words dataset (LMIB) 7 a large-scale natural language corpus comprising over 30 million English sen-
'Pretrained causal LLM can also be fine-tuned as vanilla CaDDi models, using simple causal bidirectional augmentation trick. See Appendix
7

---

the test set using the standard bits-per tences of varying lengths. We follow the tokenization and training setup introduced in DiffusionBERT 26]. For UDLM, use the pretrained weights provided by the authors. For other baselines, retrain the models using their official codehases, as checkpoints are not publicly available.
To evaluate the quality of model-generater text, report generative perplexity (Gen PPL., computed using several large language models as oracles, including GPT-2 [22.. Llama-2 (7B) [57]. and Llama- 3 (3B) [8]. All oracle models pretrained large-scale natural language corpora. To assess the diversity of generated outputs, we additionally compute the entropy of the generated text set without applying temperature scaling. Further details on these metrics are provided in Appendix
As shown Table CaDDi-AR and CaDDi consistently outperform baselines in generative per- plexity across the three language model oracles. Notably, at low temperature, CaDDi alone performs comparably to-or even better Can--AR We hypothesize that this is due to the long-tail issue being more pronounced in block generation with fewer steps, where lower temperature helps mitigate sampling noise. Importantly, our approach maintains diversity, as indicated by comparable entropy scores.
trained discrete diffusion model short text chunks of length 256 from the text8 dataset. We use the same dataset split as previous studies, training the training set and reporting performance dimension (BPD) metric. The BPD is defined as: - E/ log, p (x,). Note that the bound of log-likeling lileiiiiihood depends on the discretization of the diffusion schedule. To enable fair comparisons, we report the number of discretization steps used for each diffusion- based model.
Text8 Dataset Following prior work 2 48
We omit BPD results for CaDDi-AR due to the additional approximation error of likelihood introduced by its token-level autoregressive decomposition. As shown in Table under the same discretization setting (64 steps), CaDDí outperforms other discrete diffusion models. While any-order autoregressive models be viewed as special case of discrete diffusion with 256 steps, our 64-step model already approaches their performance.
Table 2: Evaluation results on Text8 Dataset reporting layers, steps/discretization, bits-per- dimension (BPD), perplexity, and negative log-likelihood (NLL) for all compared models. For diffusion- and any-order models, the values represent variational upper bounds. Rows shaded in gray correspond to evaluations at 64 steps, enabling direct comparison with CaDDi.
Model
Layers
Steps/Diseretization
BPD
Perplexity
NLL
Autoregressive
IAF/SCF
12
1.88
3.68
1.30
Argmax Flow
12
1.39
2.62
0.96
Discrete Flow
3x8
1.23
2.35
0.85
Autoregressive
12
1.23
2.35
0.85
Any Order Autoregressive
ARDM
12
< <1.43
$2.69
$0.99
MAC
12
< 1..0
<2.64
<0.97
Diffusion
Mult. Diffusion
12
1000
$1.72
<3.29
$1.19
D3PM Absorb
12
0o (continuous)
$1.45
$2.73
$1.01
12
64
$1.51
<2.85
$1.05
SEDD Absorb
12
2 (continuous)
$1.39
$2.62
$0.96
12
64
<1.46
<2.75
31.01
UDLM
12
8 (continuous)
$1.44
$2.71
$1.00
12
64
$1.60
$3.03
$1.11
MDLM
12
< (continuous)
11.40
52.64
<0.97
12
64
$1.46
$2.75
$1.01
CaDDi
12
64
<1.41
<2.66
<0.98
General Reasoning Datasets with Fine-tuned LLM
As shown in Section
4.3
CaDDi-AR retains
compatibility with standard causal language models and can be fine-tuned from pretrained L.MM checkpoints without architectural modifications. To evaluate its general reasoning ability, we fine-

---

Table 3: Accuracy (%) on range of natural language reasoning benchmarks.
Model
Size
ARC-Chal.
ARC-Easy
BoolQ
PIQA
RACE
Social IQA
LAMBADA
QWen2-1.5B
1.SB
33.7
66.1
72.6
75.4
36.6
45.8
63.9
GPT-2
1.5B
28.5
51.1
61.8
70.5
33.1
40.3
44.6
TinyLlama
1.1B
29.9
52.2
59.4
70.3
35.6
39.4
43.2
MDM
1.1B
48.7
62.2
69.5
35.6
41.0
52.7
CaDDi-AR
1.SB
34.2
67.8
71.6
75.4
34.3
46.9
66.3
tune 1.5B QWen model using the CaDDi-AR diffusion objective on range natural language understanding benchmarks, including ARC-Challenge ARC-Easy [14, BoolQ [13), PIQA 4]. RACE 33). Social IQA [45]. and LAMBADA [40]. These tasks span multiple reasoning paradigms, from commonsense inference to multi-hop reading comprehension. We benchmark with model of comparable size, including causal language model QWen2 [ GPT2 442, TinyLlama [65 and
discrete diffusion model MDM 39 trained from scratch.
As shown in Table 3 CaDDi-AR achieves consistent gains over diffusion-based baselines of compa- rable size. It also include + +1.9% improvement on ARC-Challenge and +2.4% gain on LAMBADA over the base QWen model. These results suggest that CaDDi-AR's non-Markovian formulation supports more robust and revisable reasoning, while maintaining compatibility with existing language model infrastructures. It 's worth noting that compared with MDM, CaDDi-AR requires much less training effort.
Conditional Text Generation We also evaluate conditional text generation on the Amazon Polarity dataset [38], which con- sists of 3.6M Amazon reviews labeled as positive or negative. We adapt this task text infilling by prepending a label-based prompt to each review (see Appendix [E.2] for details). We train the conditional gen- erator po (Xo Xe; y) alongside uncondi- tional one po (Xe X2), by preserving cer-
Table 4: Comparison of GPT-2, CaDDi, and CaDDi- CFG on the Amazon Polarity dataset. Generation is conditioned either positive negative.
Model
Condition
Sentiment Accuracy (%)
GPT-2
Positive
73.07
Negative
75.18
CaDDi-CFG
Positive
71.37
Negative
85.42
tain parts of the text fixed.
CaDDi-CFG,
Positive
73.61
Negative
85.92
We measure sentiment accuracy (SA) us- ing fine-tuned DistilBRRT classifier. As shown in Table 4 our approach achieves performance comparable to a fine-tuned GPT-2 on the same dataset while offering more flexible generation (unlike method allows for prompting from arbitrary parts GPT2 only allow prompting from begining, of the text, such as the middle or the title. Examples are shown in Figure [T]. Furthermore, by
applying classifier-fri free guidance (denoted as CaDDi-CFG) (27 46 with different guidance scales 7
we generate reviews that better align with the given prompts
Additional Analysis and Ablation Study. To assess the inference robustness of discrete diffusion models, we perform an additional ablation study by manually inject-
HELLY
Generntive Pempletty
(ue vile)
ing controlled noise (i.e., incorrect predictions) at var-
02-
ious timesteps during generation, simulating potential inference-tine errors. As shown in Figure 4 introduc- ing noise at earlier stages generally leads to greater error accumulation. Nonetheless, our proposed CaDDi frame- work consistently exhibits stronger resilience compared to D3PM and MDLM, maintaining higher generation quality under perturbations across all settings. We remain the full ablation study and discussion in Appendix
Terexins (irection
Figure 4: Generation performance der manually injected noise at different timestep
6 Conclusion
We introduced CaDDi, causal discrete diffusion framework that relaxes the traditional Markovian assumptions in favor of an autoregressive inference process. By explicitly conditioning each denoising step the entire future trajectory, CaDDi captures richer temporal dependencies and leverages

---

iterative refinement. Critically, our approach can also be built atop existing causal language models bridging standard sequenee modeling with powerful diffusion capabilities- while preserving both knowledge from large-scale training and the flexibility of iterative editing.
Acknowledgement
The authors thank collaborators and contributors from across institutions for their invaluable support and insights throughout this project. This work was supported in part by the National Institutes of Health (NIH) grant R35GM143302-01,, the National Science Foundation (NSF) grant IIS Division of Information & Intelligent Systems 2443528, and the Yale Colton Center Award, all awarded to Prof. David van Dijk; and by the National Science Foundation (NSF) grants IIS Division of Information & Intelligent Systems 2403317 and CNS Division of Computer and Network Systems 2431500 as well as the Envisioning Artificial Intelligence at Yale 2025 initiative from the Yale Office of the Provost, awarded to Prof. Rex Ying.
References
[1] Qwen2 technical report. 2024.
[2] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models discrete state-spaces, 2023. URL https://arxi> org/abs/ 2000. 2100003006 03006
[3] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' D'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[4] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourts AAAI Conference on Artificial Intelligence, 2020.
[5] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Prucessing Systems, 35:28266-28279 2022.
[6] Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Large language models to diffusion finetuning, 2025. https:t org/ /brpg///b///........ 15781
[7] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014. URL UR..ttps::// https::// //apxii///axvv..///// wwwpsiiv...gg/aaa ab/p:///ss//31a...eaaa 3005
[8] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.0133/1 2023.
[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020.
[10] Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via discrete mon-markov diffusion models with predetermined transition time. The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[11) Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers, 2019. /ttps//tts///xxxi...oo /ttttppp.//aaxvvvooorr///////// 10509
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
10

---

[13] Christopher Clark, Kenton Lee, Ming- Ming-Weii Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprini arXiv: 10044, 2019.
[14] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv:/033.cm 0845005577 2018.
[15] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and memory-efficient exact attention with 10 1O-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[16] Oscar Davis, Samuel Kessler, Mircea Petrache, Ismail ilkan Ceylan, Michael Bronstein, and Avishek Joey Bose. Fisher flow matching for generative modeling over discrete data, 2024. URL http:: http/vxxvvvv.../.. www. 22405 2405. 14664
[17] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continu- ous diffusion for categorical data. arXiv preprint arXiv:2211.15088 2022.
[18] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
[19] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
URL httpst //attss://zeodee orgeeao..rr///eeeee wrggreecord/////00 12608660
[20] C Gardiner. Stochastic methods: A handbook for the natural and social sciences2009.
[21] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching, 2024. URL https::/arxiv..rr// /br 15595]
[22] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.17891, 2024.
[23] Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai Dart: Demoising autoregressive transformer for scalable text-to-imen -iaate generation, 2024. URL https::tt/s::::: org/ ht/////s/22244 /2100 08159
[24] Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood-basee diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
[25] Sizhuang He, Daniel Levine, Ivan Vrkie, Marco Francesco Bressana, David Zhang, Syed Asad Rizvi, Yangtian Zhang, Emanuele Zappala, and David van Dijk. Calmflow: Volterra flow matching using causal language models, 2024. URL https://///aaa http:///xxivvoo/aaaata www//bag/////eeoooeo/ooooeeeeeee ab/////..221...////// 05292
[26] Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models, 2022. URL https
//arxiv.oo org/ abs/ 2211212211111111 15029
[27] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 1rXiv2200..2559. 2022.
arXiv preprint
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
httph::/t///arrt wwwpsivv.org/ee abs///0..00........ 11239
[29] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. URLhttps /// //tx/xxivvaa wwwxxvioor////.oooo abs/1904 09751
11

---

[30] Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole. Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint 10Xi::210000033,, 2021.
[31] Emiel Hoogeboom, Didrik Nielsen. Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34:12454 12465, 2021.
[32] Vincent Tao Hu and Björn Ommer. [mask] is all you need. arXiv preprint arXiv:2412.06777. 2024.
[33] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1653//1/DD17-1002 URL https://aclantholooe wwceoog..org/ia
org Brgg/D1--10 1082
[34] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[35] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL
wrw.bbs////////ee/oo/o abs/ 2210022200..0 2211222241777777771
[36] Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprint arXiv:2410.0/94/ 2024.
[37] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete difffusion modeling by estimating the ratios of the data distribution, 2024.
16834
[38] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 162,172,111 2013.
[39] Shen Nie, Fenggi Zhu, Chao Du Tianyu Pang. Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text, 2025. URL htttps org/ 2b1/ 18514
[40] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoe Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http://www aclweb.
[41] Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. arXiv preprint arXiv:2410.07761. 2024.
[42] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[43] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[44] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.075524, 2024.
[45] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Com- monsense reasoning about social interactions, 2019. URL. ltttt:t////// httpth:ttt///a:: //ta:////.....e/ htt//axvvvvrraaaaaa/// www./oo//////.../.o. 09728
12

---

[46] Yair Schiff. Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla- torre, Bernardo de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412. 101933 2024.
[47] Tianxiao Shen, Hao Peng, Ruoqi Shen, Yao Fu, Zaid Harchaoui, and Yejin Choi. Film: Fill-in language models for any-order generation. arXiv preprint arXiv:2310.09930 2023.
[48] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406. 2e:::04099,,9, 2024.
[49] Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregres- sive models the right way. Advances in Neural Information Processing Systems, 35:2762-2775. 2022.
[50] Yang Song, Jascha Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
URL StK.itttp://arxivorrr. hbtrg/abs/2011111 13456
[51] Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings, 2024. URL htttps://arxiy, org/ abs/
2402. 15449
[52] Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design, 2024. URL https:/ss//aaxxi org/ /402 05841
[53] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[54] Jaesung Tae, Hamish Ivison, Sachin Kumar, and Arman Cohan. Tess A large-scale generalist diffusion language model, 2025. /ttps:////arx wwwpxiiv.rr//eee abs/ 2502 13917
[55] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative mod- els with minibatch optimal transport, 2024.
00482
(56) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.. 13971, 2023.
[57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. http::t:s:
org/ abs abs///300 2307 099288
[58] Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. Advances in Neural Information Processing Systems, 32, 2019.
[59] Nicolaas Godfried Van Kampen. Stochastic processes in physics and chemistry, volume 1. Elsevier, 1992.
[60] Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
13

---

[61] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00:07, 2025.
[62] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprini arXiv:21 2021.
[63] Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXiv:2410.21357, 2024.
[64] Jiacheng Ye. Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025, URL https://hkunll gi thub 10 http/blog//22// dream
[65] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024. URL htttt:://///iiiiii org. 2401. 02385
[66] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908, 2024.
[67] Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pages 7673-7682, PMLR, 2019.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims accuraltely reflects the paper's contributions and scope and are well supported by the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The paper discusses its limitations, especially on inference speeds.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical claims are supported with justifications and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experiment details are discussed in great details.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper doesn't 't release its code at this time but great details on experiment setups are provided and will release its code at an appropriate time.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc/public/eee.
www.ded/eeooeeuummmmccoo ssi ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: Experiment details specified. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiment compute resourses are listed.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics lattp::/neurips:
Answer: [Yes]
Justification: This research conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Social impacts are discussed in this paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Owners of assets used are properly credited in the paper and any license is respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pataasess.paperswtt paperswi thcode. cow/coo////ooooo /atasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs not involved as core component of the formulation of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.M for what should or should not be described.
21

---

A Proof
Throughout this appendix we work with a single-token random variable that takes values in the augmented vocabulary V4 == VU [MASK]]] }. The distinguished absorbing symbol is denoted by m E V+ and we write Cm for the corresponding standard basis vector.
Let Xo ~ be drawn from the empirical data distribution, which assigns zero probability to m. Fix a diffusion horizon T E N.
A.1 Proof of Proposition 3.1
An absorbing--a state non-Markovian discrete diffusion process with marginal transition kernel Q: [I-a,)I+aaalll 0, admits bijection abxorbing-state Markovian discrete diffusion process with marginal transition kernel Q; (1 of [ + a 1e such that the two processes exhibit identical mutulliinformation decay between xx:T and Xo. provided that the coefficients satisfy of II Qr.
We begin by formalising the two forward corruption processes that will be compared.
A.1.1 Forward-process definitions
Definition A.1 (Markovian absorbing diffusion). Let B = (B1..... masking probabilities. The Markov chain (x,) is specified by
,3r) C [0, 1)" be the single-step
X₂ X--1 ~ (1 - 3,)6x, + 3,8m:
I....,T.
(10)
Because m is absorbing, the cumulative masking probability after steps is
a Pr(x: m xo) = II
Ba),
1= I. T
(11)
s=1
which yields the marginal transition kernel
(1-a)]+ ae
Definition A.2 Markovan absorbing diffusion). For noise schedule a = (01,...,,r)) C [0, 1|"" we generate the forward trajectory by
Lid.
X,
Xo
~ (1 0x0 + a.55m
1=,....,7
(12)
Each x, is obtained by independently masking xo with probability as- The associated marginal transition kernel
NM (1 t-aa)I ar le
I t= I.
A.1.2 Mutual-information computations and decay measure
For any pair of random variables (U, V) we recall that [UU; V) H (xx) Σvev (o(v) log (c) is finite and strictly positive.
VV). The entropy
To make comparisons between Markovian and non-Marte Mar-aaaoviann diffusion processes more interpretable, we introduce the normalized mutual information decay following 2
7 Xx); (Xo, xx) (x+)
D, =
(13)
H (xo)
H (xo)
which quantifies the proportion of information about that is lost after corruption step t.
Lemma A.3 (Markovian suffix information). For the process of Definition A. and every t (1....,T) T}.
IM (XeT3X)) = H (xo) (1 -at), D. oj
(14)
Here give two proofs:
22

---

Proof (Direct Expansion). Because the chain is absorbing, the suffix X,.r is deterministic function of the single variable x,; hence
[u(x)T;x0) xxxrxxo).
There are two mutually exclusive outcomes:
(i) Un masked: with probability 1 1- a- the token is uncorrupted and X Xo-
(ii) Masked: with probability oz the token is replaced by the absorbing symbol Cm, which is independent of Xp-
Writing (o( Pr(xo- = v)., the joint pmf is
(1 - (1-af)po(1)
== u em:
p(xo- x, = w)
at po(u),
Em
0,
otherwise.
The corresponding marginal of X2 is
(1 - al)aa)
u # cm
p(x: = u)
= cm
Direct expansion mutual information.
Ep(u, p(u, v) log p(u, v)
T(x;xx) =
p(u)
0,8
Only two cases have non-zere probability:
Case u = v & em- Here p(u, v) = (1 - oj (v) and p(u) = (1 - ai (v)), so
p(u, v) log
(1 - oz ai)
log p(u)p(v) (1 a (V(()) Po(u) - (o
The total contribution
Σ (1-0))p((1)) -og log) = = -1-a;) H p
(ii) Case Cm- In this case p(u, v) appo(o) and p(u) a;; the logarithmic term vanishes, so the contribution is zero.
Hence
1xx;xx0 -(1-o))((xxo)
D.M 1 Xo) (xx))
= - -1 ai) a7.
Since 1())rT))) Xx); the stated identities follow.
Proof 2 (Conditional Entropy). Condition on the observed token
i If Xx Cm (probability af), masking reveals nothing, ( Xx =m) em) = H (xx).
i If U (probability 1 a;),, we know x exactly, hence H X, v) = 0.
23

---

Averaging,
Hxxo xx) ot H(xo) (1 of)) 0= oz; (xx).
Therefore
DM = 1(x;;
X0x = H(xo) - Hxx xt) = - -1,) H(xo);
Ot-
Hxx))
Because X. is a deterministic function of Xt, the same formula holds for (xxx:T;X0).
Lemma A.4 (Non-Markovian suffix information). For the process of Definition A.2 and every t t [I.....T)
T
INM (XuT: = H (xo)
1
II
D.M
II
(15)
r=t
Proof. Define the indicator Z := 1(x, 6m for every t, ,,..., TT. Conditioned Xo, each coordinate is masked independently with probability ar, 8o
T
Pr(Z =)
II ary T=t
Pr(Z = 0) =1 II ar.
r=t
The event Z depends only the masking coins, hence is independent of the value of Xo-
For conditional entropy (xx X:T)).
(i) If Z (all tokens masked, probability 11 the suffix reveals nothing, (xo x.T) H (xx).
(ii) If Z = 0 (at least one un-masked coordinate, probability 1- II o+), every un-masked coordinate equals Xo; thus Xy is known exactly and H(xo Xx:r) 0.
Averaging over Z.
T [I
H(xo Xx.T =
H(xo).
rmt
Using I(U:V) = H(U) - HQU V),
INM(XIIT; H(xx) H(x xxr) = H(xo) (160)
II ar
NM Ix
T
H(xo)
r=t
These results show that both the Markovian and non-Markovian processes exhibit the same normalized mutual information decay their cumulative masking curves satisfy:
of
II
dr.
(16)
Tst
Since aj and 3r have correspondence as shown in Equation
11
casily further derive:

---

Proposition A.S (Equivalence in mutual information decay), Let C. 10, 1]" be non-Markovian schedule and define the effective cumulative schedule oj II Cr. Then there exists Markovian schedule B [0, 1]" defined by
aj
B. :=
og = O,
(17)
07-1
such that:
DNM D. and INM (Xt:;; xo) = TM (KrT:Xo),
(18)
for all t = 1...., T. This ensures equivalence in both absolute and relative information loss.
Matching the linear marginal used in prior work. Most existing studies absorbing-stee state Markovian diffusions
2 37
adopt the linear cumulative sumulativemmasking curve a lin,* t/T,, t
O,...,T T. For a fair comparison by default employ non-Marke Markovaov independent schedule a lin that reproduces exactly the marginals in experiments, i.e. satisfies ag 11 a lin Using the backward recursion aa = a ////a411 (Sec. A. gives the closed-form
lin,+
t+1 1=1,... -
lin OT
A.2 Derivation of the Non-Markovian Evidence Lower Bound (ELBO)
We derive variational lower bound the marginal log-likelihood of the observed data Xo under the non-Markovian discrete diffusion model.
We start with the marginal likelihood of the observed data:
Po (Xo, X1:T) q (XI:T xo) dxci:
log pe (Xo) log pe (xo, x1:T) dx1:7 log
q (X1:T Xo)
Applying Jensen's inequality yields the Evidence Lower Bound (ELBO):
Po (Xo. X1:T) (X1.T Xx)
log (xo) 2 (xxx (xo)
log
L mon-markov
In the non-Markovian case, the joint distribution the reverse generative process is:
T
Po (Xo, X1:T)= Po (xr)
II po (xt-1 XEr)
where Xy is generated at the final step. The approximate posterior, due to the independent corruption assumption, factorizes as:
(XIT xo)
II a (x, xo)
Finally, we have the ELBO expansion:
Po (xr)
log po po(xi- 1x Xx.)
Laon-markov = (y(xx::1/x))
log (xT xo)
+ log ((ir)
q(x (xx+11 xo)
Exgxs r
log po (X X1:T)
KL (g(Xr xo) po (x(r))
Reconstruction
Prior KL
(19)
(x( KL (g(x,-1 Xp) l| po xx.))
25

---

We can denote the accumulated reverse KL terms as:
T
CT
E (x)r) (x)) KL (x4-1 xp) Ilpo (xx-1 X,T)
(20)
With this notation, the non-Markovian ELBO simplifies to Equation 7
maen-markov (x (xo) log pa
(xo x1.T) KLL (g (XT Xo) ||pe (XT)) Cr
(21)
Note that the second term, corresponding to the prior KL, is constant for most diffusion kernels and be omitted during training.
A.3 Proof of Proposition 3.2
Suppose the non-Markovian diffuusin process adopt an absorbing marginal kernel (xt xo)) Cat (x;; X Q.), where Q2 = (1 (4) I + 04 oglem and os is increasing function with og Rs 0 and
OT 1. The ELBO loss in Equation can be further simplified to:
Labort = X1.r X1.T Xx)
Ela--1x0 log μo (XET, t)]
Proof. Suppose the non-Markovian diffusion process adopts an absorbing marginal kernel Q₂ (1-a,) I a,le m. we have
X Cy
q (xx(x))=
=
1
0z
X X otherwise.
(22)
0
We now evaluate the per-step KL term in the 19
KI. (g(x;-1 Xo) (x XxT)) = KI. (x xo) [1x Xe+1, Mo01,M ()))
g(xt-1 = 6m Xo)
= (x, em Xo) log g(x1-1 Em Mo(Xe.T 1))
g(x; ex x()
Σ g(x-11 Ex xx0 log
q(x1-1 = eb 1o (Xt:T 1))
k-mm
x
x log
[xo(X t)] ex
= -t-1 x log ne (X):,,1))
(23)
Next, consider the reconstruction term:
E Ex(x (o)) log po (Xo X1.T) = Ex xo) X log μo (xI:7 1) (x(x)rix))000 axxx log me (XI:T, 1)
(24)
Finally, the prior KL term vanishes:
KL (g (Xy xx) lpp (xx) KL (dxer em |x em
0
(25)
Putting all components together, obtain the full ELBO:
T
Labsorh Exi (xa)
Oa- x log pa (Xe.T, ()]
(26)
26

---

A.4 Failure to Remask Issue
In Markovian discrete diffusion models with an absorbing kernel defined as Q2 = (1 I+)++
the corresponding marginal kernel is given by Q2 = (1 af) + of ee where of
IT.= (1 - -3,,, Using the closed-form posterior Equation 3 we obtain:
Cat (x)-1:Xx)
X3 7 Cm
q (x1-1 Xo = (xo())))
Xt-li oi- Ii-1e -a; (20())))
(27)
Cat
X Em
This expression highlights an issue: once a token is unmasked (i.c., X2 o em ), is deterministically copied to X regardless of the model prediction po (x,). As result, the model is unable to revise early mistakes-an issue we refer to as failure to remask.
Non-Markovee discrete diffusion formulation defines the posterior as (x:-1 μ4 (X=r, f)), remov- ing the Markovian constraint and allowing more flexible transitions. By appropriately integrating inductiva bias into the design of ua(e.g. latent truncation described in Section C.I.I. the model is able to revisit and potentially forget earlier errors during generation.
B Related Work
generate data by learning reverse (denoising) process invert fixed forward (noising) Markov chain. Austin et al. 2 first extended such models to discrete data (D3PM) by defining uniform and absorbing diffusion kernels on finite state spaces. Subsequent work introduced improved parameterizations, such as data distribution ratio estimation [37), drawing parallels with score matching (50). Despite their efficacy, these methods typically rely on Markov chain, focusing on denoising from a single noisy state X₁. By contrast, our approach breaks the Markovian assumption and conditions on the entire future trajectory X providing more robust denoising and broader generative capabilities.
Discrete Diffusion and Discrete Flow Matching. Diffusion models 28
Flow matching 335 551 learns a continuous transformation from noise to data via an ODE governed by vector field. Recent extensions handle discrete data [21 16 52). While these methods circum- vent explicit Markovian noising, they often require continuous flow formulations and specialized training objectives. In contrast, our non-Markovian discrete diffusion remains within the discrete diffusion paradigm, retains a straightforward variational objective, and integrates naturally with causal modeling.
Autoregressive Models. Autoregressive Transformers (60, 12 56) have become foundational in language modeling, producing tokens sequentially conditioned on preceding context. While highly ef- fective for unidirectional, left-to-right generation, they often struggle with tasks requiring intermediate modification or bidirectional reasoning. Within our framework, CaDDi-AR represents a specialized variant that integrates causal (autoregressive) decoding with diffusion- based iterative denoising. This hybrid design enables CaDDi-AR to combine the strengths both paradigms- efficient left-to-right token generation and flexible multi-step refinement.
Integrating Autoregression with Diffusson and Flow Matching. Several works [23 25 try to combine difffusion or flow matching with causal transformers for improved generation. Specifically, DART [23 employs non-Markovian trajectory to let transformer model entire sequences of diffusion states. Our approach further refines this idea in two ways: (i) we focus on discrete non- Markovian diffusion with explicit multi-step stepi-step conditioning, and (ii) provide direct path for adapting pretrained LLMs, thus combining the strengths of aarge-scale language model pretraining with the controllability of discrete diffusion.
Non-Markovian Reverse Process in Physical System. Using a Non-Markov reverse process to recover the distribution introduced by Markovian forward process is not a new idea. physics, many systems exhibit this property. Langevin Dynamics: Although the forward motion of a Brownian particle (with velocity and position) can be Markovian in the full state space, attempts to reverse the position-only dynamics often require the history of the system to account for friction random kicks
27

---

Quantum Processes: Tracing out environmental degrees of freedom can yield a Markovian forward evolution, but reconstructing the entire global state upon reversal introduces non- aa-Maaaoviian memory effects.
20 [59.
Non-Markovian Discrete Diffusion. Relaxing the Markovian assumption enables a more flexible and expressive diffusion framework. Prior work, DNDM [10), focuses on accelerating inference by introducing a predetermined transition time set, enabling a training-free sampling algorithm that significantly reduces the number of function evaluations. This efficiency gain is achieved by breaking the Markovian constraint. Concurrently, ReMDM [6]] proposes non-Markovian discrete diffusion process that allows remasking of previously generated tokens during inference, enabling iterative refinement. In contrast, CaDDi introduces a more general non-Markovian discrete diffusion framework that explicitly models the entire generation trajectory.
C
Model Implementation Details
C.1 Context Window and Latent Compression
practical limitation when using causal language models within our non-Markovian discrete diffusion framework arises from their inherently bounded context windows. Traditional causal transformers can process sequential inputs effectively only up to a fixed length, which restricts the number of latent timesteps that can be accommodated-denoeed here as m. However, non-Markovian diffusion processes often require conditioning on extensive historical trajectories, frequently exceeding this limit.
To address this constraint, we introduce a general latent compression operator, (X1:T), which maps the full latent sequence X1:r to a compressed form that fits within the model's '' context window. Accordingly, the reverse denoising function expressed as:
H4g (X+r, t) := 14g (I' (Xer),
(28)
C.1.1 Latent Truncation
The simplest yet effective instantiation of T is latent truncation, defined
I Trnne (xx.) == FlattenTimm (Xe, Xi+1. ,,,mx-........mm)))))
(29)
where FlattenTime denotes the operation of temporally flattening the sequence into one- dimensional input trajectory suitable for the causal language model. For a model with a context window limit of m, this strategy retains only fixed-length segment from the most recent m timesteps, discarding carlier latents.
Such selective truncation inherently emphasizes the most recent- and typically more informa- tive-states states, which are often less affected by accumulated inference noise. Notably, by intentionally discarding earlier latents, the model is allowed to "forget" potentially noisy or erroneous history, an idea related to selective re-masking approaches in diffusion modeling [61]. This forgetfulness acts implicit regularization mechanism, helping the model focus on more stable and relevant information while reducing the risk of propagating stale or corrupted context during inference.
C.1.2 Trajectory Re-composition
While latent truncation is efficient, it can discard valuable long-range information beyond timestep t ++ m. To mitigate this, we introduce trajectory re-composition, complementary compression strategy that integrates information across the full latent sequence before applying truncation. This method first aggregates latent information through a sequential integration operation and then retains only the most recent m composite states:
X X ①
XTT
(30)
Fre (XLT) == Flatten Time (x,)
Here, @ denotes an wlement-wise integration operator. For masked- like diffusion models, this operator replaces each element with the most recently unmasked token when applicable. This re-composition process enables earlier timesteps to incorporate contextual signals from future states, effectively compressing the trajectory into a form suitable for the limited context window without losing essential long-range dependencies.
28

---

110.00
M/00
Birds MASK
Birds
MASK
Birds
fly
81da
CaDDi
Rack
MASK
fly over
MASK
MA2
(b) CaDDi generation block-level causal mask
(a) Block-level causal attention mask
Figure 5: Illustration of vanilla block-wise generation of CaDDi. Figure Sajshow the attention mask of vanilla block-wise generation. The block-level causal mask allows bidirectional attention within each time point and causal attention over the time points. Figure [5b the generation scheme. Note that the model itself predicts the clean data Zg practice but the figure highlights the next sampled time point (in color) for clarity.
Birds fly over MASK
Birds fly over MASK Birds fly over MASK
Figure 6: Illustration of causal bidirectional augmentation. On the left, direct token-level causal attention applied to the original sequence, where each token attends only to preceding tokens. On the right, the sequence is repeated, allowing tokens in the second copy to attend to their counterparts the first, thereby approximating bidirectional attention with token-level causal attention.
C.2 Causal Bidirectional Augmentation
Conventional discrete diffusion models often employ transformers with bidirectional attention across the sequence dimension. In the context of non-Markovian discrete diffusion, natural extension of this framework is to incorporate block-wise causal attention mask, as illustrated in Figure This block-level causal mask enables bidirectional attention within single timepoint and enforces causal dependencies across different timepoints.
However, unlike loken-level causal attention (CaDDi-AR), block-wisc causal attention of CaDDi is not readily compatible with standard pretrained causal language models, which adopt token-level causal mask and next-token prediction scheme. Moreover, it involves irregular attention pattern, it cannot take advantage of existing infrastructure-ssut as efficient implementations like FlashAttention [15].
For CaDDi, our goal to enable one-shot block-level generation while preserving bidirectional modeling capabilities. Directly applying token-level causal mask to predict the next block in one shot would eliminate bidirectional modeling, as the final timepoint's tokens in the context window would lack visibility into subsequent tokens (see Figure So here propose an alternative causal bidirectional augmentation, which involves repeating the final timepoint in the context window. As shown in Figure [b] this approach preserves compatibility with the standard token-level causal mask and supports block-wise generation with bidirectional modeling. Our method also bears resemblance prior work [51] repetition-basee improvements in language modeling.
C.3 CaDDi-AR Token-Level Factorization
As discussed in Section 4.2 CaDDi-AR models token-level dependencies within each denoising step by adopting autoregressive factorization over the token dimension. This approach builds prior work such as DART 23]. which proposed boken-wise autoregression for improving expressiveness in discrete diffusion.
Formally, instead of modeling the full sequence distribution in a single step, we decompose the reverse process
29

---

Binds
HASK
MASK
Birds
MASI
Binde
1443
+1
CaDDi
3011
+1
MASK MASK MASK MASK
Binda
MASK
over
MASK
Birds
my
over
MASK
Blrds
over
MASK
(b) CaDDi Generation with causal bidirectional augmentation
MADC
(a) Token Level causal attention mask
Figure Illustration of causal bidirectional augmentation. The last time point is repeated and highlighted in bold to approximate bidirectional attention within a causal framework. Figure 7a shows the token-level causal attention mask, identical the standard mask used in causal language models. Figure 7b illustrates the generation process: for the token at position i, the model attends to all tokens the generative trajectory and predicts the token at position (i + 1) in Xo, consistent with the autoregressive nature of causal language models. Both the attention mask and generation process are fully compatible with causal architectures, making CaDDi a natural extension. Note that, while the model predicts the clean data 2o in practice; the next time point is shown gray here for illustrative simplicity.
Py (xt-1 Xc.T) = II
pa (xf-1 x xx
(31)
i=1
where x
This sequential factorization enables more accurate modeling of intrastep token dependencies compared to fully factorized approaches.
(x--1...... x
In our formulation, we further adopt the no-parammttrization, predicting the clean sequence autore- gressively and applying the forward corruption kernel to reconstruct X;-1- This leads the following form:
L pe (xÉ
po (x5 X(.T) II
x Nc:T)
(32)
where denotes the clean target sequence. This structure aligns well with standard causal language models, allowing autoregressive decoding over tokens while conditioning on the latent trajectory X;T static prompt.
In implementation, CaDDi-AR is trained using standard left-to-right causal masking, with the latent variables provided as additional context. During inference, the model samples X autoregressively at each timestep and applies the corruption kernel obtain x,-1. To reduce computational cost, we employ semi-speculative decoding (Section 4.2) by partially reusing predictions from previous steps and verifying them in parallel.
This token-level decoding strategy provides finer granularity for generation, and is particularly effective tasks requiring fluent, coherent text output or strong local consistency.
E
Experiment Details
E.1 LM1B Dataset Experiment Details
Dataset Preprocessing. We follow the preprocessing setup introduced DiffusionBERT (26), using the One Billion Word Benchmark [7]. Sentences are tokenized using the tokenizer with a vocabulary size of 30,522. All sequences are padded or truncated to fixed length of 128 tokens during training.
30

---

Model Configuration. All models are based on a 12-layer Transformer decoder architecture with hidden size of 768 and 12 attention heads. For D3PM (2). MDLM 44 and SEDD [37], we adopt an absorbing diffusion kernel with log-linear noise schedule. For CaDDi and CaDDi-AR, we use the absorbing- state forward kernel described Section A.55 with total diffusion steps set to 64. Note that models such as MDLM and SEDD are trained in continuous time, whereas our models operate in discrete time. CaDDi uses context window of 5 and applies latent truncation as described in Section C.I.I In this experiment, CaDDi-AR is trained entirely from scratch without leveraging any pretrained language model weights.
Training Details. Models are trained using Adam W with learning rate of 3e-4, 2500 warm-up steps. All models use batch size of 512 and train for 1000K steps. Our models are trained on NVIDIA H100 GPUs with mixed precision.
Inference and Sampling. We evaluate generative perplexity using pretrained oracle models (GPT- 2. LLaMA-2-7B, and LLaMA-3-3B) under three sampling temperatures: 7 = 1.0, T = 0.7, and T 0.5. For each evaluation, sequences are generated according the length distribution of the dataset, and average perplexity is computed under the oracle model. All diffusion befffusion-based models use 64 denoising steps during inference. For CaDDi-AR, we additionally explore semi-speculative decoding to reduce sampling latency without sacrificing quality.
Evaluation Metrics. We report oracle-based generative perplexity (Gen PPL) our primary metric for evaluating generation quality. Gen PPL is computed using a separate, pretrained causal language model, which assesses how well can predict the next token in the generated sequence. Intuitively, lower perplexity indicates that the oracle model finds the generated text more coherent and predictable given the context-implying better fluency and consistency.
To mitigate known issues with perplexity, such as its tendency to reward repetitive or degenerate outputs [29], we adopt guided generative perplexity. Specifically, prepend natural language prompt-Does the following sentence make sense: -to each sequence before evalua- tion. This encourages the oracle model to assess coherence in more human-aligned fashion, reducing the risk of falsely low perplexity on poor-quality outputs.
In addition to quality, assess diversity using token-level entropy over the generated output distribution without applying temperature scaling. This captures how varied the model's outputs are across generations.
E.2 Amazon Polarity Conditional Generation Details
Dataset and Preprocessing. We use the Amazon Polarity dataset 38). large-scale binary senti- ment classification corpus consisting of approximately 3.6 million product reviews labeled as either positive or negative. All reviews are tokenized using the sert-baae-uncaaee truncated to a maximum length of 128 tokens. To enable conditional generation, we prepend a natural language sentiment prompt each review. This prompt takes the form of simple prefix indicating the intended sentiment (e.g., positive negative), allowing standard causal language moodels-sucee as GPT-2 to generate sentiment-alige allignd outputs. Example formatted inputs are shown below:
tokenizer and
This is positive review: Title: Great!!
Content: "This product is amazing! The quality exceeded my expectations and will definitely buy again."
This is negative review:
Title: Very disappointing
Content: "It broke after a week and customer service was not helpful."
Conditional Generation Setup. We adapt our model to perform sentiment controlle generation by training unified denoising network (c, x.T), where denotes a conditioning input (e.g., a sentiment label prompt) and X,.T is the observed noisy trajectory. During training, we simulate both conditional and unconditional modes by randomly masking the conditioning input c. This enables the model to learn both pe (c, X;:) and μN (xx.) within single parameterization.
31

---

At inference time, we generate sentiment antiment-aligned text using either direct conditioning (i.e., sampling from q(x;-1 x,))) or classifier-free guidance. the latter case, we apply the reweighted sampling distribution q( x noo (c, Xt.r)) as defined in Equation 33 which balances conditional and unconditional predictions using guidance scale ".. This allows finer control over sentiment alignment during generation.
Massifier-Free Guidance. We extend classifier-free free guidance (CFG) to the unsupervised discrete diffusion setting by reweighting the sampling distribution at each reverse step. Specifically, we define a guided transition distribution over x as:
9 (xx-1 uo (c, xx:T))
i (x4-1 po(c, XeT)) x
(33)
q (x p (x(x)))))) 7--1
where с denotes a conditioning signal (e.g., a sentiment label or prompt), and (o(+)) is the denoising network predicting the clean input X₂ from the noisy trajectory x,.7 The numerator corresponds to the conditional denoising distribution, while the denominator represents the unconditional variant, where the conditioning input c is masked out. This formulation smoothly interpolates between conditional and unconditional behavior, analogous to CFG in continuous diffusion.
The guidance scale 7 2 controls the strength of conditioning: when + the model performs standard conditional generation; as increases, the model places more emphasis on aligning with the conditioning signal, potentially improving controllability at the cost of diversity or fluency. In practice, we find that moderate values such as = 1.0 or 1.25 strike good balance between alignment and generation quality.
To enable this guidance, we train the denoising model jointly on both conditional and unconditional modes by randomly masking the conditioning input during training. This allows the model to learn both behaviors within a single parameterization.
Evaluation. To evaluate sentiment alignment, we use publicly available DistilBERT classifier fine-tuned on the Amazon Polarity datasef] For each sentiment label, we generate 1,000 samples using top-k sampling with k 50 and temperature T = 1.0. across 64 denoising steps. Sentiment accuracy is computed as the percentage of generated samples whose predicted label matches the intended conditioning prompt. Results reported in Table 4
For the GPT-2 baseline, we condition generation on the same prepended prompt used in our model and apply the same top-k sampling and temperature settings for consistency.
E.3 Text8 Dataset Experiment Details
Dataset Preprocessing. We use the standard Text8 dataset, a 100M character-level corpus derived from Wikipedia. Following prior work 22 48), we split the raw text into non-overlapping sequences of 256 characters. No tokenization applied each character treated as a discrete token from an alphabet of size 27 (26 letters + space). We use the first 90% of the dataset for training and the remaining 10% for validation and testing.
Model Setup. All models use a 12-layer Transformer with hidden size 768 and 12 attention heads. We adopt the absorbing-state forward kernel described in Section A.5 For CaDDi, we use T = 64 denoising steps. The default context window is set to 5, and we apply latent truncation and trajectory recomposition described in Section C.T Baselines (D3PM, SEDD, MDLM, UDLM) are either reimplemented or taken from their official codebases using the configuration for fair comparison.
Training Details. Models are trained using Adam with learning rate of 3e-4 and 2,500 warm-up steps. We use batch size of 512 and train for IM steps. We use the simplified ELBO objective for absorbing kernels as described in Equation (8), which reduces to a weighted cross-entropy loss over clean targets.
amazon-polarity,
32

---

Evaluation Metrics, We report bits-per per-character (BPC) on the test set, computed as:
BPC =
1og2 p(x;),
where is the sequence length and p(r;)) is the predicted probability of the i-th character. For diffusion-baset models, compute variational upper bound on the log-likelihood using the ELBO objective. For autoregressive baselines, we report the true NLL.
Note: We do not report likelihood-reltaed metrics for CaDDi-AR due to its token-level autoregressive decomposition under X parameterization po (Xo Xxrr IT Po (x) x² .x:r). This formula- tion makes direct evaluation of log pæ (xi-1 X.T) intractable, as requires marginalizing over all possible prefix sequences x A tractable lower bound can estimated via:
log pa (xi-1 Xx.T) > 2xp'
Npo (x log (x)-1 Xo Xx.T)
(34)
but this introduces an additional approximation gap in likelihood estimation, making reported values less directly comparable.
Comparison Notes. Likelihood estimation of diffusion model is sensitive the discretization of timesteps: as the number of steps increases, perplexity typically decreases. To ensure a fair comparison, all diffusion-based models are evaluated using 64 denoising steps for consistency. Some prior works report results under continuous-time settings or with 1000-step discretizations; we include these numbers for reference but highlight the corresponding rows in gray in Table 2 to indicate that they are comparable.
E.4 General Language Reasoning Dataset Details
Dataset Overview. We evaluate CaDDi-AR set of natural language understanding benchmarks covering commonsense reasoning, factual QA, and reading comprehension:
ARC-Challenge / ARC-Easy multiple-choice science questions.
BoolQ
13
binary reasoning dataset based on short context.
PIQA
commonsense physical reasoning questions with two-choice answers.
RACE
multi-choice reading comprehension from English exams.
Social IQA
45
social commonsense reasoning dataset.
LAMBADA
cloze-style word prediction requiring broad context understanding.
we fine-tune CaDDi-AR on ShareGPTI5 dataset from a pre- trained QWen- checkpoint using our diffusion-based objective with T 64 steps. The model is trained using Adam W with a learning rate of 5e-5, batch size of 64 with gradient accumulation, and 20K total steps. We adopt the absorbing kernel formulation with simplified ELBO as the training loss.
Fine-tuning Setup. Following 39
Inference Procedure. We evaluate CaDDi-AR standard natural language reasoning tasks using the Language Model Evaluation Harness [19], a widely adopted framework for assessing pretrained language models on benchmark datasets. Following common practice, we convert each task into a text completion format and measure the log-likelihood of the correct answer under the model. The final prediction is selected as the choice with the highest likelihood. This approach ensures consistency across models with different architectures (e.g., ARMs and MDMs). We report accuracy as the primary evaluation metric for all datasets.
'nttss//shareept.com//
33

---

Baselines. We compare CaDDi-AR against several established language models of comparable scale. Specifically, include GPT-2 (1.5B parameters), TinyLLaMA (1.1B), and MDM (I.1B), recently proposed diffusion-ban evaluated using the same prompt formatting and inference procedure as described above. For MDM, we directly use the performance numbers reported in the original paper.
language model. To ensure fair comparison, all baselines are
G
Additional Experiments and Ablation Study
G.1 Effect of Sampling Steps Generative Quality
2000
1500
Generrative Perplexity
1000
9
Sampling Steps
Figure 8: Generative Perplexity Distribution Across Sampling Steps
To investigate the relationship between sampling budget and generative quality, we evaluate the generation of CaDDi model trained with LMIB dataset under varying numbers of sampling steps. We report generative perplexity as the evaluation metric, computed from oracle model's likelihood on generated sentences.
CaDDi trained with a fixed discrete timestep schedule ( T 64) To enable adaptive sampling with fewer denoising steps inference, employ a uniform step-skipping strategy. Specifically, for selected timesteps f, we directly use the most recent prediction of X₂ sample from the corresponding latent distribution (x, xp), without invoking the neural network for prediction. This allows efficient generation under a reduced sampling budget while preserving the learned diffusion trajectory.
Unlike prior work in continuous- partinuuus-tmme time settings 44 37
Figure presents a violin plot illustrating the distribution of perplexity scores as function of the number of denoising steps. As the number of sampling steps increases, observe consistent reduction in perplexity, indicating improved sample quality. This trend suggests that the model benefits from longer refinement trajectories, with more steps allowing finer-grained correction of uncertainty during generation.
The observed perplexity curve approximately follows scaling behavior, reminiscent of trends in autoregressive models where performance improves predictably with increased compute or depth. This hints at potential scaling law in non-Markovian discrete diffusion generation: sample quality improves smoothly with increased inference budget. Future work may explore formalizing this relationship and connecting it to theoretical underpinnings of discrete-ting time inference refinement.
34

---

Val BPD
[Context Window
Val BPD
(Context Window
1.95
Val [Context Window
Val BPD Moving Average (Conteat Window B)
Val BPD Meving Average (Context Window 5)
Val BPD
(Contuat
1.90
1.80
1.70
1.60
10000
15000
20000 25000
30000
35000 40000
4500K
Step
Figure Validation BPD over training steps for different context window sizes (3, 5, and 8). Dashed lines represent raw validation BPD at each step, while solid lines show the smoothed moving average. Larger context windows consistently yield lower BPD, indicating improved modeling of long-rant rangerange dependencies. However, context window 5 offers favorable balance between performance and stability.
G.2
Other Ablation Study
To assess the impact of key architectural and training design choices, we conduct an ablation study subset of the Text8 dataset, using only the first 10% of the training data. We focus three factors: the number of diffusion steps, the choice of positional encoding, and the size of the context window used during latent truncation, as described in Section C.I.I Results are shown in Table 5 using bits-per-dimension (BPD), perplexity, and negative log-likelihood (NLL) as evaluation metrics.
Diffusion steps. Increasing the number of diffusion steps from 16 to 128 consistently improves performance across all metrics. Specifically, BPD decreases from 2.013 to 1.712, and perplexity drops from 4.04 to 3.28. This suggests that additional refinement steps lead to more accurate modeling of the data distribution, albeit at increased computational cost.
Positional encoding. We compare our 2D RoPE encoding (used in CaDDi) with ID RoPE and sinusoidal RoPE variants. The CaDDi encoding achieves the best results, while alternative encodings result in slight performance degradation. This highlights the importance of aligning positional encodings with the inductive biases of the diffusion-basee architecture.
Context window. We vary the context window size during latent truncation and observe a trade-off between context length and model performance. A window size of 8 yields the best results (BPD 1.740) but incurs higher computational cost. A window size of offers a favorable balance, achieving competitive performance with reduced resource requirements. Accordingly, setting in our experiments. We also plot the learning dynamic of different context window size in Figure These findings support the intuition that larger context windows aid in modeling long-range dependencies, though benefits plateau beyond a certain size.
adopt it the default
Attention masking. We compare two attention masking strategies: a block-level causal mask and token-level causal mask combined with Causal Bidirectional Augmentation, as described in Section The two approaches yield nearly identical performance. However, the token-level mask with bidirectional augmentation offers practical advantages: enables accelerated inference through flash attention and is inherently more compatible with pretrained large language models.
35

---

Table Ablation study evaluating the impact of diffusion steps. positional encoding, context window Default condiguration is denoted with
size, and attention masking strategies.
Configuration
Perplexity
Ca steps
2.013
4.0362
1.3953
CaDDi-32 steps
1.845
3.5925
1.2789
Diffusion steps
CaDDi-64 steps"
1.751
3.3659
1.2137
CaDDi-128 steps
1.712
3.2761
1.1867
2D RoPE
1.751
3.3659
1,2137
Positional Encoding
ID RoPE
1.773
3.4176
1.2290
Sinusoidal RoPE
1.801
3.4846
1.2484
Window-8
1.740
3.3404
1.2061
Context Window
Window-5*
1.751
3.3659
1.2137
Window-3
1.791
3.4605
1.2414
Block-level causal mask
1.747
3.3566
1.2109
Attention Mask
Token-level causal mask
1.751
3.3659
1.2137
Causal Bidirectional Aug.
MDLM
120
CaDDi
D3PM
Generative Perplexity
MDLM (na noise)
100
CaDDi (no noise)
(no noise)
60
40
7/8 T
13/16
6/8 Timestep (fraction of T)
11/16T T
5/8 9/16
1/2 T
Figure 10: Generation performance under manually injected noise at different timestep
G.3 Inference Robustness Under Noise Injection
To further assess the robustness of our proposed CaDDi model during generation, we conduct controlled perturbation study in which synthetic noise is manually injected into the latent trajectory various timesteps. This simulates potential inference-ting errors and allows us to systematically examine how different models propagate and recover from early-stage inaccuracies.
Setup. At chosen timestep tokens in the latent variable x, by replacing them with uniformly specific values from the vocabulary, thereby introducing controlled deviation from the expected latent distribution. We then allow the model to proceed with the remaining reverse steps without additional interventions and evaluate the final output quality.
97/1,,
... T}, randomly corrupt subset of
Metrics. We report generative perplexity computed against oracle LMs (e.g., GPT-2), following the evaluation procedure in Section 5.2. Lower perplexity indicates stronger resilience to the injected noise.
CaDDi maintains significantly lower perplexity under all noise injection conditions compared D3PM and MDLM. This highlights the benefit of CaDDi's non- Markovian architecture, which conditions the full generative trajectory rather than single latent
Findings. As shown in Figure 10
36

---

state. Notably, CaDDi is more effective at self-correction when noise is injected early (e.g.. t = T/2 ). where traditional Markovian models suffer the most from cascading inference errors.
These results validate the design motivation behind CaDDi and demonstrate its ability to recover from intermediate perturbations, making it particularly well-suited for scenarios with partial uncertain context.
37

---

H Generation Samples from CaDDi
H.1 Amazon Polarity
this is negative review:
title
were they afraid of newborns?
content this book was terrible. it did not address the issues at regarding newborns, which assume are destined to inconsequential every so often. there was no postiveness the middle the book, and most of it was rushed to press. whoever wrote the previous review should have gotten an apologia their final printer of the book. this writer did so only in the first few pages, it was as if she just taped pages and put in her books and taped it more than in
this is a negative review:
title: blechhhhh content
content: wow, isn't nice to see such juvenile drivel such toole popular on the net. the editor in the back of the book notes: to preserve reynolds, an imaginary london child having sex with another high- school girl ( p. s. ) you don't know what kind of busted ethan padar worried, or would mentor house floor fort chemist who sullys ceilings with zero light. the whole novel is lame, poorly written and awfully mediocre. the plot is transparent from the first page... ...
this is a negative review: title: pile of 1 % crap
content this movie was disaster: think the director wanted cheap hair and vaginal characters, scaring everyone on the back of the face of the audience with pointless bigs or even no, uncreative dialouge. the ruined 30 minutes most of this movie, rushes in and out of the movic. i'm huge fan of slasher flicks, even makes huge claims to hard work on hair. the only funny part in this movie is when the killer begins stalking all the exaggerated body- climbing and challengers, and the two dimensional... ...
this a negative review:
title: what an insult to music lovers!
content: : the true diva that deserves the wide spread islam and the lies of the young musicians who make this music allows for such glittery displays and excuses. she never needs spend our high time singing this exquisite in its gem the artist just shadow of her most famous but she has talent. where excels is terribile and exquisite expression of native american descent that will make dog earpin jams easy to count. she has a good voice but she hasn't mastered it. oh, she needs to go
this a negative review:
title: : miserable album content
content: i owned this album and when it came out i loved it. unlike the last albums her voice was able make coheed and connie told cool gentle sigh. figured the first few tracks could have been strengthened by absolutely no more stale vocals, but they are crafted off the page. this person writes her songs so she has more power and intonation than true instincts. like to listen to the song that she was destined to write. get the single years that are runaways. only excuse me on the radio is sweetly sung by anyone,
Figure 11:
Sample generated using CaDDi, conditioned negative prompt. In each sample, the purple text indicates the prompt.
38

---

this is a positive review:
title: one for the head bob fans!
stadium since the mid 70's. this double Ip has all of the greatest hits it! why not split their support. this has soul in the covers not on the cover hercules and freddie leslie heralbull rock at same time, even the
content: i've
and theatrics contribution is true, and paul petrie stevie ray & carole or even stevie ray drivel adds some new comparatively to the cole porter music. a truly unique collection of performers work, and the ed succeeds
this is a positive review: :
title: : jk rowling is a sci fi crime novelist!
content: in harry's hell called scream, jk rowling gets back the bullet to get to jk's responsiblity. liked this book much. the storyline was great, the characters well drawn and great, and jk rowling made me care about what happened. this is the second book the harry hallic tray fouls trilogy including a sister named calvin, who is young. lucy prince is young again following her sister and her intense ex boyfriend michaelk bo...
this a positive review
title: a great story and excellent special effects and overall worth the film content: trust me, this is much better than the original. set up for heaven's 's gate and the framer though works well it becomes predictable and starts going nowhere. the story becomes more developed later and really makes good place for truman if you enter into the dsi world you' 're never supposed to forget. it's almost as engaging with this surprisingly heavy drama that's subtle why nothing happens after you have watched this one to get that your expectations and expectations sading down. recommend definitely seeing this...
this is a positive review: title: simply the best
content: have been wanting to have safe with my son my son, we used to play w his cardboard toy box. but thought this was just hazard. well, after true terror in my baby's swing, i hunted down the safe and amazon had the best price i could find. great product that was huge relief from our efforts. my 7 month old loves to play with this at least different times that put him in the swing. can see how i will get this useful for the one he has. i am so glad i...
this a positive review:
title: a surprise!
content this box is prelude to the first golden age of horror crooning ( which correctly apes de croes he tends to say wayans juveniles below and these individual breakfast of murders. the writers of these crime mysteries new orleans exhibit that encompasses the catholic church in their b w documents. best blurb of all those who would read about the errors they witness and don't 't believe. an interesting twist ending to the loose ends. hercule poirot is cast as a smart cop undercover. the two big funny g
Figure 12:
Sample generated using CaDDi, conditioned on positive prompt. In each sample, the purple text indicates the prompt.
39


Reviews:
[{"review_id": "z3CXjanG1r", "reviewer": "Reviewer_CSEv", "summary": "The authors argue that the Markovian assumption in discrete diffusion models is a key limitation, as it can lead to error accumulation during sampling. To address this, they propose a novel algorithm called CaDDi, which conditions the denoising model’s predictions on the entire generation trajectory. Specifically, they modify the transformer architecture to incorporate the full diffusion trajectory as input. Empirically, CaDDi achieves competitive performance on natural language benchmarks.", "questions": "See weaknesses above.", "limitations": "see weaknesses above.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\nThe semi-speculative decoding sampler proposed in this paper is interesting.\n\n### Weaknesses\n\n1. In masked diffusion models, the state x_t inherently encodes the entire trajectory x_{t:T}, making the decoding sequence largely irrelevant. As shown in Table 2, CaDDi clearly lags behind MDLM in terms of perplexity, suggesting that the additional engineering to incorporate the entire trajectory does not provide a clear benefit.\n\n2. It is unclear why the perplexity of the models on LM1B is not reported, given that the authors trained their model on this dataset. Perplexity remains the gold standard for comparing language models. Generative perplexity is typically used to evaluate samplers.\n\n3. Table 1 is misleading:\n(a) What does T refer to? Is it the sampling temperature or the number of time discretizations? What discretizations were used for the baselines such as MDLM, D3PM, and UDLM?\n(a) What sampler was used for each method? If CaDDi uses the proposed semi-speculative decoding sampler, it would be helpful to report the sample quality from CaDDi using the ancestral sampler as well. This would allow the community to disentangle the improvements due to (1) the non-Markovian trajectory and (2) the semi-speculative decoding sampler, in comparison to MDLM.\n\n4. Table 4 is missing major baselines. The authors should compare their method against MDLM, UDLM [2], and an autoregressive (AR) model trained with the same transformer architecture and a similar number of training steps. They can follow the steps in [2].\n\n\n\nPS. This paper is riddled with typos. A major one is Algorithm 2 being incorrectly referenced Algorithm 3 in line 216.\n\n\n\nReferences\n\n[1] Sahoo et al., 2024, \"Simple and Effective Masked Diffusion Language Models\"\n\n[2] Schiff et al., 2025, \"Simple Guidance Mechanisms for Discrete Diffusion Models\"", "was_revised": true, "final_justification": "The ancestral sampler in masked diffusion models cannot update its predictions. This paper proposes a method to allow the diffusion model to update its predictions. However, **it does not compare against existing predictor–corrector methods [1, 2, 3]**, which achieve the same goal without changing the model architecture or requiring any fine-tuning.\n\n\nFor this reason, I strongly believe the paper should be rejected.\n\nI would like to maintain my score for the following reasons:\n1. Computational cost: CaDDi achieves a better perplexity than MDLM by using 5× more FLOPs (ignoring the quadratic attention cost in sequence length for now). A fair comparison would be an MDLM with a similar FLOP budget—equivalent to ~5× more parameters, since FLOPs ≈ 6 × (number of parameters) × (training tokens). **This method will not scale to longer context lengths, as attention will become the limiting factor**.\n\n2. Fine-tuning requirement:\n> By unifying temporal and sequential (spatial) dependencies within a single causal transformer architecture, CaDDi enables direct reuse of pretrained LLM weights without requiring architectural changes. To our knowledge, this capability has not been explored in prior work on non-Markovian discrete diffusion.\n\nWhile this is presented as a strength, CaDDi still requires fine-tuning an existing LLM. By contrast, **predictor–corrector approaches [1, 2, 3] require no fine-tuning** at all.\n\n\n3.\tArchitectural changes: Other reviewers seem particularly impressed by the 2D RoPE embeddings introduced in this paper. **However, this is actually a disadvantage**: predictor–corrector approaches achieve the same functionality without any architectural modifications.\n\n\n4. Weak experiments: LM1B with a GPT-2 tokenizer yields ~30 unique tokens per sentence, each contextually distinct, making generative perplexity meaningless. Overall, the paper relies on toy datasets and lacks comparison to sensible baselines.\n\n\n\n### References:\n[1] Wang et al., 2025 \"Remasking Discrete Diffusion Models with Inference-Time Scaling\"\n\n[2] Campbell et al. 2022, \"A Continuous Time Framework for Discrete Denoising Models\"\n\n[3] Gat et al., 2025, \"Discrete Flow Matching\"", "rebuttal": "We thank the reviewer for their thoughtful feedback and insightful comments. We appreciate the recognition of our semi-speculative decoding sampler as an interesting contribution. We address each of the raised weaknesses below.\n\n1. **\"In masked diffusion models, the state $x_t$ inherently encodes the entire trajectory $x_{t:T}$, making the decoding sequence largely irrelevant. As shown in Table 2, CaDDi clearly lags behind MDLM in terms of perplexity, suggesting that the additional engineering to incorporate the entire trajectory does not provide a clear benefit.\"**\n\nWe respectfully disagree with the assertion that $x_t$ inherently encodes the entire trajectory $x_{t:T}$ in non-Markovian masked diffusion models. In CaDDi, noise is added independently at each timestep, conditioned only on the original data $x_0$. In this case, from the very nature of how trajectories are created, the state $x_t$ itself does NOT encode the entire trajectory before it.\n\nNow, we explain why the above formulation is necessary and helpful. As discussed in Section 3.1 and further elaborated in Appendix A.4 (\"Failure to Remask Issue\"), traditional Markovian discrete diffusion models, including MDLM, suffer from a \"failure to remask\" problem. Once a token is unmasked (i.e., $x_t \\neq e_m$), it is deterministically copied to $x_{t-1}$ regardless of the model's prediction. This means the model cannot revise early mistakes, leading to error accumulation.\n\nOur non-Markovian approach ensures that **each state $x_t$ in the latent trajectory contains unique and complementary information**, rather than being a mere compression of the subsequent trajectory. This fundamental difference in the forward process is key to our non-Markovian formulation, which then allows the reverse process to condition on the historical trajectory $x_{t:T}$ (as described in Section 3.3 and Algorithm 1). This explicit conditioning enables the model to effectively revisit past states, thereby lifting the restrictive Markov constraint and leading to a more robust inference process that can self-correct errors.\n\nRegarding the perplexity comparison in Table 2, the reviewer states that CaDDi lags behind MDLM. We believe this observation may stem from an unfair comparison across different time discretizations, and we wish to clarify the following:\n\n- **Performance at Same Discretization (Table 2)**: We wish to emphasize that the reviewer might be wrongly comparing across different time discretizations. As noted in Appendix Section E.3, **likelihood estimation for diffusion models is sensitive to the number of discretization steps**. As clearly shown in Table 2 (rows shaded in gray), when comparing models at **the same 64-step discretization**, CaDDi achieves a BPD of $<1.41$ and Perplexity of $<2.66$, which is indeed **the best performance** among all discrete diffusion models evaluated at 64 steps (MDLM at 64 steps has BPD $\\leq1.46$ and Perplexity $\\leq2.75$). This demonstrates a clear benefit of our approach within comparable settings. Some baselines report results under continuous-time settings or with 1000-step discretizations, which are not directly comparable to our 64-step discrete evaluation. Since MDLM (and other continuous-time models) are trained in a continuous-time framework while our model operates in a discrete-time framework, there isn't a strict one-to-one correspondence for continuous frameworks. However, to further demonstrate our model's consistent superiority, here we trained an additional model with 256-step discretization (which is the same as the sequence length in the Text8 dataset). This model achieves a **BPD of ≤1.37** (better than MDLM even with $\\infty$ time), further solidifying CaDDi's competitive performance.\n- **LM1B Results (Table 1)**: On the larger and more representative LM1B dataset, both CaDDi and CaDDi-AR consistently outperform MDLM in generative perplexity across all three language model oracles (GPT-2, Llama-2-7B, and Llama-3.2-3B) and various sampling temperatures. This demonstrates a clear benefit of our approach on a more challenging and widely used benchmark.\n- **Ablation Study on Robustness (Figure 4 and G.3)**: Our ablation study on inference robustness (Figure 4 and Figure 10 in Appendix G.3) clearly shows that CaDDi exhibits significantly stronger resilience to manually injected noise compared to MDLM and D3PM. This directly supports our argument that conditioning on the historical trajectory improves inference robustness and self-correction, which is a key benefit not captured solely by perplexity on Text8.\nTherefore, the overall empirical evidence, especially when fairly comparing at the same discretization on Text8, on LM1B, and in terms of robustness, strongly supports the benefits of our non-Markovian approach.\n\n**2. \"It is unclear why the perplexity of the models on LM1B is not reported.\"**\n\nThanks for raising this point. Our primary intention with the Text8 experiments was to demonstrate the likelihood modeling ability of our model, and we considered reporting additional perplexity metrics on LM1B to be somewhat redundant in that regard. Instead, we established the LM1B dataset as a generative task to evaluate the direct generative ability and quality of samples across different models.\n\nHowever, we appreciate your suggestion for including these results for completeness. We are pleased to provide the perplexity results for the LM1B dataset here\n\n|Model| PPL on LM1B↓|\n|-| -|\n|D3PM|≤77.50|\n|SEDD|≤32.79|\n|MDLM|≤23.00|\n|CaDDi|≤21.91|\n\n**3. Clarification Regarding Table 1**\n\nWe sincerely acknowledge the reviewer for raising concerns. In the meantime, we highly recommend the reviewer to also refer to our comprehensive **22-page appendix in the supplementary material**. **This supplementary material has already covered most of the following concerns in greater detail**. But thanks to the suggestions, we recognize that some details might not have been sufficiently clear in the main text and will move relevant information from appendix to the main text in the next revision to improve clarity and accessibility.\n\n- **(a) What does T refer to? Is it the sampling temperature or the number of time discretizations?**: In Table 1, \"T\" refers to the sampling temperature (e.g., T=1, T=0.7, T=0.5). This is explicitly stated in the caption of Table 1. The number of time discretizations (diffusion steps) for CaDDi and CaDDi-AR is consistently 64 for the reported LM1B experiments, as stated in Appendix Section E.1 line 923. For baselines, Section E.1 notes: \"*Note that models such as MDLM and SEDD are trained in continuous time, whereas our models operate in discrete time.*\" For evaluation, \"*All diffusion-based models use 64 denoising steps during inference.*\"\n- **(b) What discretizations were used for the baselines such as MDLM, D3PM, and UDLM?** As mentioned above and in Appendix Section E.1, MDLM and SEDD are trained in continuous time. For evaluation, all diffusion-based models (including MDLM, D3PM, UDLM, SEDD, DFM) were evaluated using 64 denoising steps during inference for fair comparison on LM1B (Section E.1). For Text8 (Table 2), we explicitly state the steps/discretization for each model\n- **(c) What sampler was used for each method? If CaDDi uses the proposed semi-speculative decoding sampler, it would be helpful to report the sample quality from CaDDi using the ancestral sampler as well**. For CaDDi, we use the ancestral sampler as described in Algorithm 1. For CaDDi-AR, we employ the semi-speculative decoding sampler (Algorithm 2) to accelerate inference. We agree that disentangling the improvements is valuable. The performance reported for \"CaDDi\" in Table 1 uses the ancestral sampler (Algorithm 1), while \"CaDDi-AR\" uses the semi-speculative decoding sampler (Algorithm 2). The comparison between \"CaDDi\" and \"CaDDi-AR\" in Table 1 directly showcases the impact of the token-level autoregressive factorization and the semi-speculative decoding. CaDDi (block-level autoregressive) already outperforms MDLM, demonstrating the benefit of the non-Markovian trajectory. CaDDi-AR, with its token-level factorization and semi-speculative decoding, further improves performance, indicating the combined benefit.\n\n**4. \"Table 4 is missing major baselines. The authors should compare their method against MDLM, UDLM [2], and an autoregressive (AR) model trained with the same transformer architecture and a similar number of training steps. They can follow the steps in [2].\"**\n\nTable 4 focuses specifically on **conditional text generation** on the Amazon Polarity dataset, a task where we demonstrate the flexibility of CaDDi for text infilling and classifier-free guidance. For this task, we compare against a fine-tuned GPT-2, a strong autoregressive baseline for conditional generation that uses the same 12-layer transformer architecture as CaDDi.\n\nWhile MDLM and UDLM are general discrete diffusion models, their application to conditional generation with text infilling (as opposed to unconditional generation or simple prefix conditioning) is not their primary focus or demonstrated strength in prior work. \n\nTo further address the request for additional baselines, we have conducted an experiment with UDLM on the Amazon Polarity dataset. The results are as follows:\n\n|Model|Condition|Sentiment Accuracy (%)|\n|-|-|-|\n|GPT-2|Positive|73.07|\n|GPT-2|Negative|75.18|\n|UDLM|Positive|68.82|\n|UDLM|Negative|74.56|\n|CaDDi|Positive|71.37|\n|CaDDi|Negative|85.42|\n\nIt is important to note that this is not a standard benchmark, as the sentiment accuracy metric can be unstable due to the inherent biases in the dataset. Therefore, these UDLM results should be considered as a reference to demonstrate the general ability of conditional generation rather than a definitive comparison.\n\n**5. \"Algorithm 2 is being incorrectly referenced Algorithm 3 in line 216.\"**\nThanks for pointing this out! We will revise this typo."}, {"review_id": "QDxFxvSBL7", "reviewer": "Reviewer_qCqR", "summary": "This paper proposes a novel diffusion model, called CaDDi, for discrete sequences. Instead of relying only on the previous diffusion step, it leverages all diffusion history to provide a more stable discrete diffusion model. Notably, when the diffusion step T=1 it can be regarded as an ordinary causal language model, thus allowing finetuning existing large language model using the power of diffusion model.\nExtensive experimental results indicate that the proposed CaDDi has a better performance than previous discrete diffusion models. While it is slightly behind ordinary causal LLMs without finetuning, this is an important step towards useful diffusion model for discrete sequences.", "questions": "Nothing.", "limitations": "Nothing.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "See above. I found no clear weaknesses in this approach.\nWhile pages are limited, I would like to see some practical texts where the proposed CaDDi works better than ordinary causal LMs or other discrete diffusion models.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer qCqR for their thoughtful and encouraging review. We are glad that the reviewer recognizes the novelty, significance, and technical quality of our work. We appreciate the comment regarding practical use cases where CaDDi outperforms standard causal LMs or existing discrete diffusion models. While such examples are constrained by space in the current submission, we agree this is an important direction and will add some additional illustrative use cases in the manuscript."}, {"review_id": "WkiB4TZISJ", "reviewer": "Reviewer_wro5", "summary": "This paper introduces a Non-Markovian discrete diffusion framework for text generation. Unlike conventional discrete diffusion models that rely on the Markovian assumption, CaDDi conditions each denoising step on the full generative trajectory.\n\nFurthermore, CaDDi is introduced as a causal version of discrete diffusion, using transformer architecture as a natural bridge. It unifies sequential and temporal reasoning.\n\nExperimental results demonstrate superior generative quality and robustness on several benchmarks (LM1B, Text8, Amazon Polarity, reasoning tasks) compared to existing discrete diffusion baselines, while retaining compatibility with pretrained LLMs.", "questions": "In addition to weaknesses,\n\n1. Could the authors further quantify the computational cost of CaDDi and CaDDi-AR compared to traditional AR models and Markovian discrete diffusion in both training and inference?\n\n2. What is the sequence length in the experiments? Have the authors tested CaDDi on longer sequences, and how does it scale in terms of memory, computation, and quality?", "limitations": "I do not see the discussion on limitations in this paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n- The paper introduces a 2D positional encoding, which is novel.\n- The paper introduces a semi-speculative decoding to speed up inference.\n- Ability to reuse pretrained LLMs for diffusion tasks broadens the applicability of diffusion-based approaches.\n- Demonstrates clear improvements in generative perplexity.\n\nWeaknesses:\n\n- The authors did not point out that the idea of Non-Markovian discrete diffusion is not novel. For example, [1] first propsed a Non-Markovian discrete diffusion, in addition to allowing $p(x_t|x_{t-1}, ... x_{0})$ to be conditioned on the whole sequence, it is also aware of the sequencial and temporal relationship of Non-Markovian discrete diffusion (using transition time $\\tau$ to gain faster inference).\n- Due to the above reason, there is a lack of proper reference to prior work on Non-Markovian discrete diffusion models like DNDM, as well as Non-Markovian diffusion models like DDIM.\n- Despite semi-speculative decoding, it seems that inference remains slower than discrete diffusion models, since the complexity is L*T. The authors did not discuss the remaining computation overhead.\n- The \"VERIFY\" and \"CORRECT\" process in Algorithm 2 has never been defined in this paper. \n- Algorithm 2 is never referenced. It seems like a typo (Line 216 says \"see Algorithm 3\")\n\n[1] Chen, Zixiang, et al. \"Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time.\" NeurIPS 2024.", "was_revised": true, "final_justification": "The reviewer have addressed all my concerns in their rebuttal. I adjusted my score due to that the discussion with prior work is sufficient.", "rebuttal": "We sincerely thank the reviewer for their thorough and insightful feedback on our paper. The comments are highly valuable and will help us significantly improve the clarity, completeness, and impact of our work. We are encouraged that the reviewer found our 2D positional encoding and semi-speculative decoding to be novel and appreciated the performance improvements and the ability to reuse pretrained LLMs.\n\nBelow, we address each of the reviewer's points and detail our planned revisions.\n\n**1. & 2. Novelty and Referencing of Non-Markovian Discrete Diffusion DNDM**\nThank you for highlighting the need for a more prominent comparison with DNDM. Actually we did include a discussion of related work including DNDM in our 22-page appendix (in supplementary material), but we understand that supplementary material can be easily missed and agree that this comparison is crucial for the main paper. We appreciate the opportunity to clarify the distinction and re-emphasize our specific contributions.\n\n- **Regarding DNDM**: While both CaDDi and DNDM are non-Markovian, they address different goals with distinct methodologies. DNDM's primary contribution is achieving **fast, training-free sampling** by derandomizing the generation process with a predetermined transition time. In contrast, CaDDi leverages the non-Markovian structure to **improve generative quality and robustness** by explicitly conditioning the denoiser on the generative trajectory.\nTherefore, the conceptual overlap is limited to the relaxation of the Markovian assumption. Our core contributions are orthogonal to theirs and focus on a novel modeling paradigm:\n    1.  A unified causal framework that models both sequential and temporal dimensions in a single transformer.\n    2. A novel 2D Rotary Positional Encoding (RoPE) designed for this unified structure.\n    3. A direct method for adapting pretrained LLMs to the discrete diffusion paradigm.\n    \nThanks again for your suggestions. We will move the discussion of DNDM from the appendix to the main body of the paper, expanding the explaination.\n- **Regarding DDIM**: DDIM is a foundational work for non-Markovian sampling in **continuous** diffusion models. Our paper is focused on the discrete domain, where the challenges and solutions are fundamentally different (e.g., operating on categorical variables, different kernel designs, and no direct equivalent of an ODE formulation).\n\n**3. Inference Speed and Computational Overhead (Complexity L×T) & Quantifying Computational Cost (Training and Inference):**\nWe acknowledge the reviewer's point regarding the inference complexity and the need for a more detailed quantification of computational cost. While the theoretical complexity for naive CaDDi-AR's token-level autoregressive generation within each diffusion step is indeed L×T, our semi-speculative decoding strategy is designed to significantly reduce this in practice. The exact degree of reduction cannot be theoretically derived as it's an algorithm that dynamically skips steps; however, it consistently results in a substantially lower effective number of function evaluations (NFE) than L×T. To further quantify the inference computational cost, we provide the following table on the LM1B dataset during generation:\n| Model | NFE | GenPPL on LM1B↓ |\n| -| -| -|\n|MDLM-64|64|199.45| \n|MDLM-500|500|188.93|\n|MDLM-1000|1000|179.41|\n|CaDDi|64|142.51|\n|CaDDi-AR|8192|137.82|\n|CaDDi-AR (with semi-sepculative decoding)|~656|139.80|\n\nAs shown, semi-speculative decoding significantly reduces the NFE for CaDDi-AR while maintaining competitive performance, highlighting its effectiveness as a speedup technique. It is also worth noting that the non-AR version of CaDDi (with complexity O(L)) with 64 steps already achieves better performance than MDLM with 1000 NFE (and with larger FLOPs). CaDDi-AR is a strategy to further enhance performance, which inherently entails a higher computational cost, and semi-speculative decoding is specifically designed to alleviate this additional overhead.\n\nRegarding the overall overhead of computation in training, we discuss this practical limitation in Appendix Section C.1 (\"Context Window and Latent Compression\") and introduce latent truncation and trajectory re-composition as strategies to compress the latent trajectory to fit within a bounded context window. Specifically, latent truncation (Section C.1.1) limits the conditioning to the most recent m timesteps, which directly controls the computational cost in both training and inference. We have already performed an ablation study on the context window size (Section G.2, Table 5, Figure 9), showing that a moderate window size offers a favorable balance between performance and computational cost. These strategies ensure that CaDDi achieves better performance while not excessively sacrificing computational cost.\n\n**4. Terminology Clarification of \"VERIFY\" and \"CORRECT\" in Algorithm 2:**\n\"VERIFY\" and \"CORRECT\" refer to the standard operations within speculative decoding, adapted for CADDi-AR. Specifically:\n- **VERIFY**($p_0$, $\\mathbf{x}\\_{1:T}$, ${\\hat{x}}\\_0^{\\text{prev}}$): This step takes the model $p_0$, the current latent trajectory $\\mathbf{x}\\_{1:T}$, and the draft sequence $\\hat{x}_0^\\text{prev}$ (which is the predicted clean sequence from the previous timestep). It evaluates the probability of each token in ${\\hat{x}}_0^\\text{prev}$ given the context and determines the longest prefix of ${\\hat{x}}_0^\\text{prev}$ for which the tokens are accepted based on a specified confidence threshold (In this work, we use a criterion similar to Nucleus Sampling, rejecting samples based on the probability of choosing the same token between this timestep and the prior timestep. If the probability is higher, no rejection happens). The output $\\tilde{i}$ is the index of the first token that is rejected or where the draft ends.\n- **CORRECT**($p_0$, $\\mathbf{x}\\_{1:T}$, ${\\hat{x}}\\_0^{\\text{prev}}$): This step takes the same inputs. It returns the accepted prefix of $\\hat{x}\\_0^\\text{prev}$ (i.e., ${\\hat{x}}\\_0^{0:\\tilde{i}-1}$) and prepares the model to resume autoregressive sampling from the position $\\tilde{i}$. This process is analogous to how speculative decoding \"corrects\" a proposed draft by either accepting it fully or partially and then continuing generation from the point of divergence. We will ensure these operations are precisely described in Section 4.2 to enhance clarity.\n\n**5. Typo in Algorithm Reference (Line 216):**\nThanks for pointing this out! We will fix it in the next revision.\n\n**Q1.Further Quantification about the Computational Cost of CaDDi and CaDDi-AR**\nSee Point 3.\n\n**Q2. What's the Sequence Length in Experiment and How Does it Scale on Longer Sequences?**\nThe sequence length used for the LM1B dataset is 128 tokens, and for the Text8 dataset, it is 256 tokens. These dataset configurations are standard, consistent with prior discrete diffusion work such as D3PM and MDLM.\n\nFor CaDDi specifically, the primary factor affecting memory and computation for longer sequences is the transformer's self-attention mechanism, which scales quadratically with sequence length. To address this, Appendix C (Model Implementation Details) discusses \"Latent Truncation\" and \"Trajectory Re-composition\" as strategies to manage the context window and handle longer generative trajectories. These techniques effectively compress $x_{t:T}$ to fit within the model's context. These strategies allow CaDDi to scale to longer sequence.\n\nIt is also important to note that most existing standard discrete diffusion models focus on limited sequence length generation due to their encoder structures. However, recent work, such as Block Diffusion [1], has been proposed to extend beyond these limitations. Integrating such ongoing efforts with our framework to further scale to longer sequences presents an interesting future direction.\n\n\n[1]. Arriola M, Gokaslan A, Chiu J T, et al. Block diffusion: Interpolating between autoregressive and diffusion language models[J]. arXiv preprint arXiv:2503.09573, 2025.\n\nWe thank the reviewer once again for their valuable feedback! We believe these revisions will significantly strengthen the paper by addressing the reviewer's concerns comprehensively and transparently."}, {"review_id": "gWiRBYqQIy", "reviewer": "Reviewer_cgB8", "summary": "This paper introduces a new discrete diffusion framework, CaDDi, with its variant CaDDi-AR. This framework removes the Markovian assumption and regards the noise as independently injected into the clean data. From the new assumption, the author derives an inference process and a specific training objective through the ELBO. Built upon the mathematical formulation and the observation of the connection between non-Markovian diffusion and the causal language model, the author proposes to integrate token order and timesteps into a single transformer model. To account for both dimensions, the authors also propose a 2D Rotary Positional Encoding method. The CaDDi-AR factorizes over the token space to capture the true posterior distribution and uses speculative decoding to reduce the computation. Since the causal language model can be regarded as a special case of the CaDDi-AR, pre-trained LLMs can be fine-tuned to serve as the backbone of the CaDDi-AR. Comprehensive experiments have been done to prove its effectiveness. The experiments include 5 baselines, on 3 open-source LLMs, on a billion-word dataset, Text8 dataset, and a bunch of reasoning benchmarks. The overall experimental results suggest that CaDDi is an effective discrete diffusion framework across many models and datasets.", "questions": "N/A.", "limitations": "N/A.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The core idea behind CaDDi is great. It's a non-Markovian approach to discrete diffusion, meaning it conditions the denoising on the entire generation history, not just the last step. This is a well-motivated way to tackle the error accumulation problem that plagues standard Markovian models. The theory holds up, and the training objective is straightforward.\n2. The 2D RoPE is a really clever architectural choice. It neatly handles both token position and the diffusion timestep within a regular decoder-only Transformer. The best part is that this makes it backward-compatible, so you can fine-tune existing LLMs without having to change their architecture.\n3. The authors clearly anticipated the potential for high computational cost with the token-level version (CaDDi-AR). Their \"semi-speculative decoding\" strategy is a smart fix. By using the prediction from the previous timestep as a draft for the current one, it cuts down on sampling steps and makes the whole process more efficient.\n4. The experiments are solid. CaDDi was benchmarked against a bunch of SOTA discrete diffusion models (D3PM, SEDD, MDLM, DFM) and consistently came out on top in both perplexity and reasoning tasks.\n\nWeakness:\n1. My main issue is with the comparison to standard autoregressive models. While CaDDi is clearly better than other diffusion models, it's may inherently slower than a single-pass AR model because it requires T sampling steps. The paper brings up semi-speculative decoding to help with this, but there's no direct, wall-clock time comparison to a similarly-sized AR model. I'm left wondering how practical CaDDi-AR really is. How close does it get to the speed of a fine-tuned AR baseline for the same quality? This is a key question for real-world use.\n2. The non-Markovian design means conditioning on the history of latent states. This could create a massive input sequence for the transformer, especially with a large T. The paper doesn't really dig into how well the model scales with longer context lengths, which seems like an important limitation to explore.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. The paper is a solid contribution and is suitable for the NeurIPS.", "rebuttal": "We thank the reviewer for their constructive feedback. We appreciate that they recognize the core idea of CaDDi, and we are grateful for their positive remarks on the architectural design, the semi-speculative decoding strategy, and the experiments.\n\n1. **Time efficiency of CaDDi-AR**\n\nWe thank the reviewer for highlighting the issue of CaDDi’s time efficiency. We agree that, due to the iterative refinement nature of the diffusion process, multiple passes are unavoidable—this being a key distinction from autoregressive (AR) models. Similarly, CaDDi-AR, which interpolates between the diffusion and AR paradigms, also inherits this increased time complexity. To address this, we proposed a semi-speculative decoding strategy, which significantly accelerates sampling and mitigates the computational overhead.\n\nWe would like to highlight that CaDDi-AR is a special case within the broader CaDDi framework, which is designed as a flexible interpolation between discrete diffusion and autoregressive (AR) modeling. CaDDi can be tuned to lean more toward the diffusion paradigm or the AR paradigm, depending on the application needs. This flexibility reflects an inherent trade-off between sampling efficiency and modeling expressiveness, allowing practitioners to balance performance and speed according to their specific use case.\n\nFinally, to demonstrate the time complexity of CaDDi, we include below the Number of Function Evaluations (NFE) comparison between different versions of CaDDi and MDLM with varying diffusion steps, together with their performance on the LM1B dataset. We see that while CaDDi-AR uses a high number of NFEs, semi-speculative decoding strategy reduces the numer to a reasonable regime without a significant drop in performance. Also, CaDDi with 64 steps already achieves better performance than MDLM with 1000 NFE (and with larger FLOPs). Therefore, practitioners are safe to choose the specific CaDDi model they could afford based on their need and resources.\n\n| Model                                     | NFE  | GenPPL on LM1B↓ |\n| ----------------------------------------- | ---- | --------------- |\n| MDLM-64                                   | 64   | 199.45          |\n| MDLM-500                                  | 500  | 188.93          |\n| MDLM-1000                                 | 1000 | 179.41          |\n| CaDDi                                     | 64   | 142.51          |\n| CaDDi-AR                                  | 8192 | 137.82          |\n| CaDDi-AR (with semi-sepculative decoding) | ~656 | 139.80          |\n\n\n2. **Context length**\n\nWe appreciate the reviewer for highlighting a key distinction of our method: conditioning on the full history trajectory rather than a separate time step. Indeed, without special treatment, the context length would grow with the timestep t, potentially making it computationally expensive or unstable for large T. In Appendix C.1 (\"Context Window and Latent Compression\"), we discuss this practical limitation and introduce latent truncation and trajectory re-composition as strategies to compress the latent trajectory to fit within a bounded context window. Specifically, latent truncation (Section C.1.1) limits the conditioning to the most recent m timesteps, which directly controls the computational cost. We have performed an ablation study on the context window size (Section G.2, Table 5 and Figure 9), showing that a moderate window size offers a favorable balance between performance and computational cost."}, {"review_id": "QZgYjtSJp4", "reviewer": "Reviewer_WEg3", "summary": "This paper proposes a reformulation of discrete diffusion by relaxing the standard Markovian assumption. Specifically, the noising distribution assumes conditional independence given $x_0$ that is $p(x_{t} \\mid x_{1:t-1}, x_0) = p(x_{t} \\mid x_0)$ instead of $ p(x_t \\mid x_{t-1})$ in standard diffusion, thereby removing the dependence on the previous noisy state.\n\nThe work begins by highlighting a key limitation of standard discrete diffusion models: the accumulation of decoding errors due to the strong dependency on previous state in the denoising step $p(x_{t-1} \\mid x_t)$. This dependency exposes the inference procedure to compounding deviations from small prediction errors at each timestep.\n\nTo address this, the authors propose a non-Markovian approach in which the noise at each timestep is conditionally independent given the clean data $x_0 $. That is, each noisy state depends solely on time and the original input, eliminating dependence on previous noisy states.\n\nBuilding on this conceptual framework, the paper introduces the Causal Discrete Diffusion Model (CaDDi). This model incorporates 2D Rotary Positional Encoding to handle the sequential and temporal dimensions inherent to discrete sequences. Furthermore, the authors propose an autoregressive variant, CaDDi-AR, which factorizes the joint distribution across the sequence. To mitigate the associated computational overhead, they introduce a semi-speculative decoding strategy.\n\nThe method is empirically validated through experiments, demonstrating its effectiveness.", "questions": "## Suggestions for Improvement\n\n1. The authors should clearly articulate the novel contributions of this work compared to [1], especially given the close conceptual overlap.\n\n2. A thorough literature review is necessary. Related works on compounding denoising errors and advanced discrete diffusion strategies should be discussed in one or more dedicated sections.\n\n3. An empirical analysis of the computational cost (including the effect of semi-speculative decoding) would make the evaluation more rigorous and transparent.\n\n4. A detailed explanation or formal proof of the first equality in Equation (6) would help ensure the mathematical soundness of the method.\n\n## Question:\n\nWhat is the benefit of conditioning on the whole sequence rather than on $x_t$?", "limitations": "## Limitations\n\n- The paper does not discuss (or evaluate) the residual computational overhead resulting from the computation of the whole sequence $x_{t:T}$ during training.\n- The paper does not discuss (or evaluate) the residual computational overhead after applying the semi-speculative decoding strategy.\n- The paper does **not** include a discussion of potential negative societal impacts.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "## Strengths\n\n- The paper is clearly written and easy to follow.\n- The core methodology is intersting and sound.\n- Beyond the central contribution, the paper introduces secondary innovations, including 2D rotary positional encoding, semi-speculative decoding, and leveraging pre-trained language models.\n- The experiments demonstrate some benefits of the proposed approach.\n\n---\n\n## Weaknesses\n\n### 1. Novelty\n\nThe main conceptual claim, namely, that compounding denoising errors in discrete diffusion can be alleviated via a non-Markovian assumption, has been previously introduced in [1]. While [1] focuses on graph generation, the underlying idea and motivation closely align with the present work. Therefore, the novelty of the contribution is limited.\n\n### 2. Positioning and Related Work\n\nThe paper does not adequately position itself within the existing literature. In particular:\n\n- The concept of compounding denoising errors is well-known and explicitly discussed in [2].\n- Several recent works [3–7] propose methods to mitigate such errors through informed or planned denoising and sampling strategies.\n- The relationship between the proposed semi-speculative decoding and prior decoding strategies (e.g., planning, correction, or speculative sampling [1, 3-7]) is not discussed. It remains unclear how the proposed strategy differs from or improves upon these approaches.\n\n### 3. Computational Efficiency and Evaluation\n\nThe proposed method introduces two components that increase computational cost: conditioning on the full sequence and the use of an autoregressive (AR) sampling strategy.\n\n1. The decoder conditions on the entire sequence $x_{t:T}$, which requires computing the full sequence during training. This introduces additional computational overhead compared to standard discrete diffusion conditioning only on $x_t$. However, this overhead is neither discussed nor evaluated in the paper. Furthermore, it is not evident that conditioning on the full sequence provides substantial benefit over conditioning only on $x_t$. An ablation study would be valuable to assess the impact of this design choice.\n\n2. The authors acknowledge the computational cost of their AR method resulting from the high number of function evaluations (NFE) needed. There is however no quantitative evaluation of this cost, nor of the extent to which the proposed semi-speculative decoding mitigates it. Moreover, the experiments do not disentangle the benefits of the method itself from those attributable to increased computational budget. A more detailed analysis is necessary to clarify how much of the observed performance gain results from the method rather than from additional compute.\n\n\n### 4. Theoretical Clarity\n\nThe first equality in Equation (6) is not self-evident. Providing a derivation or proof would be useful\n\n---\n[1] Boget, Y. *Simple and Critical Iterative Denoising: A Recasting of Discrete Diffusion in Graph Generation*. ICML 2025. https://openreview.net/forum?id=kPTW6hGrJy  \n[2] Lezama, J., et al. *Discrete predictor-corrector diffusion models for image synthesis*. ICLR 2023.  \n[3] Liu, S., et al. *Think while you generate: Discrete diffusion with planned denoising*. ICLR 2025.  \n[4] Zhao, Y., et al. *Informed correctors for discrete diffusion models*. https://arxiv.org/abs/2407.21243  \n[5] Kim, J., et al. *Train for the worst, plan for the best: Understanding token ordering in masked diffusions*. https://arxiv.org/abs/2502.06768  \n[6] Peng, F., et al. *Path planning for masked diffusion model sampling*. https://arxiv.org/abs/2502.03540  \n[7] Nie, S., et al. *Large language diffusion models*. https://arxiv.org/abs/2502.09992", "was_revised": true, "final_justification": "1 & 2.\nMy initial evaluation was based on the assumption that the proposed model resembled existing non-Markovian discrete diffusion models. However, these models are non-Markovian only in the forward process. The authors have committed to clarifying this distinction and adding relevant references. Given this, the method presents a novel contribution to discrete diffusion and is a meaningful contribution to the field.\n\n3 & 5.\nThe authors have provided a theoretical justification for the benefits of the method in the rebuttal. I encourage the author including elements of this discussion into the main paper. It would strengthen the core arguments and improve clarity.\n\n4.\nThe computational cost is a recognized limitation, which the authors address by restricting the context window. Since this trade-off is central to the contribution, the empirical analysis would benefit from evaluating smaller window sizes. New experiment were not realistic in the tight rebuttal schedule, but the revised version would benefit of these additional experiments. \n\n6.\nThe authors have committed to including a section on the societal impacts of their work.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We are pleased that the reviewer found our core methodology interesting and sound, and appreciated our secondary innovations, including 2D rotary positional encoding, semi-speculative decoding, and leveraging pre-trained language models. We summarize and address each of the raised weaknesses/limitation below.\n\n**1. Novelty and comparison with [1]**\n\nWe appreciate the reviewer bringing [1] to our attention. While both works share the term \"non-Markovian\", we wish to emphasize that the two are conceptually distinct frameworks and that our contributions are novel and independent, and should not be overshadowed due to similarities merely in terminology. Key distinctions and novelties of our approach compared to [1] and other related works include:\n- **Reverse Process Formulation**: The key difference lies in how the reverse process is formulated. While [1] constructs non-Markovian forward kernels as $q(x_t | x_{t-1}, x_0)=q(x_t | x_0)$, their **reverse kernel $p_\\theta(x_{t-1} \\mid x_t)$ remains \"Markovian\"**. In contrast, a core insight of CaDDi is that under the assumption of an independent non-Markovian forward process, incorporating history into the reverse process $p_\\theta(x_{t-1} \\mid x_{t:T})$ yields a strict mutual information gain $I(x_{t-1};x_{t+1:T}|x_t) > 0$. This result highlights that conditioning on the generative trajectory strictly reduces the uncertainty about $x_{t−1}$ compared to conditioning on $x_t$ alone. As a result, only coupling the non-Markovian forward process with a non-Markovian reverse process can fully leverage the structure of the data. CaDDi is explicitly designed with a non-Markovian reverse process, which enables it to consistently outperform baselines across a wide range of experiments. This architectural and theoretical distinction is central to our contribution and sets our framework apart from [1].\n- **Unification with Causal Language Models**: Another central and novel contribution of CaDDi is its **unification of sequential (causal) and temporal (diffusion) reasoning within a single non-Markovian transformer architecture**. We demonstrate that CaDDi-AR with 2D Rotary Positional Encoding treats standard causal language models as a special case (T=1), enabling the direct reuse of pretrained LLM weights with no architectural changes. This is a practical advantage that allows us to **leverage the vast knowledge encoded in large-scale pretrained LLMs for discrete diffusion as shown in experiments in Table 3**. This specific unification and the ability to fine-tune pretrained LLMs for non-Markovian discrete diffusion is not explored in [1].\n\nPS: We also note that the simple iterative denoising in [1] can also be considered a special case of ReMDM (where all tokens are remasked in the subsequent step). We have already detailed the key differences between our approach and the more general ReMDM work in Appendix Section B. We appreciate this valuable feedback and will further expand it with [1] and clarify this discussion in the revised manuscript to explicitly highlight these distinctions.\n\n\n**2. Clarification on Positioning and Related Work**\n- **Compounding Denoising Errors Issue**: We acknowledge compounding denoising errors as a known issue in discrete diffusion, discussed in prior works like [2]. Our paper mentions this in the Introduction (Page 1, Section 1, paragraph 2) and Section 2 (\"Error During Inference\"). We will explicitly cite [2] and other relevant works. Our contribution is not identifying the problem, but proposing a fundamentally different architectural solution (non-Markovian conditioning on the full trajectory) to mitigate it, unlike predictor-corrector or sampling strategies.\n\n- **Other Mitigation Strategies [3-7]**: Thanks for referring to these work, some of them are already cited in Appendix Section B. While these works also aim to alleviate the compounding error problem, we notice that **all of them focus on inference techniques, which are orthogonal to our model's core contribution**. For instance, in [3], the improvement comes from an adaptive Gillespie algorithm replacing tau-leaping. [2,4] adopts additional corrector steps. In [5,6], they address the problem through additional planning (position choosing). Similarly, [7] uses a confidence score to bias the position for unmasking. All these methodologies are inference-time techniques. In contrast, our model uniquely defines a totally different non-Markovian discrete diffusion framework that inherently learns more robust dependencies across the  generative trajectory to solve the same problem. Crucially, **these techniques are orthogonal to our approach and can be combined with CaDDi**. For example, we can integrate low-confidence remasking from [7] and additional planning using Top-K probability margin from [5] into our CaDDi model, deciding which tokens need to be remasked again from the logits. Experiments have shown that these combinations can further benefit our model, proving their orthogonality as sampling tricks. **We thank the reviewer for raising these sampling techniques, as the suggestion helped improve the quality of our work**.\n\n| Model Configuration| GenPPL on LM1B↓ |\n| -| -|\n| CaDDi|142.51|\n| CaDDi + Low-Confidence Remasking [7]|98.81|\n|CaDDi + Top-K Probability Margin [5]|112.49|\n\n- **Relationship between Semi-Speculative Decoding and Prior Decoding Strategies**: It is important to emphasize that semi-speculative decoding is a speedup technique for AR-like models, specifically CaDDi-AR, without performance sacrifice. It is fundamentally different from the sampling/decoding techniques in [2-7], and thus not comparable. We will expand this explanation.\n\n\n**3. Clarification on the Benefit of Conditioning on the Whole Sequence Rather than just $x_t$**\nAs discussed in point 1, our model uniquely conditions on the generative trajectory in the reverse process, thereby allowing the model to \"revisit past states\" and maintain a comprehensive view of the generation process, which significantly benefits performance. If the reviewer is referring to an ablation study concerning this non-Markovian reverse process, our Experiment Table 1 already provides this comparison, where MDLM can be considered a strong Markovian counterpart to CaDDi. CaDDi consistently outperforms MDLM across various metrics, demonstrating the benefits of our non-Markovian reverse process.\n\nFurthermore, if the reviewer is seeking an ablation study with models like [1], where the forward process is defined as non-Markovian but the reverse process remains Markovian (conditioning only on $x_t$), we have additionally run experiments to address this. Results with GPT2 generative perplexity show that conditioning on the generative trajectory in the reverse process offers more robust and effective inference:\n\n\n| Model | Reverse Process Condition | GenPPL on LM1B↓ |\n| -| -| -|\n| SID[1]|$x_t$-only|185.32|\n| CaDDi (Our Model)|trajectorty| 142.51|\n\n\n\n**4. Computational Cost and Additional Analysis**\n\n- **Overhead of Conditioning on Full Sequence**: The reviewer correctly identifies that conditioning on the entire sequence $x_{t:T}$ can introduce computational overhead. In Appendix Section C.1 (\"Context Window and Latent Compression,\" Page 28), we discuss this practical limitation and introduce latent truncation and trajectory re-composition as strategies to compress the latent trajectory to fit within a bounded context window. Specifically, latent truncation (Section C.1.1) limits the conditioning to the most recent m timesteps, which directly controls the computational cost. We have already performed an ablation study on the context window size (Section G.2, Table 5, Figure 9), showing that a moderate window size offers a favorable balance between performance and computational cost. \n- **Quantitative Evaluation of Computational Cost**: Thanks for your suggestions! Below is a detailed quantitative evaluation of this cost on the LM1B dataset during generation. Speculative decoding is a dynamic speed-up technique, so the NFE has some randomness between different runs. As shown in the table below, semi-speculative decoding significantly reduces the NFE for CaDDi-AR while maintaining competitive performance, highlighting its effectiveness as a speedup technique. It's also worth noting that the non-AR version of CaDDi with 64 steps already achieves better performance than MDLM with 1000 NFE (and with larger FLOPs), which demonstrates that the improvement does not purely benefit from scaling up the computation, but also from the methodology (non-Markovian formulation, 2D ROPE, unification with LLMs).\n\n| Model | NFE | GenPPL on LM1B↓ |\n| -| -| -|\n|MDLM-64|64|199.45| \n|MDLM-500|500|188.93|\n|MDLM-1000|1000|179.41|\n|CaDDi|64|142.51|\n|CaDDi-AR|8192|137.82|\n|CaDDi-AR (with semi-sepculative decoding)|~656|139.80|\n\n**5. Theoretical Clarity about the first equality in Equation (6)**\nThe first equality, $p_{\\theta}(x_{t-1}|x_{t:T}):=q(x_{t-1}|x_{t:T}, x_0=\\mu_{\\theta}(x_{t:T},t))$, is a definition (as stated in line 68) based on the standard practice of parameterizing the reverse diffusion process using a prediction of the clean data $x_0$, which is widely used in prior work like D3PM, MDLM, ...In diffusion models, the true posterior $q(x_{t-1}|x_t,x_0)$ is often used as a target for the learned reverse process $p_{\\theta}(x_{t-1}|x_t)$. In our non-Markovian setting, we extend this by conditioning on the future trajectory $x_{t:T}$.\n\n**6. Discussion of Potential Negative Societal Impacts:** \nWe acknowledge the omission of a discussion on potential negative societal impacts and will add a dedicated section.\n\nThanks again for your suggestions! We are confident that addressing these points will significantly strengthen our paper. We are committed to incorporating these revisions and providing any further clarifications needed."}]

Author rebuttal:

Meta-review: The paper presents CaDDi, a non-Markovian discrete diffusion model that lets a single transformer attend to the full denoising trajectory and reuse pretrained causal language-model weights, leading to stronger language generation than prior diffusion baselines. Before the rebuttal, reviewers cgB8, WEg3, and qCqR praised the novelty, unified 2D positional design, and empirical gains, while reviewer CSEv stressed missing predictor–corrector comparisons, higher compute, and architectural changes, so scores were widely split. After the authors supplied clarifications, extra ablations, and new baseline results, the supportive reviewers held their positive stance. However,  CSEv maintains that CaDDi has major disadvantages over predictor-corrector methods (in that it requires re-training) and would not improve over baselines when put on equal FLOP footing.

Overall, the AC concurs with the majority of reviewers - the proposed method is intriguing and promising, and may be inspiring to the community. Hence, the AC recommends acceptance as poster.
Decision: Accept (poster)