Title: Non-Markovian Discrete Diffusion with Causal Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Generative Model

Causal Language Model
Abstract: Discrete diffusion models offer a flexible, controllable approach to structured sequence generation, yet they still lag behind causal language models in expressive power. A key limitation lies in their reliance on the Markovian assumption, which restricts each step to condition only on the current state, leading to potential uncorrectable error accumulation.
In this paper, We introduce CaDDi, a discrete diffusion model that conditions on the entire generative trajectory, thereby lifting the Markov constraint and allowing the model to revisit and improve past states. By unifying sequential (causal) and temporal (diffusion) reasoning in a single non‑Markovian transformer, CaDDi also treats standard causal language models as a special case and permits the direct reuse of pretrained LLM weights with no architectural changes. Empirically, CaDDi outperforms state‑of‑the‑art discrete diffusion baselines on natural‑language benchmarks, substantially narrowing the remaining gap to large autoregressive transformers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a reformulation of discrete diffusion by relaxing the standard Markovian assumption. Specifically, the noising distribution assumes conditional independence given $x_0$ that is $p(x_{t} \\mid x_{1:t-1}, x_0) = p(x_{t} \\mid x_0)$ instead of $ p(x_t \\mid x_{t-1})$ in standard diffusion, thereby removing the dependence on the previous noisy state.\n\nThe work begins by highlighting a key limitation of standard discrete diffusion models: the accumulation of decoding errors due to the strong dependency on previous state in the denoising step $p(x_{t-1} \\mid x_t)$. This dependency exposes the inference procedure to compounding deviations from small prediction errors at each timestep.\n\nTo address this, the authors propose a non-Markovian approach in which the noise at each timestep is conditionally independent given the clean data $x_0 $. That is, each noisy state depends solely on time and the original input, eliminating dependence on previous noisy states.\n\nBuilding on this conceptual framework, the paper introduces the Causal Discrete Diffusion Model (CaDDi). This model incorporates 2D Rotary Positional Encoding to handle the sequential and temporal dimensions inherent to discrete sequences. Furthermore, the authors propose an autoregressive variant, CaDDi-AR, which factorizes the joint distribution across the sequence. To mitigate the associated computational overhead, they introduce a semi-speculative decoding strategy.\n\nThe method is empirically validated through experiments, demonstrating its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is clearly written and easy to follow.\n- The core methodology is intersting and sound.\n- Beyond the central contribution, the paper introduces secondary innovations, including 2D rotary positional encoding, semi-speculative decoding, and leveraging pre-trained language models.\n- The experiments demonstrate some benefits of the proposed approach.\n\n---\n\n## Weaknesses\n\n### 1. Novelty\n\nThe main conceptual claim, namely, that compounding denoising errors in discrete diffusion can be alleviated via a non-Markovian assumption, has been previously introduced in [1]. While [1] focuses on graph generation, the underlying idea and motivation closely align with the present work. Therefore, the novelty of the contribution is limited.\n\n### 2. Positioning and Related Work\n\nThe paper does not adequately position itself within the existing literature. In particular:\n\n- The concept of compounding denoising errors is well-known and explicitly discussed in [2].\n- Several recent works [3–7] propose methods to mitigate such errors through informed or planned denoising and sampling strategies.\n- The relationship between the proposed semi-speculative decoding and prior decoding strategies (e.g., planning, correction, or speculative sampling [1, 3-7]) is not discussed. It remains unclear how the proposed strategy differs from or improves upon these approaches.\n\n### 3. Computational Efficiency and Evaluation\n\nThe proposed method introduces two components that increase computational cost: conditioning on the full sequence and the use of an autoregressive (AR) sampling strategy.\n\n1. The decoder conditions on the entire sequence $x_{t:T}$, which requires computing the full sequence during training. This introduces additional computational overhead compared to standard discrete diffusion conditioning only on $x_t$. However, this overhead is neither discussed nor evaluated in the paper. Furthermore, it is not evident that conditioning on the full sequence provides substantial benefit over conditioning only on $x_t$. An ablation study would be valuable to assess the impact of this design choice.\n\n2. The authors acknowledge the computational cost of their AR method resulting from the high number of function evaluations (NFE) needed. There is however no quantitative evaluation of this cost, nor of the extent to which the proposed semi-speculative decoding mitigates it. Moreover, the experiments do not disentangle the benefits of the method itself from those attributable to increased computational budget. A more detailed analysis is necessary to clarify how much of the observed performance gain results from the method rather than from additional compute.\n\n\n### 4. Theoretical Clarity\n\nThe first equality in Equation (6) is not self-evident. Providing a derivation or proof would be useful\n\n---\n[1] Boget, Y. *Simple and Critical Iterative Denoising: A Recasting of Discrete Diffusion in Graph Generation*. ICML 2025. https://openreview.net/forum?id=kPTW6hGrJy  \n[2] Lezama, J., et al. *Discrete predictor-corrector diffusion models for image synthesis*. ICLR 2023.  \n[3] Liu, S., et al. *Think while you generate: Discrete diffusion with planned denoising*. ICLR 2025.  \n[4] Zhao, Y., et al. *Informed correctors for discrete diffusion models*. https://arxiv.org/abs/2407.21243  \n[5] Kim, J., et al. *Train for the worst, plan for the best: Understanding token ordering in masked diffusions*. https://arxiv.org/abs/2502.06768  \n[6] Peng, F., et al. *Path planning for masked diffusion model sampling*. https://arxiv.org/abs/2502.03540  \n[7] Nie, S., et al. *Large language diffusion models*. https://arxiv.org/abs/2502.09992\n\nQUESTIONS:\n## Suggestions for Improvement\n\n1. The authors should clearly articulate the novel contributions of this work compared to [1], especially given the close conceptual overlap.\n\n2. A thorough literature review is necessary. Related works on compounding denoising errors and advanced discrete diffusion strategies should be discussed in one or more dedicated sections.\n\n3. An empirical analysis of the computational cost (including the effect of semi-speculative decoding) would make the evaluation more rigorous and transparent.\n\n4. A detailed explanation or formal proof of the first equality in Equation (6) would help ensure the mathematical soundness of the method.\n\n## Question:\n\nWhat is the benefit of conditioning on the whole sequence rather than on $x_t$?\n\nLIMITATIONS:\n## Limitations\n\n- The paper does not discuss (or evaluate) the residual computational overhead resulting from the computation of the whole sequence $x_{t:T}$ during training.\n- The paper does not discuss (or evaluate) the residual computational overhead after applying the semi-speculative decoding strategy.\n- The paper does **not** include a discussion of potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 1, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new discrete diffusion framework, CaDDi, with its variant CaDDi-AR. This framework removes the Markovian assumption and regards the noise as independently injected into the clean data. From the new assumption, the author derives an inference process and a specific training objective through the ELBO. Built upon the mathematical formulation and the observation of the connection between non-Markovian diffusion and the causal language model, the author proposes to integrate token order and timesteps into a single transformer model. To account for both dimensions, the authors also propose a 2D Rotary Positional Encoding method. The CaDDi-AR factorizes over the token space to capture the true posterior distribution and uses speculative decoding to reduce the computation. Since the causal language model can be regarded as a special case of the CaDDi-AR, pre-trained LLMs can be fine-tuned to serve as the backbone of the CaDDi-AR. Comprehensive experiments have been done to prove its effectiveness. The experiments include 5 baselines, on 3 open-source LLMs, on a billion-word dataset, Text8 dataset, and a bunch of reasoning benchmarks. The overall experimental results suggest that CaDDi is an effective discrete diffusion framework across many models and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The core idea behind CaDDi is great. It's a non-Markovian approach to discrete diffusion, meaning it conditions the denoising on the entire generation history, not just the last step. This is a well-motivated way to tackle the error accumulation problem that plagues standard Markovian models. The theory holds up, and the training objective is straightforward.\n2. The 2D RoPE is a really clever architectural choice. It neatly handles both token position and the diffusion timestep within a regular decoder-only Transformer. The best part is that this makes it backward-compatible, so you can fine-tune existing LLMs without having to change their architecture.\n3. The authors clearly anticipated the potential for high computational cost with the token-level version (CaDDi-AR). Their \"semi-speculative decoding\" strategy is a smart fix. By using the prediction from the previous timestep as a draft for the current one, it cuts down on sampling steps and makes the whole process more efficient.\n4. The experiments are solid. CaDDi was benchmarked against a bunch of SOTA discrete diffusion models (D3PM, SEDD, MDLM, DFM) and consistently came out on top in both perplexity and reasoning tasks.\n\nWeakness:\n1. My main issue is with the comparison to standard autoregressive models. While CaDDi is clearly better than other diffusion models, it's may inherently slower than a single-pass AR model because it requires T sampling steps. The paper brings up semi-speculative decoding to help with this, but there's no direct, wall-clock time comparison to a similarly-sized AR model. I'm left wondering how practical CaDDi-AR really is. How close does it get to the speed of a fine-tuned AR baseline for the same quality? This is a key question for real-world use.\n2. The non-Markovian design means conditioning on the history of latent states. This could create a massive input sequence for the transformer, especially with a large T. The paper doesn't really dig into how well the model scales with longer context lengths, which seems like an important limitation to explore.\n\nQUESTIONS:\nN/A.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Non-Markovian discrete diffusion framework for text generation. Unlike conventional discrete diffusion models that rely on the Markovian assumption, CaDDi conditions each denoising step on the full generative trajectory.\n\nFurthermore, CaDDi is introduced as a causal version of discrete diffusion, using transformer architecture as a natural bridge. It unifies sequential and temporal reasoning.\n\nExperimental results demonstrate superior generative quality and robustness on several benchmarks (LM1B, Text8, Amazon Polarity, reasoning tasks) compared to existing discrete diffusion baselines, while retaining compatibility with pretrained LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The paper introduces a 2D positional encoding, which is novel.\n- The paper introduces a semi-speculative decoding to speed up inference.\n- Ability to reuse pretrained LLMs for diffusion tasks broadens the applicability of diffusion-based approaches.\n- Demonstrates clear improvements in generative perplexity.\n\nWeaknesses:\n\n- The authors did not point out that the idea of Non-Markovian discrete diffusion is not novel. For example, [1] first propsed a Non-Markovian discrete diffusion, in addition to allowing $p(x_t|x_{t-1}, ... x_{0})$ to be conditioned on the whole sequence, it is also aware of the sequencial and temporal relationship of Non-Markovian discrete diffusion (using transition time $\\tau$ to gain faster inference).\n- Due to the above reason, there is a lack of proper reference to prior work on Non-Markovian discrete diffusion models like DNDM, as well as Non-Markovian diffusion models like DDIM.\n- Despite semi-speculative decoding, it seems that inference remains slower than discrete diffusion models, since the complexity is L*T. The authors did not discuss the remaining computation overhead.\n- The \"VERIFY\" and \"CORRECT\" process in Algorithm 2 has never been defined in this paper. \n- Algorithm 2 is never referenced. It seems like a typo (Line 216 says \"see Algorithm 3\")\n\n[1] Chen, Zixiang, et al. \"Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time.\" NeurIPS 2024.\n\nQUESTIONS:\nIn addition to weaknesses,\n\n1. Could the authors further quantify the computational cost of CaDDi and CaDDi-AR compared to traditional AR models and Markovian discrete diffusion in both training and inference?\n\n2. What is the sequence length in the experiments? Have the authors tested CaDDi on longer sequences, and how does it scale in terms of memory, computation, and quality?\n\nLIMITATIONS:\nI do not see the discussion on limitations in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper proposes a novel diffusion model, called CaDDi, for discrete sequences. Instead of relying only on the previous diffusion step, it leverages all diffusion history to provide a more stable discrete diffusion model. Notably, when the diffusion step T=1 it can be regarded as an ordinary causal language model, thus allowing finetuning existing large language model using the power of diffusion model.
Extensive experimental results indicate that the proposed CaDDi has a better performance than previous discrete diffusion models. While it is slightly behind ordinary causal LLMs without finetuning, this is an important step towards useful diffusion model for discrete sequences.

STRENGTHS AND WEAKNESSES:
See above. I found no clear weaknesses in this approach.
While pages are limited, I would like to see some practical texts where the proposed CaDDi works better than ordinary causal LMs or other discrete diffusion models.

QUESTIONS:
Nothing.

LIMITATIONS:
Nothing.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors argue that the Markovian assumption in discrete diffusion models is a key limitation, as it can lead to error accumulation during sampling. To address this, they propose a novel algorithm called CaDDi, which conditions the denoising model’s predictions on the entire generation trajectory. Specifically, they modify the transformer architecture to incorporate the full diffusion trajectory as input. Empirically, CaDDi achieves competitive performance on natural language benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\nThe semi-speculative decoding sampler proposed in this paper is interesting.\n\n### Weaknesses\n\n1. In masked diffusion models, the state x_t inherently encodes the entire trajectory x_{t:T}, making the decoding sequence largely irrelevant. As shown in Table 2, CaDDi clearly lags behind MDLM in terms of perplexity, suggesting that the additional engineering to incorporate the entire trajectory does not provide a clear benefit.\n\n2. It is unclear why the perplexity of the models on LM1B is not reported, given that the authors trained their model on this dataset. Perplexity remains the gold standard for comparing language models. Generative perplexity is typically used to evaluate samplers.\n\n3. Table 1 is misleading:\n(a) What does T refer to? Is it the sampling temperature or the number of time discretizations? What discretizations were used for the baselines such as MDLM, D3PM, and UDLM?\n(a) What sampler was used for each method? If CaDDi uses the proposed semi-speculative decoding sampler, it would be helpful to report the sample quality from CaDDi using the ancestral sampler as well. This would allow the community to disentangle the improvements due to (1) the non-Markovian trajectory and (2) the semi-speculative decoding sampler, in comparison to MDLM.\n\n4. Table 4 is missing major baselines. The authors should compare their method against MDLM, UDLM [2], and an autoregressive (AR) model trained with the same transformer architecture and a similar number of training steps. They can follow the steps in [2].\n\n\n\nPS. This paper is riddled with typos. A major one is Algorithm 2 being incorrectly referenced Algorithm 3 in line 216.\n\n\n\nReferences\n\n[1] Sahoo et al., 2024, \"Simple and Effective Masked Diffusion Language Models\"\n\n[2] Schiff et al., 2025, \"Simple Guidance Mechanisms for Discrete Diffusion Models\"\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nsee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-09"}