Title: Theoretical Benefit and Limitation of Diffusion Language Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Discrete Diffusion Model

Efficiency

Masked Diffusion Model

Language Modeling

Metrics for NLP
Abstract: Diffusion language models have emerged as a new approach for text generation. By enabling the parallel sampling of multiple tokens in each diffusion step, they appear to offer a more efficient alternative to auto-regressive models. However, our observations show that current open-sourced diffusion language models require more sampling steps to achieve comparable accuracy on representative tasks--resulting in even higher inference costs than their auto-regressive counterparts. To investigate whether this is an inherent limitation, we conduct a rigorous theoretical analysis of a widely adopted variant: the Masked Diffusion Model (MDM). Surprisingly, our analysis reveals that the conclusion is highly sensitive to the choice of evaluation metric. Under mild conditions, we prove that when the target is near-optimal perplexity, MDMs can achieve this goal in a constant number of sampling steps, independent of sequence length. This result demonstrates that efficiency can, in principle, be attained without compromising generation quality. However, when targeting low sequence error rate--which is important for assessing the ``correctness" of a generated sequence, such as a reasoning chain--we show that in the worst case, the required sampling steps must scale linearly with sequence length, thereby eliminating the efficiency advantage. Our analysis establishes the first theoretical foundation for understanding the comparative strengths and limitations of MDMs, offering practical guidance on when to favor MDMs over the auto-regressive models and vice versa.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors find that the utility of MDMs is highly dependent on the evaluation metric used. When assessed by token-level accuracy, MDMs can achieve near-optimal results with a constant number of sampling steps, regardless of the length of the sequence. However, when evaluated based on sequence-level correctness, which is critical for tasks like logical reasoning, the required number of sampling steps for MDMs must increase linearly with the sequence length. This research provides a theoretical framework for understanding when to use MDMs, suggesting they are well-suited for tasks where fluency is paramount, while auto-regressive models are better for tasks demanding high accuracy and logical consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The authors study an important problem from a theoretical perspective on the inference of MDMs with comparison to auto-regressive LM.\n2. The authors provide insights on the strength of parallel sampling in terms of two types of methods as token-level independent metric and sequence-level holistic accuracy metric.\n3. The authors carry out empirical evaluation on synthetic tasks (HMM and n-gram) to corroborate the theoretical finding.\n\nWeakness\n1. Though the chosen metrics provide good theoretical finding, it is not directly related to performance of real-world task performance. SER should be too harsh on the success of even reasoning tasks as errors on a few tokens may not have large effects on final accuracy.\n2. It is hard to achieve a fair comparison of MDM and AR models especially on real datasets due to differences in model structure and training to isolate the effect of inference. As the authors suggest, even “AR” generation of MDM still leads to gap in performance compared to true AR models.\n\nQUESTIONS:\nDoes the metrics TER and SER defined in section 4.1 generalized to conditional generation?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the theoretical limitations of masked diffusion models (MDMs). It proposes two evaluation metrics: token error rate and sequence error rate, which aim to quantify the uncertainty in token prediction and the correctness of entire generated sequences, respectively. Through theoretical analysis and toy experiments, the paper shows that while MDMs can achieve low perplexity with few sampling steps, minimizing sequence error requires steps that scale with sequence length.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper analyzes the theoretical limitations of masked diffusion models, which is an important and timely topic in generative language modeling.\n\nWeaknesses:\n- The soundness of the proposed evaluation metric appears questionable. SER is defined as one minus the sum of model probabilities over all sequences in the ground-truth support set $\\\\mathcal{L}\\_q$, regardless of their probabilities under the ground-truth distribution. This leads to unintuitive behavior: if all sequences have non-zero probability under the ground-truth distribution, then SER = 0, regardless of the actual distributional alignment. Consider the following two cases:\n    - 1) The model distribution has the same support as the ground truth but a different probability assignment.\n    - 2) The model distribution is nearly optimal but assigns a small probability $\\\\delta$ outside the support.\nIn these cases, SER would be 0 for the first and $\\\\delta$ for the second, incorrectly favoring the worse model. This raises concerns about whether SER is a reliable or meaningful metric for analyzing language models or their reasoning capabilities.\n- The significance of the theorems is also unclear. Theorems 4.2 to 4.4 provide bounds on the required sampling steps. However, these bounds may be of limited practical value since it's known that MDMs can approximate any ground-truth distribution if the number of sampling steps ≥ the sequence length—because the issue of conditional independence disappears when generating one token per step. (I am not an expert on these proofs and acknowledge the possibility of misinterpretation.)\n- The experimental design is limited to toy settings up to 4-gram synthetic languages. While this may help illustrate the theoretical claims, it does not adequately reflect the complexity of real natural language, limiting the broader applicability of the results.\n\nMinor:\n- Line 279: \\citet\n\nQUESTIONS:\n- How does SER relate to the performance on reasoning tasks?\n- It is mentioned that SER is computed directly using Equation (5). How is the summation over $\\\\mathcal{L}\\_q$ performed in practice, given that the cardinality appears intractable?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work theoretical studies the benefits and limitations of Masked diffusion models for text generation. It first empirically shows that existing autoregressive models such as qwen perform better in terms of efficiency and accuracy than diffusion models such as Llada, which claims better efficiency. It is shown that for n-gram langauges diffusion models can outperform autoregressive models in efficiency with competitive accuracy at token-level metrics such as perplexity. However, when it comes to sequence-level tasks such as correctness of answers or logic, it is shown that there exists an Hidden Markov Model language such that diffusion models show no such advantage over autoregressive models.\n\nExperimental results on synthetic data shows similar pattern as claimed by the theorems, i.e., for sequence-level metrics, the performance of diffusion models are much worse than autoregressive models. However, for token-level metrics the performance of diffusion model looks nearly independent of the number of steps taken by the diffusion model, and the performance of diffusion and autoregressive models look comparable.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper provides interesting theoretical results showing that the performance of emerging diffusion language models are dependent on the nature of the evaluation metrics, in particular, for token level metrics diffusion models show promise of outperforming autoregressive models. But for sequence-level metrics such as correctness of reasoning or complete answer, diffusion model fails to offer benefits compared to autoregressive models. These results are shown for Markov-style language assumption, nevertheless, the results are interesting.\n\n- Experimental results on synthetic data validates both the claims for token-level as well as sequence-level metrics.\n\n\nWeaknesses\n- One weakness already mentioned in the paper is that all theoretical analysis and experimental results are restricted to HMM-style language. However, a) this assumption may not hold for general tasks, b) even if the assumption of HMM holds, tasks like infilling may benefit from diffusion models more. Even if the theoretical analysis for such non-HMM languages may be difficult, the paper must have some experimental evaluations provided to better understand the possible generalization of the results in the paper.\n\n- Fig 1 compares performance of qwen versus Llada models on GSM8k to claim the poor performance of diffusion models compared to autoregressive models. But to better understand the result, we should compare with more available models of comparable sizes such as llama 8b and also test on various other benchmarks such as Math500, HumanEval, MBPP, etc.\n\nQUESTIONS:\nPlease also answer the weaknesses.\n\nAdditional Questions\n-  Line 248: even though it is claimed that the number of samples required is constant. Is it not dependent on the Markovity n of the language model?\n\n- Why is the TER result in Thm. 4.2 only for n-gram language whereas the SER result in Thm. 4 gives an existence result but does not mention the n-gram of the language. Does the result in Thm. 4 claim the existence of an n-gram language for which the theorem holds or is it for any possible Markovian language?\n\n- Since diffusion models perform well on token-level metric but not on sequence level metric. Can we not modify the sampling algorithm for using speculative decoding? That is, use diffusion model as a draft model and use autoregressive to fix sequence level metric. This is a general question. No need for theoretical analysis or experiments. I am trying to understand if the limitation of diffusion models shown a major issue for sequence-level metric or is there a way around that is not discussed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an analysis of the properties of MDMs in the context of language modeling and compares them to autoregressive LMs. According to the paper's findings, the number of steps required to achieve near-optimal perplexity or a reasonable Token Error Rate (TER) for MDMs is relatively constant, regardless of sequence length. Conversely, the number of steps needed to achieve a specific Sequence Error Rate (SER) scales linearly with increasing input sequence length. Theoretical findings are validated on synthetic tasks, and the experimental results confirm these theoretical insights.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n - S1: This contribution could be very important for the field of diffusion language models. Diffusion LMs are believed to be more efficient on code and math, however, this work suggests that the number of steps scales linearly with sequence length to achieve low SER. Therefore, using diffusion for this task could be less efficient compared to AR models.\n - S2: Measuring TER and SER on synthetic tasks could be a useful metric for future research on Diffusion LMs.\n - S3: The paper is easy to follow because of the structure.\n\nWeaknesses\n\n- W1: The main text only includes results on the GSM8K dataset. Adding additional datasets and reporting mean accuracy across them would help address this limitation.\n - W2: The apparent superiority of the AR model in Figure 1 might be explained by differences in the dataset used and the amount of FLOPs during training. \n - W3: Setup where #steps < sequence length should be compared to AR model with lower number of parameters during inference, matching the speed or computational cost (FLOPs) of MDMs like LLaDA with lower number of steps.\n\nQUESTIONS:\nQ1: Does Theorem 4.4 hold for continuous diffusion language models like [1]?\nQ2: Diffusion models are less sample-efficient during training, as each training step involves only a single noise level [2]. Would the results of the experiments in Section 5.1 change if the number of training epochs for MDMs were increased compared to the AR model?\nQ3: Where would be Qwen 2.5 3B and Qwen 2.5 1.5B in Figure 1?\n\n[1] [Likelihood-Based Diffusion Language Models](https://arxiv.org/pdf/2305.18619)\n[2] https://sander.ai/2023/01/09/diffusion-language.html\n\nLIMITATIONS:\nThe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}