Large Language Diffusion Models
Authors: Shen Nie
Fengqi Zhu
Zebin You
Xiaolu Zhang
Jingyang Ou
Jun Hu
JUN ZHOU
Yankai Lin
Ji-Rong Wen
Chongxuan Li
Venue: NeurIPS 2025 oral ()
Keywords: diffusion language models
large language models
masked diffusion models
discrete diffusion models
diffusion models
Primary area: deep_learning
TL;DR: We present LLaDA, a diffusion  language model trained from scratch that is competitive to LLaMA 3 in performance.

Abstract: The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing *LLaDA*, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong *scalability* and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in *in-context learning* and, after SFT, exhibits impressive *instruction-following* abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: \url{https://ml-gsai.github.io/LLaDA-demo/}.

Large Language Diffusion Models
Shen Nie! 12,3+ Fengqi Zhu 122,+ Zebin You' 12,3f Xiaolu Zhang
Jingyang Ou 122,3
Jun Hu 11 Jun Zhou
Yankai Lin3.2,32 Ji-Rong W..3.2.3 Chongxuan Li
Gaoling School of Artificial Intelligence, Renmin University of China
2 Beijing Key Laboratory of Research Large Models and Intelligent Governance
3 Engineering Research Center of NexI-Generation Intelligent Search and Recommendation, MOE
Ant Group
Mrieshen,,engee zhu, thhngeg -rruc.euu.u.. edu.
Abstract
The capabilities of large language models (LLMs) are widely regarded relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, diffusion model trained from scratch under the pre-training and supervised fine- tuning (SFT) paradigm. LLaDA employs forward data masking process and a reverse generation process. parameterized by a Transformer to predict masked tokens. It provides principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B competitive with strong LLMs like LLaMA3 8B in in-contexi learning and, after SFT, exhibits impressive instruction- followiig abilities in case studies such as multi- turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml- gsai. ....githhe github. LeoLLaaD-demo//
Introduction
Large language models (LLMs) fall entirely within the framework of generative modeling. Specifically, I.L.s aim to capture the true but unknown language distribution Paata-) by optimizing a model distribution (-()) through maximum likelihood estimation, or equivalently KI. divergence minimization between the two distributions:
max E (2) log po(x) + min (L (z) (o()))
modeling principles
The predominant approach relies on the autoregressive modeling (ARM)-commonly referred as the "next-token prediction" paradigm-to define the model distribution:
(=(x) = (a(x
II (o(a"
x
(2)
Autoregressive formulation
'Equal contribution.
Work done during internship at Ant Group.
1 Project leaders.
"Correponndecce to Chongxuan Li.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Genoeal TANAS
Mestettttic
ARC-C
Omnond Tadss
Malk
Mathrmatios
MMLU
MMLLI
LLaDA ES Busc LLaMA
LLaMA #B Iiotact
Mahh
LLaMA 719 Insrrct
LLaMA
G-bool
XBB
Code
MBR
BatseEval
Chimase
Figure 1: Zero/Few-Shoo Benchmarks. We scale LLaDA to 8B parameters from scratch and observe competitive zero/few-shot performance compared with strong autoregressive LLMs 66.
where x is a sequence length L, and x" is the i-th token. This paradigm has proven remarkably effective [255 and has become the foundation of current LLMs. Despite its widespread adoption, fundamental question remains unanswered: the autoregressive parudigm the only path to achieving the core capabilities of LLMs, such as scalability, in-context learning. and instuuctionfoolloinng
We argue that the answer is not a simple "yes". The key insight overlooked previously is: is the generative modeling principles (i.e., Eq. ① rather than the autoregressive formulation (i.e., Eq. (2 itself, that fundamentally underpin the essential properties of L.LMs.
In particular, we argue that scalability primarily a consequence of the interplay between Trans- formers (7), model size, data size, and Fisher consistency 8 induced by the generative principles in Eq. I rather than a unique result of the ARMs in Eq. 2 The success of diffusion trans- visual data (TT] supports this claim. Furthermore, the instructionnfollloinng and in-context learning [ capabilities appear to be intrinsic properties of all conditional generative models on structurally consistent linguistic tasks, rather than exclusive advantages of ARMs. In addition, while ARMs can be interpreted as a lossless data compressor expressive probabilistic model can achieve similar capabilities [4]
formers 9
[2] 13
any sufficiently
However, certain inherent limitations of LLMs can be directly attributed to their autoregressive nature. For instance, the left-to-right generation process restricts their ability to handle reversal reasoning tasks 15). highlighting a representative failure in the generalization capabilities of current models.
Motivated by these insights, we introduce LLaDA (Large Language Diffusion with mAsking) to inves- tigate whether the capabilities exhibited by LLMs can emerge from generative modeling principles beyond ARMs, thereby addressing the fundamental question posed carlier. In contrast to traditional ARMs, LLaDA leverages a masked diffusion model (MDM) [16) which incorporates a forward data masking process and trains a mask predictor approximate its reverse process. This design enables LLaDA to construct a model distribution with bidirectional dependencies and optimize variational lower bound of its log-likelihood, offering principled and previously unexplored perspective on the core capabilities of LLMs discussed above.
We adopt the standard pipeline of data preparation, pre-training, supervised fine-tuning (SFT), and evaluation, scaling LLaDA to an unprecedented language diffusion of size SB. In particular, LLaDA 8B was pre-trained from scratch on 2.3 trillion tokens using 0.13 million H800 GPU hours, followed by SFT on 4.5 million pairs. Across diverse tasks, including language understanding, math, code, and Chinese, LLaDA demonstrates the following contributions:
LLaDA scales effectively compute budget of 20 FLOPs, achieving comparable results to ARM baselines trained on the same data across six tasks, e.g., MMLU and GSM8K.
'It suggests the ability to recover the true data distribution with infinite data, a sufficiently large network and optimal training.

---

[a]
Mask all lakens independenty
(b)
P Prompt
Response
Prompt
Responsa
Mask ratio
prodictor
prodictor
prodictor
Remasic
Non-mask tokan
Random mask
Overview of LLaDA. (a) Pre-training. LLaDA is trained text with random masks applicd independently to all tokens at the same ratio t ~ U10, 1]. (b) SFT. Only response tokens are possibly masked. (c) Sampling. LLaDA simulates diffusion process from i (fully masked) to == (unmasked), predicting all masks simultaneously at each step with flexible remask strategies.
Figure 2:
The pre-trained LLaDA 8B Base surpasses LLaMA2 7B Base 21 on nearly all 15 standard zero/few-shot learning tasks while performing on par with LLaMA3 8B Base 6 showcasing effective in-context learning capability.
LLaDA significantly enhances the ability to follow instructions after SFT, demonstrated in case studies such as multi-turn dialogue.
LLaDA effectively breaks the reversal curse [15 with consistent performance across forward and
reversal tasks. Notably, it outperforms GPT-4o a reversal poem completion task.
Approach
In this section, we introduce the probabilistic formulation along with the pre-training, supervised fine-tuning, and inference procedures for LLaDA, as illustrated in Fig. 2
2.1 Probabilistic Formulation
Unlike ARMs in Eq. (2.. LLaDA defines a model distribution po (zo) through a forward process and a reverse process [16420]. The forward process gradually masks tokens independently in ao until the sequence is fully masked at / 1. For / E (0, 1). the sequence is partially masked, with each being masked with probability l remaining unmasked with probability
L. The reverse process recovers the data distribution by iteratively predicting masked tokens as / moves from 1 to 0.
The core of LLaDA is mask predictor, parametric model pa po(- that takes Iı as input and predicts all masked tokens (denoted as M) simultancously. It is trained using a cross-entropy loss computed only on the masked tokens 18 20
L
((0) A -E Er...
M
| M] log po
where æo is a training sample, is a continuous random variable drawn uniformly from [0, 1], 2e is sampled from the forward process and L is the sequence length. The indicator function 1[-] ensures that the loss computed only for masked tokens.
Once trained, we can simulate a reverse process (see Sec. 2.4e for details) parameterized by the mask predictor and define the model distribution (a())) the marginal distribution induced 0. The loss function in Eq. (3) has been proven to be an upper bound on the negative log- likeliioddood of the model distribution, making it principled objective for generative modeling:
Ex))(a)) [log (r()) < C(0).
(4)
Notably, LLaDA employs masking ratio that varies randomly between 0 and while BERT 22 uses a fixed ratio. The subtle differences have significant implications, especially at scale: as shown in
'Here we focus on the approach of LLaDA. A rigorous formulation of MDM is provided in Appendix A for interested readers.

---

Eq. 4. LLaDA is a principled generative model with the potential to perform in in-context learning and naturally, akin LLMs. Moreover, its generative perspective implies strong scalability with large data and models as discussed in Sec. In addition, MaskGIT [23] adopts heuristic training objective, which misses the y term compared to Eq. (3). and lacks a theoretical link to maximum likelihood. We emphasize that it is precisely the theoretical foundation of maximum
likelihood estimation that motivated us to scale discrete diffusion models for language modeling.
2.2 Pre-training
LLaDA employs a Transformer 7 as the mask predictor, similar to existing LLMs. However, LLaDA does not use a causal mask, as its formulation allows it to see the entire imput for predictions.
We trained two variants of LLaDA with different sizes: IB and 8B. We summarize the model architecture of LLaDA 8B and LLaMA3 8B [6] here, and details are provided in Appendix B.2 We
have ensured consistency in most hyperparameters while making several necessary modifications. We
use vanilla multi-head attention instead of grouped query attention 24 for simplicity, LLaDA is
incompatible with KV caching, resulting in a different number of key and value heads. Consequently, the attention layer has more parameters, and we reduce the FFN dimension maintain a comparable model size. Additionally, the vocabulary size differs due to a tokenizer 4 adapted on our data.
The LLaDA model is pre-trained a dataset comprising 2.3 trillion (T) tokens, adhering to data protocol that aligns closely with existing LLMs [25] 26). without the incorporation of any special techniques. The data are derived from online corpora, with low-quality content filtered through manually designed rules and LLM-basced approaches. Beyond general text, the dataset encompasses high-quality code, math, and multilingual data. Please refer to Appendix for more details about datasets. The mixing of data sources and domains is guided by scaled-down ARMss. The pre-training process utilizes fixed sequence length of 4096 tokens, incurring a total computational cost of 0.13 million H800 GPU hours, similar to ARMs of the same scale and dataset size.
For training sequence To, we randomly sample L E [0, 1], mask each token independently with the same probubility to obtain zz (see Fig. 2 (a)) and estimate Eq. (3) via the Monte Carlo method for stochastic gradient descent training. In addition, following Nie et al. (27), to enhance the ability of LLaDA to handle variable-length data, we set 1% of the pre-training data to a random length that is uniformly sampled from the range [1, 4096).
We adopted the Caammup-Stable-eeee
(2 learning rate scheduler to monitor the training progress without interrupting continuous training. Specifically, we linearly increased the learning rate from to x 10-4 over the first 2000 iterations and maintained it at 4 x 10-4, After processing 1.2T tokens, we decayed the learning rate to 1 x 10-1 and held it constant for the next 0.8T tokens to ensure stable training. Finally, we linearly reduced the learning rate from x 10-1 to x 10-5 for the last 0.3T tokens. Furthermore, we utilized the AdamW optimizer (29) with weight decay of 0.1, a batch size of 1280, and a local batch size of 4 per GPU. The 8B experiment was executed once, without any hyperparameter tuning.
2.3 Supervised Tine-Tuning
We enhance the capability of LLaDA to follow instructions by supervised fine-tuning (SFT) with paired data (Po. ro), where Po is the prompt and ro denotes the response. This is the simplest and most basic post-training method for LLMs. Technically, this requires to model the conditional distribution (r (Po) instead of po (Tro in pre-training.
The implementation is similar to pre-training. As shown in Fig. 2 (b). leave the prompt unchanged and mask the tokens in the response independently, as done for xo. Then, we feed both the prompt and the masked response rr to the pre-trained mask predictor to compute the loss for SFT:
L'
E Et.po-rost
Σ
1 = M] log pog re)
(5)
where Lt' denotes a dynamic length specified later, and all other notations remain the same as before.
Note that this approach is fully compatible with pre-training. Essentially, the concatenation of po and ro can be treated as clean pre-training data 2o. while the concatenation of po and Te serves the

---

masked version The process is identical to pre-training, with the only difference being that all masked tokens happen appear in the ro portion.
The LL 8B model undergoes SFT on a dataset comprising 4.5 million pairs. Consistent with the pre-training process, both data preparation and training follow the SFT protocols utilized in existing LLMs [25 26], without introducing any additional techniques to optimize LLaDA's performance. The dataset spans multiple domains, including code, mathematics, and instruction-followii append |EOS| tokens to the end of short pairs in each mini-batch to ensure equal lengths across all data. We treat EOS) as normal token during training and remove it during sampling, enabling LLaDA to control the response length automatically. Please refer to Appendix B. for more details.
We
We train for 3 epochs the SFT data using similar schedule to the pre- pre-training phase. The learning rate is linearly increased from O to 2.5 x 10-5 over the first 50 iterations and then kept constant. During the final 10% of iterations, it is linearly reduced to 2.5 10-6, 6 Additionally, we set the weight decay to 0.1, the global batch size to 256, and the local batch size to 2 per GPU. The SFT experiment was executed once, without any hyperparameter tuning.
2.4 Inference
As a generative model, LLaDA can sample new text and evaluate the likelihood of candidate text in a diffusion manner instead of the left-to-right autoregressive fashion.
We begin with the reverse generation process. As illustrated Fig. (c), given a prompt po. we discretize the reverse process to sample from the model distribution pa (rolpo))), starting from a fully masked response. The total number of sampling steps is hyperparameter, which naturally provides LLaDA with trade-off between efficiency and sample quality, as analyzed in Sec. Sec. We employ uniformly distributed timesteps by default. In addition, the generation length is also treated as hyperparameter, specifying the length of the fully masked sentence at the beginning of the sampling process. After generation, tokens appearing after the [OO) token are discarded. As detailed in Appendix B.5 since both pre-training and SFT conducted using datasets with variable lengths, the final results are insensitive to this length hyperparameter.
At an intermediate step from time E (0, to E (0, we feed both po and Ta into the mask predictor and predict all masked tokens simultaneously. Subsequently, we remask & of the predicted tokens in expectation to obtain rs, ensuring that the transition of the reverse process aligns with the forward process for accurate sampling [18220] In principle, the remasking strategy should be purely random. However, inspired by the annealing tricks of sampling in LLMs 4 [3]. adopt a low-confidence remasking strategy, where & of predicted tokens with the lowest confidence are remarked based on the predictions, same as the approach of Chang et al. [23
We mention that LLaDA enables flexible sampling. particular, it supports autoregressive and block diffusion [3I] sampling directly after the pre-training or SFT processes described above, without requiring any further modifications or training. We provide detailed analysis in Appendix [B.4 Nevertheless, the diffusion sampling (i.e., the reverse generation process) yields the best performance and is adopted as the default throughout this paper, especially for all experiments presented in Sec.
For conditional likelihood evaluation, we can naturally utilize the upper bound in Eq. (5. However, we find that the following equivalent form 20 exhibits lower variance and is more stable:
I Σ 7
1 = Mog
(6)
where L is the sequence length of ro. / is uniformly sampled from {1. 2.... L}. and r is obtained by uniformly sampling tokens from ro without replacement for masking.
We present the training and inference algorithms, along with theoretical details, in Appendix A
Experiments
We evaluate the scalability, mnollui,ionffollowingg and in-context learning capabilities LLaDD on standard benchmarks, followed by analyses and case studies to provide comprehensive assessment.
5

---

60
60
60
(S.-hot) 4hoc))
LLabb
ARC-C (0. shoo))))
Bacline
CMMLU (5-shos)
-0
-0
MMLU
40
+0
40
30
20
30
20
20
10
10
10
10
10"
10
10
10
10
10"
10
10
FLOPs
FLOPs
FLOPs
60-
24
-Asarapneeee Hanith
HumanEval (0-shot)
e-Assggrrain Buclao
PIQA (O. chot)))
+LLabA
GSM8K (4-shol)
+LaaA
70
40
16
60
20
0
10
10"
10
10"
10
10
10"
FLOPs
FLOPs
FLOPs
Figure 3: Scalability of LLaDA. We evaluate the performance LLaDA and our ARM baselines trained on the same data across increasing pre-training computational FLOPs. LLaDA exhibits strong scalability, matching the overall performance of ARMs on six tasks.
3.1 Scalability of LLaDA on Language Tasks
We first investigate the scalability of LLaDA on downstream tasks comparison with the ARM baselines we constructed. Specifically, at the 1B scale, we ensured that LLaDA and ARM shared the same architecture, data, and all other configurations. At larger scales, we also report results for LLaDA and ARM models of slightly differrnt sizes trained limitations. Please refer to Appendix B.2 for more details. We use the pre-training computational cost unified scaling metric. For evaluation, focused on six standard and diverse tasks.
the same data due to resource
Fig- shows that LLaDA demonstrates impressive scalability, with its overall trend highly competitive with ARMs. Notably, on tasks such as MMLU and GSM8K, LLaDA exhibits even stronger scalability. Even on relatively weaker tasks like PIQA, the performance gap with ARMs narrows as scale increases. To account for the influence of outliers, we opted not to fit quantitative curves, avoiding potential misinterpretation. Nevertheless, the results clearly demonstrate the scalability of LLaDA.
Considering LLaDA's advantages on certain benchmarks, we hypothesize that this performance gain stems from key architectural difference: while autoregressive models optimize only left-to-right conditional probabilities, LLaDA is trained to consider multiple conditioning directions, as detailed in Appendix A.2, which may offer greater flexibility and lead to better generalization. This hypothesis motivated by LLaDA's strong performance on reversal reasoning in Sec. 3.3 and the ablation studies on sampling strategies in Appendix BB. B.4
Nie et al. 27 suggests that MDM requires 16 times more computation than ARM to achieve the same likelihood. However, key differences make findings more broadly applicable. In particular, likelihood is a relatively indirect metric for downstream task performance, and diffusion optimizes a bound of the likelihood, making it not directly comparable ARM. Additionally, we extended the scaling range from 18 1020 FLOPs in Nie et al. (27 to 1020 1023 FLOPs in this work.
3.2 Benchmark Results
To comprehensively evaluate the in-context learning and instruction folllwin capabilities of LLaDA 8B, we conducted detailed comparisons with existing LLMs 21
26
32
33 of similar scale.
Task selection and evaluation protocols followed existing studies, covering popular benchmarks in general tasks, mathematics, code, and Chinese. Further details provided in Appendix [B.6] For more direct comparison, we re-evaluated representative LLMs 6 (1 in our implementation.
As shown in Tab. after pretraining on 2.3T tokens, LLaDA 8B Base demonstrates remarkable performance, surpassing ILaMA2 7B Base on nearly all tasks, and is overall competitive with LLaMA3 8B Base. LLaDA shows advantages in math and Chinese tasks. We conjecture that the

---

Table Benchmark Results of Pre-trained LLMs. indicctes that models evaluated under the same protocol, detailed in Appendix B.6 Results indicated by and are sourced from Yang et al. 25 26 and Bi et al. [32 respectively. The numbers in parentheses represent the number of shots used for in-context learning. "_" indicates unknown data.
LLaDA 8B"
LLaMA3 SB"
LLaMA2 7B"
Qwen2 7B
Qwen2.5 7B!
Mistral 7B'
Deepseck 7B*
Model
Diffusion 2.3T
AR
AR 2T
AR
AR
AR
AR 2T
Training tokens
General Tasks
MMEU
65.9
65)
(5)
70.3
74.2 (5)
64.2
48.2 (5)
BBH
49.7 (3)
62.1 (3)
(3)
62.3 (3)
70.4 (3)
56.1 (3)
39.5 (3)
ARC-C
53.1 (0)
(25)
(25)
(25)
(0)
Hellaswag
70.5 (0)
79.1 (0)
76)0
80.7 (10)
80,2 (10)
83.3
75.4 (0)
TruthfulQA
46.1 (0)
44.0
39.0 (0)
54.2 (0)
$6.4 (0)
42.2 (0)
WinoGrande
74.8
77.3 (5)
72.5 (5)
(7.0
75.9
78.4
70.5
PIQA
73.6 (0)
80.6 (0)
79.1 (0)
79.2 (0)
Mathematics & Science
GSMBK
70.3 (4)
48.7 (4)
13.1 (4)
80.2 (4)
85.4 (4)
36.2 (4)
17.4 (8)
Math
31.4 (4)
16.0
4.3 (4)
43.5
49.8
10.2
6.0 (4)
GPQA
25.2 (5)
25.9 (5)
25.7 (5)
30.8 (5)
36.4 (5)
24.7 (5)
Code
HvaanEval
35.4 (0)
34.8 (0)
12.8 (0)
$1.2 (0)
$7.9 (0)
29.3 (0)
26.2 (0)
HumunEval-FIM
73.8
73.3 (2)
26.9
MBPP
40.0 (4)
48.8 (4)
23.2 (4)
64.2 (0)
74.9 (0)
51.1 (0)
39.0 (3)
Chinese
CMMLU
69.9
50.7 (5)
32.5 (5)
83.9 (5)
47.2 (5)
C-Eval
70.5
51.7 (5)
34.0
83.2 (5)
45.0
Table Benehmark Results of Post-trained L.MMs. LLaDA only employs an SFT procedure, while other models have extra reinforcement learning (RL) alignment. indicates models are evaluated under the same protocol, detailed in Appendix B.6 Results indicated by f and are sourced from Yang et al. (26] and Bi et al. 32 respectively. The numbers in parentheses represent the number of shots used for in-context learning. "_ indicates unknown data.
LLaDA 8B"
LLaMA3 8B
LLaMA2 7B
Qwen2 7B
Qwen2.5 7B
Gemma2 9B²
Deepseck 7B
Model Training tokens
Diffusion 2.3T
AR
AR
AR
AR
1ST
7T
18T
2T
Post-training
SFT
SFT+RL
SFT+RL
SFT+RL 0.5M
SFT+RL
SFT+RL
SFT+RL 1.5M
Alignment pairs
4.5M
1M + D.15M
General Tasks
MMLU
65.5 (5)
68.4
44.1 (5)
49.4
MMLU-pro
37.0 (0)
41.9 (0)
4.6 (0)
44.1 (5)
56.3
52.1
Hellaswag
74.6
75.5 (0)
51.5 (0)
68.5 (-)
ARC-C
88.5 (0)
82.4
57.3 (0)
49.4
Mathematics & Science
GSMBK
69.4 (4)
78.3 (4)
29.0 (4)
85.7 (0)
91.6 (0)
76.7 (0)
63.0 (0)
Math
29.6
3.8
75.5 (0)
44.3 (0)
GPQA
33.3 (5)
31.9
28.4 (5)
34.3 (0)
36.4
32.8 (0)
Code
HumanEval
49,4 (0)
59.8 (0)
16.5 (0)
79.9 (0)
84.8 (0)
68.9 (0)
48.2 (-)
MBPP
41.0.4)
57.6
20.6 (4)
67.2
(0)
74.9 (0)
35.2
strengths stem from the same factors as its relatively weaker performance in some lasks-differencee data quality and distribution, largely due to the closed-source situation LLM datasets.
Notably, we have carefully ruled out the possibility data leakage by taking GSMSK as an example. First, shown in Fig. 3 LLaDA outperformed ARM baselines regarding GSM8K. Moreover, the conclusion remains on a fully unseen GSM8K like task (4] in Appendix B.8
Further, Tab. compares the performance of LLaDA 8B Instruct with existing LLMs. SFT improved LLaDA's performance on most downstream tasks. few metrics, such as MMLU, showed declines, possibly due to the suboptimal quality of the SFT data. Overall, since we did not perform alignment

---

Table 3: Visualization of the Sampling Process and Generated Multi-round Dialogue. In the response of LLaDA, darker colors indicate tokens predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.
Sampling Process
User
Lily can run 12 kilometers per hour for 4 hours. After that, she runs 6 kilometers per hour. How many kilometers she in hours?
LLaDA
in
hours
Lily
runs
12
48
kilometerss
After
hours
she
runs
24
kilometers
total
she
runs
48
72
kilometers
hours
Result
Multi round Dialogue Case
User
Could you please share with me what the initial two lines of the renowned poem "The Road Not Taken"?
LLaDA
Certainly! The initial two lines of the renowned poem "The Road Not Taken" by Robert Frost are: "Two roads diverged a yellow wood, And sorry could not travel both." These lines set the stage for the poem, introducing the idea a speaker facing a choice between two paths.
User
Please help me translate into Chinese. Sure! The first two lines of "The Road Not Taken" by Robert Frost can be translated into Chinese as: "W' A 22 di E 0. 21 Please translate into German.
LLaDA
User
LLaDA
Sure! The first two lines of "The Road Not Taken" by Robert Frost be translated into German as: "Zwei Wege trennten sich im gelben Wald, und leider konnte ich nicht beide tun."
User
Please also write poem about life choices. I require sentences, and the first word each sentence starts with as the letter.
LLaDA
Certainly! Here's poem about life choices, with each sentence starting with the letter "C": Choices cascade through life's journey. Creating connections with each decision. Careful consideration leads to better paths, Courageous choices shape our destiny, Choosing wisely guides us through the years.
with reinforcement learning (RL). our results are slightly behind LLaMA3 8B Instruct, though the gaps in many metries remain small. Notably, even with only SFT, LLLDA demonstrates impressive instruction-followinng abilities, as detailed in Sec. 3.4 We leave RL-based alignment for future work.
All results in Sec. are based on pure diffusion methods, as they achieve better overall performance than approaches incorporating autoregressive components. Specifically, we use Eq. (6) for conditional likelihood estimation and apply low-confidence remasking for sampling. For LLaDA 8B Instruct, block diffusion style sampling performs better on GSMSK and Math, with scores of 78.6 and 42.2, compared 69.4 and 31.9 in Tab. This gain is due to extensive [EO|| token padding in the SFT data, causing early termination in confideiee remasking. Please refer to Appendix B.4 for details.
Overall, despite the lack of data transparency, we have made every effort to adopt standardized procedures and introduce diverse tasks, we believe they sufficiently demonstrate the extraordinary capabilities of LLaDA, which the only competitive non-autoregressive model to our knowledge.
3.3 Reversal Reasoning and Analyses
To quantify the reversal reasoning [15] ability models, we follow the protocol established in Allen- Zhu and Li [35]. Specifically, we construct dataset of 496 famous Chinese poem sentence pairs, Given sentence from a poem, models are tasked with generating the subsequent line (forward) or the preceding line (reversal) without additional fine-tuning. Examples can be found in Appendix B.9 This setting provides a straightforward and more realistic evaluation compared to previous studies [7 36
As shown in Tab, LLaDA effectively addresses the reversal curse [15] demonstrating consistent zero-shot performance across both forward and reversal tasks. contrast, both Qwen 2.5 and GPT-4o exhibit significant gap between the two. The results on forward generation confirm that both ARMs are strong, benefiting from significantly larger datasets and greater computational resources than LLaDA. However, LLaDA outperforms both by large margin in the reversal task.
8

---

We did not design anything special for reversal tasks. Intuitively, LLaDA treats tokens uniformly without inductive bias, leading to balanced perfor- mance. See Appendix A.2 for details.
Table Comparison on the Poem Comple- tion task.
Forward
Reversal
We also analyze the effect of different sampling strategies for LLaDA, including autoregressive sampling, block diffusion [3]] sampling, and pure diffusion sampling, showing that pure diffusion sampling achieves the best overall performance, as detailed in Appendix[B.4
GPT-4o (2024-08-06)
82.7
34.3
Qwen2.5-7B Instruct
75.9
38.0
LLaDA-8B Instruct
51.8
45.6
In addition, we examine LLaDA's sampling speed and memory consumption, showing that it enables flexible trade-off between generation quality and speed. See Appendix B.7 for more details.
Classifier-free guidance (CFG) [3 27 is widely used technique in diffusion models improve generation quality. To ensure fair comparison with ARMs, we do not apply CFG to LLaDA in the main text. However, we show that LLaDA is compatible with CFG and consistently benefits from its application. See Appendix B.3 for more details.
3.4 Case Studies
We present samples generated by LLaDA 8B Instruct in Tab. showcasing its instruction- followiin capabilities. First, the table illustrates LLaDA's ability generate coherent, fluent, and extended text in a non-autoregressive manner. Second, it highlights the model's multi-turn dialogue capability, effectively retaining conversation history and producing contextually appropriate responses across multiple languages. Such chat capabilities of LLaDA are impressive, as it departs from conventional ARMs for the first time, the best of our knowledge. See more case studies in Appendix B.10
Related Work
Diffusion models (38-00 48]40 have achieved remarkable success in visual domains but remain unverified for large-scale (c.g., models trained with over 102h FLOPs) language modeling, despite growing interest and extensive research efforts.
A simple approach is to continuousize text data and apply continuous diffusion models directly 41
51.
Alternatively, some methods model continuous parameters of discrete distributions instead [52
56 However, scalability remains a significant challenge for these approaches. For instance, a 1B
model may require 64 times the compute of an ARM to achieve comparable performance (57).
Another approach replaces continuous diffusion with discrete processes featuring new forward and reverse dynamics, leading to numerous variants (1) The original diffusion model paper [38] introduced both continuous-ste
and discrete-state transition kernels under a unified diffusion framework. Austin et al. [16] was among the pioneering works that introduced discrete diffusion models into language modeling, demonstrating the feasibility this approach. Lou et al. (17] showed that masked diffusion, or surpassing ARMs at GPT-2 scale. Shi et al. [18), Sahoo et al. [19], Ou et al. (20) established fundamental theoretical results, which motivated our model design, training, and inference (see Appendix for details). Nie et al. [27] introduced the scaling laws for MDMs in language modeling and explored how MDMs can be leveraged for language tasks such question answering the GPT-2 scale. Gong et al. 722 demonstrated the potential of fine- tuning an ARM within the MDM framework. However, the improvements observed by Gong et al. 722 are limited to specific metrics, and their approach does not address the performance achievable through pure diffusion- based training. Concurrent work [73 demonstrates the potential of diffusion language models in code generation and highlights their advantages in inference efficiency. Nonetheless, as it is closed-souee source product, specific details such as training procedures and sampling methods remain unknown.
special case of discrete diffusion, achieves perplexity comparable to
In comparison, this study scales MDM to an unprecedented size of 8B parameters from scratch, achieving performance comparable to leading L.LMs such as LLLMA 3.
Additionally, a parallel line of work on image generation 23
7 aligns well with the application
of MDMs to text data. Morcover, MDMs have also shown promise in other domains such as protein

---

generation 776 77. where they have achieved promising results. Notably, series of studies [31 78 87 have explored techniques such as architectural optimization, distillation, and sampling algorithm design to accelerate MDMs sampling.
5
Conclusion and Discussion
We introduce LLaDA, diffusion language model trained from scratch with an unprecedented scale of 8B parameters. LLaDA demonstrates strong capabilities in scalability, in-context learning, and instruction-followingg achieving performance comparable to strong LLMs such as LLaMA3. In addition, LLaDA offers unique advantages, such bidirectional modeling and enhanced robustness, effectively addressing the relevant limitations of existing LLMs. Our findings show the promise of diffusion models for language modeling scale and challenge the common assumption that these essential capabilities are inherently tied to ARMs. These results represent a new paradigm for language modeling and uncover novel insights, demonstrating high degree of scientific innovation.
Limitations. While promising, the full potential of diffusion models remains to be fully explored. Several limitations of this work present significant opportunities for future research. The generation length is a user-specified hyperparameter. Although LLaDA is insensitive to this hyperparameter detailed in Appendix B.S we believe that adopting adaptive generation length would offer a more efficient solution. Due to computational constraints, direct comparisons between LLaDA and ARMs such training identical datasets were restricted
computational budget of less than 10 FLOPs. To allocate resources for training the largest possible LLaDA model and showcasing its potential,
unable to scale the ARM baseline to the same extent. Moreover, no specialized attention mechanisms or position embeddings were designed for LLaDA, nor were any system-leve. architectural optimizations such as KV cache applied. On the inference side, more efficient and controllable (37 88 89) sampling algorithms remain preliminary. Furthermore, LLaDA has yet undergo alignment with reinforcement learning (90 91] which is crucial for improving its performance and alignment with human intent.
Looking ahead, both the model scale and the amount of training data for LLaDA remain smaller than those leading ARM counterparts [6] 26 92 155,, highlighting the need for further scaling to fully evaluate its capabilities. In addition, LLaDA's ability to process multi-modal data remains unexplored. Its impact on prompt tuning techniques [96] and integration into agent- agent-based systems (97 98] is still not fully understood. Finally, systematic investigation into post- trst-nraining for LLaDA (e.g.. O1-like systems 999 100 is needed to further unlock the potential of diffusion language models.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 924701 18); Beijing Natural Science Foundation (No. L247030); Beijing Nova Program (No. 20220484044); the Beijing Major Science and Technology Project under Contract no. Z251 100008425002; the fund for building world-class universities (disciplines) of Renmin University of China; and Ant Group Research Fund.
References
[1) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 2023.
[2] Alec Radford. Improving language understanding by generative pre-training, 2018.
[3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[4] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[5] OpenAl. ChatGPT: Optimizing Language Models for Dialogue. OpenAl blog, November 2022. URL http:://///paaaa.cmoooo///gg///
10

---

[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
[7] Ashish Vaswani. Attention is all you need. arXiv preprint 1rXiv:170..03770000
[8] Ronald A Fisher. On the mathematical foundations theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594-604)3399-38
1922.
[9] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22669-226777 2023.
[10] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195 4195-4205, 2023.
[11] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL httts::/
com/research/
Sideo-generation moooommoools---ooooooeo-ooo as- world- simulators
[12] Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. In The Twelfth International Conference on Learning Representations.
[13] Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelli- gence linearly. arXiv preprint arXiv:2404.0999 099377, 2024.
[14] Claude Elwood Shannon, A mathematical theory of communication. The Bell system technical journal, 27(3):379-422) 1948.
[15] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" is b" fail to learn" b a". arXiv preprint arXiv:2309. /2288, 2023.
[16] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 1ee17991-19999 2021.
[17] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023.
[18] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.0443 2024.
[19] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint 24ti::2400.072222 2024.
[20] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.037366 2024.
[21] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[22] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018
[23] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
11

---

[24] Joshua Ainslie, James Lee-Thorp. Michiel de Jong. Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi- head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural 2023.
Language Processing, pages
[25] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Kegin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https: lbtps://arxive org/ abs/ 220/2240..20 10671
[26] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15 2024.
[27] Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arX1v:2410.1851, 2024.
[28] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Miniepm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395. 2024.
[29]
Loshchilov. Decoupled weight decay regularization. arXiv preprint
2017.
[30] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 1904v:1904.099 2019.
[31] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573 2025.
[32] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek 01:: Scaling open-source language models with longtermism. arXiv preprint 1xXiv:2401.02952 2024.
[33] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0688 2023.
[34] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2. Grade-School Math and the Hidden Reasoning Process. ArXiv e-prints, abs/2407.20311, July 2024. Full version available at hattp://arxis: httt:://arxdv.or//bbbs///// abs.//0/s22222/../ 20311
[35] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. ArXiv e-prints, abs/2309. 14402, September 2023. Full version available at
http://arxiv. org/ wwwpaaa../..// 14402
[36] Ouail Kitouni, Niklas Nolte, Diane Bouchacourt, Adina Williams, Mike Rabbat, and Mark Ibrahim. The factorization curse: Which tokens you predict underlie the reversal curse and more. arXiv preprint XXX.:24446005/88 2024.
[37] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
12

---

[38] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-22655. PMLR, 2015.
[39] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
[41] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 15:4328-4343 2022.
[42] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint 1rXiv:2210.08933, 2022.
[43] Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressivy simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17455 2022.
[44] Robin Strudel, Corentin Tallec. Florent Altché, Yilun Du Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:22/1.0e: 1.0.2226, 2022.
[45] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self self-conditioning. arXiv preprint arXiv:2208.042205 2022.
[46] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Amaud Doucet, Robin Strudel, Chris Dyer. Conor Durkan, et al. Contin- uous diffusion for categorical data. arXiv preprint arXiv:2211. 2022.
[47] Pierre H. Richemond, Sander Dieleman, and Arnaud Doucet. Categorical sdes with simplex diffusion, 2022.
[48] Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai- Tao Zheng, Juntao Li. Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. Ar-diffusion: Auto-regressive diffusion model for text generation, 2023.
[49] Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew E. Peters, and Arman Cohan. Tess: Text-to-text self-conditioned simplex diffu- sion, 2024.
[50] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302. 10025, 2023.
[51] Yizhe Zhang. Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly. Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 14:8018--01999 2023.
[52] Aaron Lou and Stefano Ermon. Reflected diffusion models, 2023.
[53] Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308. 07037, 2023.
[54] Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference on Machine Learning, pages 21051-21064. PMLR, 2023.
[55] Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, and Chongxuan Li. Unifying bayesian flow networks and diffusion models through stochastic differential equations. arXiv preprint arXiv:2404.15766 2024.
13

---

[56] Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornoon, Zijing Ou, Joshua Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. arXiv preprint arXiv:2504. 16431, 2025.
[57] Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
[58] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 00l:1244--2440 2021.
[59] Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXiv:2) 10.020377 2021.
[60] Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029 2022.
[61] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022.
[62] Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing 2022.
Systems,
[63] Machel Reid, Vincent Hellendoom, and Graham Neubig. Diffuser: Diserete diffusion via edit-based reconstruction, 2022.
[64] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous- time discrete diffusion models. arXiv preprint arXiv:2211.16750, 2022.
[65] Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. Disk: A diffusion model for structured knowledge. arXiv preprint ar drXiv:2312.05253, 2023.
[66] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.05737, 2023.
[67] Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint arX1v:2312.09193, 2023.
[68] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language models can perform many tasks with scaling and instruction-finetuning. arXiv preprins arXiv:2308. /22/9,222222 2023.
[69] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, 2024.
and Yaron Lipman. Discrete flow matching. arXiv preprint
[70] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling, 2024. URL httpp::/arxiv......:///////// wwtp:/o/////a/aeeeeeoe abs//499./2444... 00044990029088
[71] Shreyas Kapur, Erik Jenner, and Stuart Russell. Diffusion on syntax trees for program synthesis. arXiv preprint arXiv:2405.20519, 2024.
[72] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410. 17891, 2024.
[73] Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mercury: Ultra-fast. language models based on diffusion. arXiv preprint arXiv:2506.1720 2025.
14

---

[74] Huiwen Chang, Han Zhang. Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Asuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint M.Xiv:2331.007000 2023.
[75] Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, and Chongxuan Li. Effective and efficient masked image generation models. arXiv preprint arXiv:2503.07197, 2025.
[76] Xinyou Wang. Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners, arXiv preprint urXiv:2402.18867, 2024.
[77] Xinyou Wang. Zaixiang Zheng. Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410.13782, 2024.
[78] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Clims: Consistency large language models. arXiv preprint arXiv:2403.00835, 2024.
[79] Chenkai Xu, Xu Wang, Zhenyi Liao, Yishun Li, Tianqi Hou, and Zhijie Deng. Show-o turbo: Towards accelerated unified multimodal understanding and generation. arXiv preprint 4rXiv:2502.05415, 2025.
[80] Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael Gómez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising. arXiv preprint arXiv:2410.06200 2024.
[81] Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. Dimo: Distilling masked diffusion models into one-step generator. arXiv preprint 15X45 2025.
[82] Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.002.34, 2025.
[83] Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsu- fuji. Distillation of discrete diffusion through dimensional correlations. arXiv preprint arXiv:2410.08700 2024.
[84] Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffuusion models. arXiv preprint arXiv:2407.21223, 2024.
[85] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.0200 2024.
[86] Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. In The Thirteenth International Conference on Learning Representations, 2024.
[87] Justin Deschenaux and Caglar Gulcehre. Beyond autoregression: Fast ilms via self-distillation through time. arXiv preprint arXiv:2410.21035. 2024.
[88] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[89] Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla- torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint arXiv:2412.10113, 2024.
[90] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
15

---

[91] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[92] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint WrXiv:233308874, 2023.
[93] Google. Our next gentrraenootiin model: Gemini 1.5, 2024. URL httpp:://blo.gooooottto pachnology/aiggooeee
-ili--nee -e--eeett-eeeee- mewpraaiioonmmoee-eeeeee
[94] Anthropic. Claude 3.5 sonnet, 2024.
URL
-laude-3-5-sone
[95] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 11X1v:2412.19437, 2024.
[96] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[97] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposhum user interface software and technology. pages 1-22, 2023.
[98] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 18345, 2024.
[99] OpenAl. Learning to reason with 1lms, 2024. URL htttss://ppema..cmm/ndeee
learning- tor.niinggo-roooo-iiiiiia. wo-rresso-wwith-mmms
[100] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501. 12948, 2025.
[101] Benigno Uria, lain Murray, and Hugo Larochelle. deep and tractable density estimator. In Proceedings of the 31th International Conference on Machine Learning, 2014.
[102] Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregres- sive models the right way. In Proceedings of the 31th International Conference on Machine Learning, 2022.
[103] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Ilms with nothing, arXiv preprint arXiv:2406.0844 2024.
[104] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with oss-instruct. arXiv preprint arXiv:2312.02120, 2023.
[105] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[106] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202,, 2020.
[107] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[108] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 200.0000000..611,, 2020.
16

---

[109] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprini arXiv:2203. 15556, 2022.
[110] Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 2020.
arXiv:2009.0333
[111] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09266 2022.
[112] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1eXiv:/803.00 180..05457 2018.
[113] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprini 1905.07800 2019.
[114] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 2021.
[115] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
[116] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.
[117] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüchhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1exxiv2110014111 2021.
[118] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. 2021.
arXiv preprint
[119] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-Ievel google-proof q&a benchmark. arXiv preprint arXiv:2311 1..3111.2222.... /2022, 2023.
[120] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[121] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models fill in the middle. arXiv preprint arXiv:2207. 14255, 2022.
[122] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 arXiv:2108.0077 00732, 2021.
[123] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.092/2. 2023.
[124] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
17

---

[125]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. framework for fhw-shot language model evaluation, 07 2024. http:ttss://eeem org/ records/12608602
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of the paper are well summarized the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
18

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix A we provide an overview the theoretical foundations required for this work, along with precise citations. Readers may refer to our overview consult the original references for full theoretical details.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details on data collection, training setup, model architecture, and evaluation protocols in Sec. 2 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
19

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: As with most large language models, we do not provide open access to the training data at this stage due data ownership restrictions. We provide details data collection, training setup, model architecture, and evaluation protocols in Sec. and Appendix B In addition, will release the model weights and evaluation code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //ipps.ccppullle.. mublic/guidess www.lic/guudees/oddduubiiis.ss www.oeeuumiisiioooooolloooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details on data collection, training setup, model architecture, and evaluation protocols in Sec. and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
20

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high cost of model training, it is difficult for us to rum repeated experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the training computer resources in Sec. We also provide an inference efficiency analysis in Appendix B.7
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp/s//eerrpp..../u/ll//////
Answer: [Yes]
Justification:
The authors have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
21

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide discussion about societal impacts in the Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Appendix B. details the data collection process, including the filtering of risky content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
22

---

Justification:
We have cited the original owners of assets used in the paper properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the assct's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
This paper does not release new assets. While we provide a detailed account of our dataset creation process in Appendix B.1 the dataset is not publicly released at this stage due to data ownership restrictions. addition, the model weights will be made available after the double-blind review process.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
23

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We evaluate our proposed large language model standard benchmarks and present example dialogues with the model in Sec. and Appendix B.10
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: /ttpa:://///ippe/ cc. for what should or should not be described.
24

---

Algorithm Pre-training of LLaDA
Require: mask predictor pe, data distribution Pdata
I: repeat
2: 20 Pdata
# with a probability of 1%, the sequence length of To follows U[1, 4096]
3: U(0, 1]
4:
# q: is defined in Eq. 7
5: Calculate - Tol Σ [z ()logpo(z((zz)
# L, the sequence length of aro
6: Calculate VgL. and run optimizer.
7: until Converged
8: Return p8
Algorithm 2 Supervised Fine-Tuning of LLaDA
Require: mask predictor pe. pair data distribution Pdata
I: repeat
2: Po. ru Pdata
# please refer to Appendix B.1 for details about the SFT dat
3: U0,
4:
(ralrr)
# q:1o is defined in Eq. 7
5: Calculate L ---roh Fel. El 1[r = M] logpe(rilpo.
L' is the sequence length of ro
6: Calculate V&L and run optimizer.
7: until Converged
8: Return Po
Algorithm 3 Conditional Log- likelihod Evaluation of LLaDA
Require: mask predictor po- prompt Po. response To, the number of Monte Carlo estimations 1mm
1: log likelihood =
Tmme do 3: [~ (1, L}
2: for
L is the sequence length of ro 4: Obtain rı by uniformly sampling tokens from ro without replacement for masking
5: log likelihood log likelihood + 4 Σ 1[r] log po (r) Po. rt)
6: end for
7: log likelihood = log likelihood/nan Tmm
8: Return log likelihood
A Formulation of Masked Diffusion Models
A.1 Training
MDMs [06 define the model distribution pş(xo) in a manner distinct from autoregressive models.
These models introduce forward process {z,} indexed by a time t E [0. 1]. This process gradually and independently masks all tokens in the sequence Tp. At time = O, the data point xo is fully observed with no masks, while for & (0, 1), 24 represents latent variables with varying mask ratios in expectation.
Formally, the conditional distribution of xe given un is defined by fully factorized form:
L
(xx (r))
II
ueoo (#) (c)))
1=1
where the conditional distribution each token is given by:
-
t,
o 20
aeo (x/())))
(8)
t. z M.
Here, M denotes the mask token. Intuitively, each token either remains unchanged is masked, with the probability of being masked increasing linearly f progresses from 0 to 1. At 1, all
25

---

Algorithm 4 Random Remasking Strategy of LLaDA
Require: mask predictor pe. prompt Po. answer length L, sampling steps
I: Set r1 fully masked sequence of length L.
2: for t f- down to N step 1 do N
3:
s= t-- -
4:
To = arg m8xxg Po (rolpooo rz) we employ greedy sampling when predicting masked tokens
5:
for 10 do
6:
r H then
7:
ró r
8:
else
9:
with probability 2. is set to M
10: end if
11: end for
12: Ts ro
13:
end for
14:
Return ro
tokens guaranteed be masked, meaning that 21 follows Dirac distribution concentrated sequence of fully masked tokens. Notably, the linear masking probability is analogous to but distinct from, the noise schedule in continuous diffusion models [3--40] This linearity is motivated by the assumption that the information in the text is proportional to the number of tokens on average, making it reasonable lose information linearly during the forward process.
The forward process is not only reversible but also corresponds to a reverse process that is fully factorized across all tokens. The reverse process, from time / = to O, generates new data from sequences of fully masked tokens. The conditional distribution for the reverse process, for 8 < t s 1, is factorized as:
II Moje (o
(9)
where the conditional distribution for each token is:
x x M, x zt.
M, x M.
(10)
Qoje (x
(x),
x
M, x o M,
otherwise.
Thus, the key function to estimate is the conditional distribution 4o e ( (x)), which predicts the original token if it masked in the input It. This is analogous the data prediction form in continuous diffusion models.
As proven in 20 an equivalent yet time-free parameterization can be derived
(0)((="1<x)) = Panta (zg UM,
Vi such that z M,
(11)
where .rUM denotes the collection of unmasked tokens in æe, which is identical to the corresponding tokens the original data Ho since unmasked tokens are solely determined by zo and are independent of time Intuitively, this implies that estimating the data prediction function is equivalent to estimating the conditional distributions on clean data, which is time-invariant. Consequently, the time t need not be provided input to the parametric model.
Although the development of masked diffusion is nontrivial, the implementation is straightforward. We first introduce the mask predictor, a parametric model (=(+(2;) (e.g., Transformer without causal mask), which takes F, for any as input and predict masked tokens simultaneously. Then, we define the model distribution po(xo) as follows: starting with as sequence of fully masked tokens, we simulate an approximate reverse process parameterized by po(-(ry) (ax) from to The marginal distribution induced at t then represents the model distribution (x())..
26

---

Return ro Algorithm 5 Low-confidence Remasking Strategy of LLaDA
Require: mask predictor pe, prompt Po. answer length L, sampling steps N 1: Set r1 fully masked sequence of length L.
2: for + down to N step N do
3: 8= =t--
for f- 11
4:
do
5:
if r o M then
6:
ro - r/,c==1
7:
else
8:
r = arg max, pe(rolpo.m)
9:
c po (ri ro)
10:
end if
end for
12:
Bun LL1 - s)]
the number of unmasked tokens is Ran in timestep
13:
for f- to do
14:
cf Lowest Ran (1c
then
15:
rá M
# the Nann positions with the least confidence selected for remasking.
16:
end if
17:
end for
18:
end for
20:
Formally, the mask predictor is trained using cross-entropy loss with masking:
(())= EEEEEEE mo
| Σ
log log
(12)
where zn sampled from the training data, is sampled uniformly from [0, 1), and v, sampled from (2a The indicator function [.]] ensures that the cross-entropy loss is computed only for masked tokens. In Shi et al. [18), Sahoo al. 19). Ou al. (20), it has been proven that the loss function C(0) is an upper bound on the negative log-likelihood of the model distribution:
-EzgrPantao) [[oo) s L(0).
(13)
In summary, this principled approach trains generative model by progressively masking tokens during forward process and learning to recover the data distribution during a reverse process, all under the (approximate) maximum likelihood estimation framework.
A.2 Inference
The cross-entropy loss in Eq. (2) has several equivalent forms
20
The first one given by
-E 2...1................... so,am
i = M] Mo po po Σ 1=1
(14)
where is uniformly sampled from 21,2..... L}. and xj obtained by uniformly sampling tokens from an without replacement for masking. Despite masking exactly tokens is different from masking each token independently with probability i, these two masking methods lead to equivalent results in expectation 20
While Eq. 12) and Eq. share the same expectation, their variances differ. Intuitively, Eq.
we expect 2, to have fraction of tokens masked. However, the randomness of the forward process (i.e., Eq. (7) often causes deviations, especially when x, contains few tokens. In contrast, in Eq. [14.. fraction of masked tokens in æ is deterministically 4. While theoretical analysis depends on the data distribution, empirical results show that Eq. (12) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. (14) achieves stability with only 128 estimates. addition, we can
simply modify Eq. (14) to its conditional version (i.c., Eq. (6) based on Eq. 5
27

---

Any-order autoregressive models (AO-ARM) [59] 1oI 102 characterize the joint distribution au- toregressively for all possible orders x of the L variables. To learn such a distribution, an AO-ARM utilizes a weight-sharing neural network to model all univariate conditionals and employs mask tokens to represent absent variables. During training, the expected negative log-likelihood over the uniform distribution of all orders U is minimized:
Σ log (2 (0) 120 (<00 ;
(15)
can be understood as a masked token with index in (2 i) being masked. It can be further proved that Eq. (15) is equivalent to Eq. (1). This connection explains the bidirectional reasoning capabilities of LLaDA, even though it was never used explicitly in the inference procedure.
(<<() Intuitively, 20
In addition, Nie et al. (7 introduces unsupervised classifier-free guidance (CFG), plug-and-pla play technique that balances alignment with prompts and text diversity. Specifically, unsupervised CFG employs the following modified mask predictor for inference:
Po(rolpo, rz) x pa(rolpo, re)
(16)
Po (m, rz)
where m is a mask sequence of the same length as po and w is tunable hyperparameter that controls the strength of po- To ensure a fair comparison with ARMs, we do not apply CFG LLaDA in the main text. However, as demonstrated in Appendix B.3 LLaDA is fully compatible with CFG and consistently exhibits improved performance when is applied.
A.3 Algorithms
In this section, we present the training and inference algorithms. Specifically, we introduce the pre-training and supervised fine-tuning algorithms in Algorithm and Algorithm [2] respectively. In addition, the likelihood evaluation algorithm is provided in Algorithm [ Finally, we present the reverse generation process in Algorithm 4 and Algorithm 5 which correspond to the random remasking and the low-confidence 223 remasking strategy, respectively.
B Experiments
B.1 Data Collection and Preprocessing
In this section, we first introduce the data collection and filtering processes for both pre-training and SFT. We then describe how LLaDA leverages these datasets during training.
Our pre-training corpus is constructed from diverse publicly available sources, including web data, books, academic papers, social media, encyclopedias, mathematics, and code, with approximately 1 11% Chinese, English, and 28% code. The data cleaning process involves PDF text extraction, deduplication, and harmful content filtering. To further ensure quality, we fine-tune BERT [22] model for automated data quality annotation, enabling the selection of higher-quality samples. Our SFT dataset consists of I million human-annotated samples and 3.5 million synthetic samples, generated using methods similar to those proposed in Xu et al. 103], Wei et al. [044.
We concatenate the collected documents in the pre-training corpus and segment the text into fixed- length sequences according to the predefined sequence length.
For SFT, a dynamic sequence length strategy is employed, where EOOS tokens are appended to the end of shorter pairs to ensure uniform sequence lengths across all samples within each mini-batch. Notably, the padding [EOS) tokens are treated as part of the response, i.e., masked and included in the training objective. The JEOS| tokens are removed from the generated outputs during sampling. This strategy ensures that the model learns to control the length of its responses by generating JEOSS. enabling the response length to align effectively with the given prompt.
In addition, for n-turn dialogues (pg. rā Po: r ......... pairs, i.e., (pg, r.), (p8r0rg. 02:: rd).... (pgr ... Pb partitioning strategy not only equips L.LaDA with multi-ture dialogue capabilities but also aligns with the above EOS padding strategy.
n 11,,,, we treat it as single-turn dialogue 16-1,1 r 1-1 and randomly sample one. This data
28

---

Table Model Architecture. We report the architectural configurations for our 1B and 7B ARM baselines, the IB and 8B LLaDA models, and the 8B LLaMA3 model.
Our ARM Baseline 1B
LLaDA IB
Our ARM Baseline 7B
LLLDD 8B
LLaMA3 8B
Layers
22
22
28
32
32
Model dimension
2048
2048
4096
4096
4096
Attention heads
32
32
32
32
32
Vocahulary size
126,464
126,464
126,464
126,464
128,000
FFN dimension
5634
5634
13,440
12,288
14,336
Key/Value heads
8
32
Total parameters
B
1.49
6.83
8.02
8.03 B
Non-embedding parameters
6.98
6.98
Table 6: Ablation on CFG. CFG consistently improves the performance of LLaDA.
ARC-C
Hellaswag
TruthfulQA
WinoGrande
GPQA
PIQA
w/o CFG
45.9
70.5
46.1 46.4
74.8
25.2
73.6
w/ CFG
47.9
72.5
74.8
26.1
74.4
B.2 Details about Model Training
This section provides the training details of LLaDA and the corresponding ARM baselines.
Firstly, for efficiency, trained ARM and MDM, both with 1.5B parameters and identical architectures. Additionally, we scaled the MDM to 8B parameters. Due to computational resource constraints, did train an 8B autoregressive model with the same architecture. Instead, utilized our previously trained 7B autoregressive model for comparison. These four models are utilized in the scalability analysis in Sec.
We adopted a Transformer architecture similar to LLaMA (6 21] for the ARMs and MDMs we trained. Specifically, employ RMSNorm [155] to stabilize training, use SwiGLU [106] as the activation function to enhance non-linearity, and integrate RoPE [107] for more expressive positional encoding. Tab. provides overview of the model architectures.
For IB and 7B ARM baselines, as well as IB and 8B LLaDA models, we utilized the Adam optimizer [29 with a weight decay of 0.1 and adopted the Warmup-Stable-Des
(28 learning rate scheduler. The learning rate was linearly increased from 0 to the maximum value over first 2000 iterations and then held constant. For LLaDA 8B, to ensure stable training, the learning rate was reduced once during pre-training, as detailed in Sec. 2.2 For the IB ARM baseline and both the IB and 8B LLaDA models, the maximum learning rate is set to x 10-4 with a batch size of 1280, without any hyperparameter tuning. For the 7B ARM baseline, the maximum learning rate set to 4.2 10- with batch size of 4224, both selected via grid search.
Additionally, we employ the widely used 6ND D formulation [108] 109 to calculate the training
FLOPs in Fig. where N represents the number of non-embenn non-ednedding parameters, and D denotes the total number of training tokens. The detailed results corresponding to Fig. are provided in Tab. [8] and Tab. 19
B.3 Ablation Classifier-free Guidance
This section presents ablation study classifier-free guidance (CFG). Theoretical details about CFG can be found in the Appendix A.2
For simplicity, we select six representative benchmarks, including ARC-C, HellaSwag, TruthfulQA, WinoGrande, PIQA, and GPQA, and conduct experiments using LLaDA 8B Base. We search the CFG scale in (0.5, 1, 1.5, 2} for each task and report the best result. As shown in Tab. 6 CFG consistently improves the performance of L.aDA. We emphasize that, to ensure fair comparison with ARMs, CFG is not used in the main results reported in the paper.
29

---

Prompt
Reeporiee
Prompl
Resporse
Prompl
Response
(a) Autoregressive.
(b) Block Diffusion.
Block Diffusion LLaDA.
Figure 4: Flexible Sampling Strategies Supported by LLaDA. Colored squares depict non- masked tokens, while squares marked with x denote masked tokens. In this illustration, the block length for both block diffusion and block diffusion LLaDA sampling is 4.
Table 7: Ablation on Sampling Strategies for LLaDA 8B Base. L' is the block length. Pure diffusion sampling achieves the best overall performance.
BBH
GSM8K
Math
HumanEval
MBPP
Autoregressive
38.1
63.1
23.6
18.3
33.4
L' = 2
37.3
62.6
25.2
14.6
33.6
L' 4
40.0
65.7
26.6
15.9
36.0
Block Difusion
I' 8
42.0
68.2
27.7
19.5
39.2
32
45.7
68.6
29.7
29.9
37.4
L' = 2
48.0
70.0
30.8
26.2
40.0
L' 4
48.5
70.3
31.3
27.4
38.8
Block Diffusion LLaDA
L' = 8
48.6
70.2
30.9
31.1
39.0
L' 32
48.3
70.3
31.2
32.3
40.0
Pure Diffusion
49.7
70.3
31.4
35.4
40.0
Table Ablation on Sampling Strategies for LLaDA 8B Instruct. The block length set to 32
for efficiency. Pure diffusion sampling achieves the best overall performance.
GSM8K
Math
HumanEval
MBPP
GPQA
MMLU-Pro
ARC-C
Autoregressive
0
9.5
0
0
84.4
Block Diffusion
24.6
23.5
17.1
21.2
29.3
32.5
88.1
Block Difusion LLaDA
77.5
42.2
46.3
34.2
31.3
34.8
85.4
Pure Diffusion
69.4
31.9
49.4
41.0
33.3
37.0
88.5
B.4 Details and Ablation on Sampling Strategies
In this section, first introduce the different sampling strategies supported by LLaDA. We then present ablation studies to evaluate the performance of these sampling strategies.
Flexible Sampling Strategies. In Sec. 2.4 Fig. (c) illustrates the reverse generation process of LLaDA. As shown Fig. 4 in addition to the reverse generation process, LLaDA also supports autoregressive and block diffusion [3I] sampling directly after the pre-training or SFT stages, without requiring any further modifications or retraining. Block diffusion sampling applics the origin reverse process within each block and the autoregressive sampling across blocks. In the original block diffusion process, the sequence length varies dynamically. As shown Fig. 4 (c), LLaDA can also adopt fixed-length block diffusion strategy, which refer to as block diffusion LLaDA, also known as cemi-autoregressivv remasking.
Experimental Setup. We evaluate different sampling strategies using both LLaDA 8B Base and LLaDA 8B Instruct for comprehensive analysis. LLaDA 8B Base, we use the five benchmarks in Tab. that are evaluated based on sampling rather than likelihood estimation. For LLaDA 8B Instruct, we use seven metrics Tab. excluding MMLU and HellaSwag, since these two tasks only require the model to generate single token (i.c., A. B, C, or D). In all settings, token is generated
30

---

Table Analysis on Random and Low-confidence Remasking Strategies. The low-confidence remasking consistently outperforms the random remasking.
Length
BBH
GSM8K
Math
HumanBval
MBPP
Random Remasking
32.1
21.3
9.2
11.6
21.0
Low-confidence Remasking
45.0
70.0
30.3
32.9
40.2
Table 10: Ablation on Generation Length. The results are not sensitive to the length hyperparameter.
Length
BBH
GSM8K
Math
HumanEval
MBPP
256
45.0
70.0
30.3
32.9
40.2
512
50.4
70.8
30.9
32.9
39.2
1024
49.7
70.3
31.4
35.4
40.0
per sampling step. For autoregressive and block diffusion sampling, generation terminates when [EOS| token is produced. For block diffusion LLaDA (i.e., semi-autoreerressive remasking) and pure diffusion sampling, the generation length is fixed 1024 for LLaDA Base, while for LLaDA 8B Instruct, it is tuned from [64, 256, 512] to balance efficiency and performance. Low-confidenee remasking is applied to intra-block diffusion sampling in both block diffusion and block diffusion LLaDA, well to pure diffusion sampling. We also test different block lengths for LLaDA 8B Base. For LLaDA 8B Instruct, we only evaluate block length 32 for efficiency, as it yields the best results on LLaDA 8B Base.
Additionally, for LLaDA 8B Instruct, due to heavy padding of [EOS] tokens in the SFT data (as detailed in Sec. B.1 we observe that under pure diffusion sampling, the proportion of [EOS] tokens model outputs becomes very high. This leads to extremely short generations and degrades model performance. To mitigate this issue, for HvmanEval, MBPP, GSM8K, Math, and GPQA, we set the confidence score of the [EOS| token to zero during pure diffusion sampling. This adjustment helps maintain an appropriate ratio of [OS| tokens during generation.
Finally, we conduct ablation studies analyze the effects of random and low-confidence remasking strategies using the pure diffusion sampling. For efficiency, we use LLaDA 8B Base with generation length and sampling steps set to 256 in this analysis.
Results. As shown in Tab. 7 for block diffusion sampling, overall performance improves as the block length increases. Moreover, both Tab. 7 and Tab. show that block diffusion sampling consistently outperforms autoregressive sampling. and block diffusion LLaDA sampling further improves upon standard block diffusion sampling. Finally, pure diffusion sampling achieves the best overall performance.
In addition, Tab. shows that the low-confidence remasking strategy consistently outperforms the random remasking strategy. We hypothesize that low-confidence remasking functions similarly to the annealed sampling method used by default in ARMs, improving accuracy by reducing the diversity of generated sentences.
We discover that autoregressive sampling leads very poor performance for LLaDA 8B Instruct. This is because each SFT data is complete sentence, so given a sequence length, LLaDA 8B Instruct tends to generate full sentence within that length. In contrast, LLaDA 8B Base does not suffer from this issue, as the pre-training data consists of truncated documents (as detailed in Appendix [B.I] and the model is trained with random sequence lengths (as detailed in Sec. 2.2). As result, when given short sequence length, LLaDA 8B Base tends to generate only part of a sentence, which can then be used prefix continue generation.
Setting the block length to 8 in Tab. further improves the GSM8K score from 77.5 78.6.
B.5 Ablation on Generated Length
In this section, we conduct ablation studies on the generated length.
31

---

70-
LLLDA RB
LLADA BB
UbMMA' HiR Wo Cacho
LIMMAT 4Bs wib KV Cachs
GSM8K
Math
20
x
5-
10
20
30
40
50
60
70
10
20
30
40
50
60
70
80
Throughput (tokens/s)
Throughput (tokens/s)
35
XA
LLaDA
X
LLaDA SB
LsMAS 6B KV Cadie
LisMA3 Cacle
20-
8B WV Cache
LIMMAT wis K.V. Cacho
HunanEval
25
MBPP
30-
20
15-
10-
20
40
40
100
120
140
150
10
20
30
40
50
10
Throughput (tokens/s)
Throughput (tokens//s
Figure 5: Analysis of Sampling Efficiency. The generation length for LLaDA is set to 256, with sampling steps set to 32, 64, 128, and 256 across the figures. This corresponds to decoding 8, 4, 2, and token(s) per forward pass, respectively. LLaDA enables a flexible trade-off between generation quality and sampling speed.
To ensure fairness, for each setting, we set the number of sampling steps equal to the generated length, ensuring that in each sampling step, just one tokens are transferred from the mask to the text. We conduct experiments using LLaDA 8B Base.
As reported in Tab. 10 the results are not sensitive to the length hyperparameter.
B.6 Standard Benchmarks and Evaluation Details
In this section, we introduce the benchmarks we used and present the details of our evaluation process.
Following standard LLM
25
26
evaluation practices, we assess L.LDD across four dimensions:
General ability: MMLU 110
BBH ARC-C 112
Hellaswag
113
TruthfulQA [14].
WinoGrande 115 and PIQA
116.
Math and science ability: GSM8K 117. Math 118 and GPQA 119
Code generation: HumanEval 120 HumanEval-FIN, 121 and MBPP
122
Chinese understanding: CMMLU 123 and C-Eval 214
For the aforementioned benchmarks, we follow the widely adopted evaluation process 125 used in LLM assessments, primarily employing conditional likelihood estimation and conditional generation. Specifically, for certain benchmarks, prompt and multiple candidate answers provided, and the model is required to compute each candidate's conditional likelihood. The candidate with the highest likelihood is then selected the model's 's final answer, and accuracy is used the evaluation metric. For the remaining benchmarks, the model generates responses based on the given prompt, and performance is evaluated using metrics such as exact match and other relevant criteria.
For the base model, we use conditional likelihood estimation for MMLU, CMMLU, C-Eval, ARC-C, Hellaswag, TruthfulQA, WinoGrande, PIQA, and GPQA, while the remaining benchmarks are evaluated using conditional generation. For the instruct model, we evaluate all benchmarks using conditional generation.
32

---

Table 11: Analysis of Memory Consumption. Memory is measured GB. Without any inference optimization techniques (e.g., KV Cache), LLaDA has memory usage comparable to LLaMA3, and slightly higher than LLaMA3 when the latter uses KV Cache.
Input Length
Output Length
LLaDA 8B
LLaMA3 8B w/o KV-Cache
LLaMA3 8B w/ KV-Cache
512
17.03
16.70
16.32
512
1024
17.53
17.49
16.43
2048
18,52
20.00
16.73
17.53
17.16
16.36
1024
1024
18.01
18.26
16.41
2048
19.02
21.39
16.74
For the base model, we use the widely adopted open-source evaluation framework Im-evaluation- harness [[25], except for the HumanEval-FiNh metric, which is not supported by the framework. For HumanEval-FIN on the base model and all evaluation metries on the instruct model, we use an internal evaluation library. We choose the internal library as Im-evaluation harness shows greater deviation from the LLaMA3 results reported by Yang al. 25 relative to our internal evaluation.
For benchmarks evaluated via conditional likelihood estimation, we use Monte Carlo estimation to approximate Eq. G for LLaDA. Since MMLU, CMMLU, and C-EVAL only require the likelihood of a single token, a single Monte Carlo estimate is sufficient for these benchmarks. For all other benchmarks, we find that 128 Monte Carlo samples are adequate to produce stable results.
For benchmarks evaluated using conditional generation, we apply pure diffusion sampling with a low-confidence remasking strategy to both LLaDA Base and LLaDA Instruct. For LLaDA Base, set both the generation length and the number of sampling steps to 1024. For LLaDA Instruct, the number of sampling steps is set equal to the answer length, which is configured as follows: 3 for MMLU and HellaSwag, 64 for GPQA, 256 for MBPP and MMLU-Pro, and 512 for HumanEval, GSM8K, Math, and ARC-C. As detailed in Appendix B.4 for HumanEval, MBPP, GSM8K, Math, and GPQA, we set the confidence of the |EOS| token to zero during sampling for LLaDA Instruct.
B.7 Analysis of Sampling Efficiency
In this section, we first analyze the sampling efficiency of LLaDA, including both sampling speed and memory consumption. We then discuss potential optimizations to further improve its efficiency.
We four representative open-ended generation benchmarks for sampling speed analysis: GSM8K, Math, HvaanEval, and MBPP. We use the widely adopted throughput metric to measure generation speed, defined the number of tokens generated per second. We compare LLaDA 8B Base and LLaMA3 8B Base, both using 6 precision. All experiments in this section were conducted on single A100- 80GB GPU with batch size of 1. For LLaDA, the output length fixed 256 tokens across all four benchmarks.
Fig. shows that LLaDA enables a flexible trade-off between generation quality and speed by adjusting the number of sampling steps. Specifically, on the GSM8K and Math datasets, LLaDA 8B Base achieves comparable performance to LLaMA3 8B Base while delivering 1.5 and 1.8 times higher throughput, even though LLaMA3 uses KV Cache and LLaDA operates without any inference optimization techniques. For the HumanEval benchmark, LLaDA 8B Base performs comparably to LLaMA3 8B Base when the throughput is matched. On the MBPP benchmark, LLaDA 8B Base lags behind LLaMA3 8B Base.
For LLaMA3, the acceleration benefit provided by KV caching is notably weaker on the HumanEval dataset, which can be attributed to its relatively short prompt lengths. Specifically, the average prompt lengths for GSM8K, Math, MBPP, and HumanEval are 894, 680, 628, and 132 tokens, respectively.
Tab. compares of memory consumption between LLaDA 8B Base and LLaMA3 8B Base. To avoid variations generation length caused by differences in training data, we fix both the input and output token lengths during the memory analysis. For LLaDA, memory usage remains constant regardless of the number of sampling steps. Its memory consumption is comparable to LLaMA3 8B Base without KV cache, but slightly higher than with KV cache.
33

---

Table 12: Comparison on iGSM Dataset.
4 steps
5 steps
6 steps
LLaMA3 8B Base
38.0
35.0
34.0
LLaDA 8B Base
64.0
41.0
44.0
We emphasize that the goal of this study is not to propose model that faster than ARMs. Instead. we aim to show the promise of diffusion models for language modeling at scale and challenge the common assumption that core I.LM capabilities such as scalability, in-context learning, and instruction-folle follloiigg are inherently depend on ARMs. A substantial body of research [31 (87 has focused on improving the generation efficiency of MDMs through algorithmic or architectural innovations. We leave similar efficieency-oriented exploration for LLaDA to future work.
B.8 Evaluation iGSM Dataset
To further assess the mathematical capabilities of LLaDA, we test its performance on iGSM [34]. an infinite, synthetic GSM8K-like dataset. iGSM is generated via specific rules, with parameters that control the difficulty of problems (i.c., the number of solution steps). For evaluation consistency, we append "##### Sanswer" the final solution, adhering the GSM8K format. Below is an example with solution steps set to 4:
(Question) The number of each North Star Elementary's ' Cultural Studies Classroom equals 1. The number of each Westridge Elementary's Dance Studio equals times as much as the sum of each North Star Elementary's Classroom and each North Star Elementary's 's Cultural Studies Classroom. How many Dance Studio does Westridge Elementary have?
(Solution) Define North Star Elementary's 's Cultural Studies Classroom as x; X = 1. Define North Star Elementary's 's Classroom as m; so m x I. Define Westridge Elementary's 's Dance Studio as so n == * w & = 1 #####
+ 1
Since there are slight differences between GSM8K and iGSM (e.g., the use of mod 5 algorithmic system), we follow 34 and provide a system prompt along with four-shot question-answer pairs for each problem.
(Prompt) You're an expert solving elementary math problems involving addition, subtraction, and multiplication. You solve all the problems in a uniform format. All calculations are done modulo For example, + equals + equals 1, 3 equals * equals and 2 2 equals 4. When providing your solution, please end with "#H## x.' where x is your final answer, an integer between and 4. You must solve all the problems using the same solution format. Our scenarios involve up to four categories of objects: schools, classrooms, backpacks and stationeries. Each school may contain classrooms, each classroom may contain backpacks, and each backpack may contain stationeries. We can specify quantities, such as the number of dance studios at each Lakeshore nigh.Assume that every entity with the same name has an identical configuration; for example, each Lakeshore High contains the same number of dance studios. Another guiding principle is that what is not mentioned does not exist: when we refer to classrooms at Lakeshore High, are only discussing the classrooms explicitly mentioned in our scenario. Furthermore, Lakeshore High is not even mentioned, any classroom within is automatically considered to be non mon-existent (i.c. 0).
For solution steps ranging from to 6, we generate 100 questions for each case and report the corresponding accuracy in Table [2] As shown in the table, LLaDA 8B Base demonstrates significant and consistent advantages over I.L.MA3 8B Base on unseen mathematical problems, aligning with the results in Table
34

---

Table 13: Visualization of the Block Diffusion LLaDA Sampling Process. In the response of LLaDA, darker colors indicate tokens predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.
User LLaDA
What are the benefits of regular exercise for physical and mental health?
Regular
exercise
improve
physical
health
by
strengtheniig
muscles
and
bones
Improving
reducing
diabetes
and
heart
disesse
can
also
improve
mental
health
reducing
strees
and
anxiety
Improving mood
and
enhanding
cognitive
function
Additionally
exercise
quality
levels
B.9
Poem Completion Tasks
In this section, we present examples from our poem completion dataset as follows.
Example 1: Prompt: x
Answer: #T
Example 2:
Prompt: -
\
Answer: 2
B.10
More Case Studies
In this section, we present additional case studies LLaDA 8B Instruct. First, Tab. 13 shows the sampling process of the block diffusion LLaDA sampling, while Tab. 14 depicts the sampling process for multi-turn dialogues with random remasking. Additionally, Tab. 15 and Tab. [6 provide further examples of single-tumn and multi-turn dialogues. Finally, Tab. [ab.[1]|prer pressettss examples of poem reversal completions where the LLaDA 8B Instruct model succeeds, in contrast to the failure of GPT-4o.
C Impact Statement
Our work shows the promise of diffusion models for language modeling at scale and challenges the common assumption that core LLM capabilities such as scalability, in-context learning, and mnstruction-folloviin are inherently dependent on ARMs. Our findings open new avenues for explor- ing alternative probabilistic paradigms in natural language processing, with potential applications in conversational AI, code generation, and complex reasoning tasks.
However, diffusion models, like traditional LLMs, raise similar societal concerns. These include the environmental impact of large-scale training, the potential misuse for generating harmful content, and the amplification of biases present in training data. Addressing these challenges is critical to ensuring the responsible development and deployment of diffusion language models.
35

---

Table 14: Visualization of the Multi-turn Dialogue. We employ random remasking strategy. In the response of LLaDA, darker colors indicate tokens predicted in the later stages of sampling, while lighter colors correspond to earlier predictions.
User
I want to have a pet, but don't know whether to get cat or a dog. Do have any suggestions?
LLaDA
really
depends
on
your
preferences
lifestyle
and
the
type
pet
you
want
LE
you
want an independent: pet
and
have
busy
schedule
cat
good
choico
you
prefer
companion
and
have
more
active
lifestyle
dog
might
ba
better
Hava
you
considerad
your
options
User
I'm very busy at work, so it seems like would be more suitable option. Do you have
any suggestions getting a cat?
LLaDA
Surn
First
make
sure
you
have
enough
apace
your
home
and
provicle
food
water
and
tays
your
Also
research
different
breeds
and
personalities
and
choese
COTS that
cuitable for
your
lifestyle
Lastly
make
sure
You
have
the vacoinations and insurance place
before
36

---

Table 15: Single-turn Dialogue Cases of LLaDA 8B Instruct.
Daily life User LLaDA
What's piece of advice you't ve received that you ve found helpful?
One helpful piece of advice I've received is to always be open to learning and growth. I've found that being willing to explore new ideas and experiences has helped me become more resilient and adaptablc. By embracing growth mindsct, I've been able to identify opportunitics for improvement and development, both personally and professionally.
Daily life User LLaDA
Please recommend three famous movies Certainly! Here are three famous movies that have achieved widespread popularity and critical acclaim:
1. **The Shawshank Redemption*** (1994) Inspired by novella and directed by Frank Darabont, this film tells the story of Andy Dufresne, wife, dealing with his harsh life in Shawshank Prison and forming an unlikely friendship with a fellow inmate.
convicted of murdering his
2. T* Godfather** (1972) Directed by Francis Ford Coppola, this film is based Mario Puzo's novel of the name. It follows the story of the Corleone family, particularly Michael Corleone, who rises to power and takes over the family's criminal empire.
3. ** Empire Strikes Back** (1980)- Directed by George Lucas, this film continues the story of young Skywalker in the Star Wars saga. follows Luke Skywalker as he helps the Rebel Alliance fight against the Galactic Empire and destroy the powerful weapon, the Death Star.
All of these films celebrated for their einematic achievements, powerful storytelling, and memorable characters.
Translation User
Please help me translate into Chinese: 'What now proved was once only imagined' 1
LLaDA
Translation
User
Please help me translate into German: 'What is now proved was once only imagined' Wer heute bestätigt ist, einst immaginiert.
LLaDA
Math
User
Emma decides run sprints times week. She runs 75 meters each sprint. How many total meters does she run a week?
LLaDA
Emma runs 4 sprints each time, so she runs total of 4 x 5 20 sprints week. Each sprint is 75 meters long, Emma runs total of 20 75 1500 meters a week. The final result 1500
Code
User LLaDA
am writing python code, have an array, please help me sort from largest to smallest Certainly! You can sort an array from the largest to smallest in Python using the "sort()" method or the 'sorted()' function. Here's 's an example: ### Using 'sort(" method:
my my_array = [3, 1, 4, 5.
my aaray..ort(eevere=TTnu faintmy__array
37

---

Table 16: Multi-turn Dialogue Cases of LLaDA 8B Instruct.
Daily life
User LLaDA
How do I prepare for job interview?
To prepare for job interview, you should research the company and the job, practice common interview questions, and dress appropriately.
User LLaDA
What are some common interview questions? Some common interview questions include "Tell me about yourself" and "Why do you want work for us"?
Daily life
User
How do start learning programming?
LLaDA
To learn programming, you can start by choosing programming language that you are interested in. Some popular languages include Python, Java, and C++.
User
Which programming language should Iearn first?
LLaDA
It depends on your interests and goals. Python is a great language for beginners for its simplicity and versatility.
User
How long does take to learn Python?
LLaDA
It can take anywhere from a few weeks to several months to learn Python.
User
What are some good Python projects for beginners?
LLaDA
Some good Python projects for beginners include building a simple calculator, game, and web application.
Math
User
Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?
LLaDA
Natalia sold clips to friends April. She sold half many clips May, so she sold 48/2 = 24 clips in May. Altogether, Natalia sold 48 + 72 clips in April and May. The final result is 72.
User
Natalia sold three times many items in June she did in May. How many clips did Natalia sell altogether in these three months?
LLaDA
Natalia sold clips to 48 friends in April. She sold half as many clips in May, so she sold 48/2 24 clips in May. Natalia sold three times many items in June she did in May, so she sold these three months. The final result is 144.
clips in = 72 clips in June. Allogether, Natalia sold 48 + 24
Table 17: Poem Reversal Completion Cases where LLaDA 8B Instruct Succeeds but GPT-4o Fails.
Question! User
1
GPT-4o
# T MB BAN "(
LLaDA
A
Question2
User
GPT-4o
" N4 i11
5 T (WE
LLaDA
"7
38

---

Table 18: Detailed results of LLaDA in Fig. observations regarding the scalability of LLaDA These missing values are due to hardware failures.
indicates missing values, which do not affect the
Model
Training Tokens
FL.OPs
MMLU
CMMLU
ARC-C
PIQA
GSM8K
HumanFival
LLaDA IB
37.75B
2.20e20
25.52
25.95
25.17
59.41
1.82
0.00
LLaDA IB
88.08B
5.13e20
27.11
26.52
26.96
61.86
3.03
1.83
IB
138.41B
8.06e20
29.32
27.13
30.20
63.38
2.35
0.00
LLaDA 1B
239.08B
1.39e21
31.48
30.77
27.99
63.11
3.26
1.22
LLaDA IB
352.32B
2.05e21
35.86
34.35
31.31
65.34
3.64
3.05
LLaDA IB
461.37B
2.69e21
31.86
30.98
30.12
65.51
2.35
0.61
LLaDA 8B
62,91B
2.63e21
32.22
28.5
30.20
63.82
3.87
2.44
LLaDA 8B
125,83B
5.27e21
33.39
33.9
34.64
66.54
8.72
3.66
LLaDA 8B
251.66B
1.05e22
42.84
40.59
40.10
69.04
15.31
3.66
LLaDA SB
377.49B
1.58e22
45.11
43.99
39.25
68.61
25.40
9.76
LLaDA SB
503.32B
2.11e22
43.57
41.38
42.06
70.24
27.52
9.76
LLaDA 8H
629.14B
2.63e22
48.80
47.13
42.24
72.09
30.10
12.80
LLaDA 8B
679.48B
2.85e22
49.61
48.19
41.30
70.84
26.31
8.54
LLaDA
792.72B
3.31e22
42.58
70.51
31,99
6.10
981.47B
4.11e22
49.47
48.10
40.27
71.38
6.10
LLaDA 8B
1107,30B
4.64e22
51.13
47.57
41.13
69.26
36.69
10.37
LLaDA 8B
1233.13B
5.16e22
50.52
49.72
45.05
71.49
38.97
9.76
L.LDDA 8B
1358.95H
5.69e22
54.61
53.97
49.40
74.05
48.14
17.68
L.LDDA 8B
1547.708
6.48e22
57.38
56.04
49.49
74.59
53.30
20.73
LLaDA 8B
1975.52B
8.27e22
58.52
57.87
50.68
75.35
19.51
Table 19: Detailed results of the autoregressive baelines Fig.
Model
Training Takens
FL.PPs
MMLU
CMMLU
ARC-C
PIQA
GSM8K
HumanEyal
ARM IB
37.75B
2.20e20
25.47
25.38
30.20
67.36
2.20
4.88
ARM IB
88.08B
5.13e20
24.67
25.23
33.96
70.02
7.51
10.37
ARM IB
138.41B
8.06e20
29.25
27.48
33.45
70.29
8.34
9.76
ARM 7B
17.30B
6.02c20
26.92
25.18
21.02
57.18
1.29
1.22
ARM 7B
34.60B
1.20e21
25.83
25.38
24.07
62.84
1.59
2.44
ARM 7B
86.50B
3.01e21
24.41
24.90
25.42
71.11
2.88
7,93
ARM 7B
173.02B
6.02c21
26.20
24.78
26.10
74.27
6.67
9.15
7B
207.62B
7.23e21
28.86
31.86
74.48
8.57
12,80
ARM 7B
224.92B
7.83e21
29.49
32.26
31.19
74.37
8.95
8.54
ARM 7B
242.22B
8.43e21
33.62
31.38
34.92
75.41
10.84
9.15
ARM 7B
259,52B
9.03e21
34.11
34,20
32.88
75.19
9.33
10.98
ARM 7B
311.43B
1.08e22
35.66
35.49
36.61
75.14
11.30
10.37
ARM 7B
363.33B
1.26e22
34.54
37.67
34.58
76.55
12.28
14.02
ARM 7B
415.24B
1.45e22
35.37
38.37
35.25
76.39
14.40
12.80
ARM 7B
449.84B
1.57e22
39.51
39.24
34.92
76.82
14.94
14.63
ARM 7B
519.09B
1.81e22
40.30
40.69
37.29
77.15
14.03
14.63
ARM 7B
778.57B
2.71e22
43.33
43.50
38.31
77.53
17.59
14.63
ARM 7B
1038.09B
3.61e22
45.06
46.12
41.69
77.86
20.02
15.85
ARM 7B
1384.12B
4.82e22
47.63
48.18
47.80
76.93
22.82
15.24
ARM 7B
2076.18B
7.23e22
47.68
50.85
44.07
77.37
24.79
14.63
ARM 7B
2214.59B
7.71e22
49.26
$2.08
53.56
77.69
27.37
17.07
39


Reviews:
[{"review_id": "KbOEcQR5QU", "reviewer": "Reviewer_YafR", "summary": "This paper presents LLaDA, a diffusion large language model trained with a masked token prediction objective. The core idea resembles that of MaskGIT or BERT, employing a high and random masking ratio. During inference, the model iteratively predicts masked tokens, selectively retains certain predictions, and re-masks others. Experimental results demonstrate that LLaDA outperforms LLaMA2-7B and achieves performance comparable to LLaMA3-8B.", "questions": "1. I find the evaluation procedure of LLaDA somewhat unclear, particularly in how its performance is made comparable to that of autoregressive models. According to Algorithm 3, the evaluation involves Monte Carlo estimation with random masking of tokens. Could the authors clarify whether the target tokens (i.e., the ground-truth answers) are included in the pool of randomly masked tokens, or whether they are always masked? A more detailed explanation of how log-likelihood is computed is preferred.\n\n2. When employing the confidence-based decoding strategy, how much does the generation behavior diverge from the autoregressive paradigm? \n\n3. Given that some proprietary models achieve strong performance on code generation tasks. It would be interesting to know whether LLaDA offers any inference-time advantages over autoregressive models in these datasets, such as HumanEval and MBPP. Has the model been evaluated on such benchmarks, and if so, how does its speed and performance compare?\n\n4. Have you tried to scale a continuous diffusion model instead of the MDM, or say the discrete diffusion models?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. This paper successfully scales diffusion language models to the billion-parameter regime, which marks a significant contribution to the community. Demonstrating that an alternative paradigm can achieve performance on par with traditional autoregressive models highlights the potential of non-AR approaches and broadens the design space for future language model development.\n\n2. The performance of LLaDA is impressive. Notably, the model exhibits in-context learning capabilities, which challenge the common belief that such behavior is exclusive to AR models. This suggests that in-context learning may be an emergent property of language, rather than a consequence of autoregressive modeling alone.\n\n3. The authors further fine-tune the model with instruction-tuning and SFT, showing that the pretrained diffusion-based model can match the capabilities of AR models in instruction-following tasks. This underscores the viability of diffusion-based models for practical downstream applications.\n\n\nWeaknesses: \n1. I remain unconvinced that the proposed model truly qualifies as a diffusion model in the conventional sense. While the diffusion perspective offers an interesting theoretical lens for interpreting BERT/MaskGIT-style training, it departs from the standard formulation of diffusion models, particularly those involving continuous-time processes as used in image generation. \n\n2. The training efficiency claims in Figure 3 need further clarification. It is not immediately clear why LLaDA would achieve better training efficiency than autoregressive baselines, especially considering that only a subset of tokens is utilized per iteration. In contrast, autoregressive models typically leverage 100% of the tokens during training. The current results do not convincingly support the claim of higher efficiency.\n\n3. From the inference perspective, the proposed method may be significantly less efficient than autoregressive models, particularly in long-context scenarios. It would be helpful if the authors could provide a direct latency comparison between LLaDA and strong AR baselines such as LLaMA2 or LLaMA3, to better assess the practical deployment feasibility of their approach.", "was_revised": true, "final_justification": "The authors' rebuttal solves most of my concerns, except the one about the training efficiency problem compared to ARs. But I think it's not a big problem and I still vote the strong accept of this paper due to the contribution of this paper.", "rebuttal": "# Response to Reviewer YafR\nWe thank Reviewer YafR for the recognition of our contributions and the insightful comments. Below is our point-by-point response.\n\n\n## Q1: Diffusion Formulation\nThanks for your question. First, we would like to kindly clarify that LLaDA is a **continuous-time but discrete-state** diffusion model (see Eq. (3) in our submission, where the time variable $t$ is continuous). In the first diffusion model paper [1], both continuous-state and discrete-state transition kernels were proposed under the same diffusion formulation (see Table App.1 in Appendix B of [1]). The continuous-state variant is widely known due to its success in image generation, but it currently faces scalability challenges when applied to language tasks (see details in our response to Q7).\n\n\nThis has motivated exploration of discrete diffusion language models [2, 3, 4]. Following such a formulation, LLaDA adopts a randomly sampled masking ratio between 0 and 1 (see Lines 82–86), whereas BERT uses a fixed ratio. This fixed masking ratio prevents BERT from being used for sampling and limits its scalability. MaskGIT’s training objective is heuristic and lacks a theoretical link to maximum likelihood, whereas LLaDA’s objective is explicitly derived from maximum likelihood estimation. In particular, MaskGIT's objective misses the $\\frac{1}{t}$ term compared to our Eq. (3), further indicating that it is a heuristic formulation.\n\n\nMore importantly, **the theoretical insights from diffusion motivated our decision to scale discrete diffusion models for language as a counterpart to autoregressive approaches.** The theoretical analysis of discrete diffusion shows that its training process corresponds to maximum likelihood estimation, which gives us confidence in scaling discrete diffusion to larger models and datasets (as detailed in Lines 17–37). This theoretical foundation also helps explain why LLaDA demonstrates scalability comparable to that of autoregressive models.\n\n\nGiven that **LLaDA shares the same theoretical foundation as continuous diffusion models and differs significantly from BERT and MaskGIT in both theoretical origin and practical implementation**, we have adopted the terminology used in the diffusion literature and refer to LLaDA as a diffusion model. We will revise the related work section to make this distinction clearer in the revised version.\n\n[1] Sohl-Dickstein et al. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015.\n\n[2] Austin et al. Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021.\n\n[3] Lou et al. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. ICML 2024.\n\n[4] Ou et al. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. ICLR 2025.\n\n\n## Q2: Insight on Training Efficiency\nThank you for your question. First, we would like to kindly clarify that Fig. 3 (Page 6) demonstrates that LLaDA exhibits strong scalability, achieving overall performance comparable to the autoregressive baseline across six tasks (as presented in the caption of Fig. 3).\n\nWe appreciate your concern that only a portion of the tokens (50% in expectation with a linear schedule) produce loss during training. However, we believe it may be premature to conclude that the data utilization rate of LLaDA is only 50%. Intuitively, although some tokens are not explicitly predicted, they are still processed by the neural network during training. \n\nIn practice, even when adjusting the noise schedule to increase or decrease the average masking ratio during training, the final performance remains unaffected. Theoretically, this is because the loss function can be reformulated—via a change of variables—into an equivalent form that is independent of the specific noise schedule (see Sec. 3.4 in [5] and Sec. 3 in [6]). Experimentally, [5] tested different noise schedules, including average masking ratios of 36%, 50%, and $1 - \\frac{1 - e^{-10^8}}{10^8} \\approx$100%, all of which yielded same loss value (i.e., likelihood) (as detailed in Appendix E.1 of [5]).\n\nMoreover, when modeling the joint distribution of data, autoregressive models optimize only a single factorization order, typically from left to right. In contrast, LLaDA is trained to consider multiple conditioning directions, which may provide greater flexibility and lead to better generalization. The advantage of LLaDA on reversal reasoning is a concrete example of this property.\n\nWe will include the above discussion in the experimental section of the revised version.\n\n[5] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS 2024.\n\n[6] Shi et al. Simplified and Generalized Masked Diffusion for Discrete Data. NeurIPS 2024.\n\n\n\n## Q3: Inference Efficiency\nThank you for your suggestion. We have analyzed the inference efficiency of LLaDA in comparison with LLaMA3 in Fig. 5 (Page 25, with detailed experimental settings described in Lines 863–889).\n\nWe employ the widely adopted throughput as the evaluation metric [7]. Compared with latency (i.e., the total time required to generate all tokens) as suggested, throughput measures the number of tokens generated per second and helps avoid discrepancies in output length caused by variations in training data, ensuring a fair comparison.\n\nAs shown in Fig. 5 (Page 25), for the GSM8K and Math benchmarks, where the average prompt lengths are 894 and 680 tokens respectively, LLaDA achieves slightly better accuracy than LLaMA3 with KV Cache while delivering 1.5× and 1.8× higher throughput respectively, when configured to sample 4 tokens per step (that is, the number of sampling steps is one-quarter of the generation length). For the HumanEval benchmark, which has a much shorter average prompt length of 132 tokens, LLaDA achieves comparable accuracy and throughput to LLaMA3 with KV Cache even when generating one token per step. However, for the MBPP benchmark, where the average prompt length is 628 tokens, LLaDA lags behind LLaMA3 in accuracy when the throughput is matched.\n\nThe above results are obtained without applying KV Cache or advanced parallel sampling techniques to LLaDA. However, recent methods such as these, which are specifically developed for diffusion language models (e.g., [8]), can be applied to LLaDA without retraining to further improve its efficiency.\n\n\n[7] Inception Labs. Mercury: Ultra-Fast Language Models Based on Diffusion. arXiv 2025.06.\n\n[8] Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. arXiv 2025.05.\n\n\n## Q4: Details on Likelihood Evaluation \nThank you for your question. As detailed in Lines 836–842, following standard practice for autoregressive models, likelihood evaluation is only conducted on multiple-choice tasks. The model computes the conditional likelihood for all candidate options, including the correct answer (without knowing which is correct), and selects the one with the highest likelihood. Accuracy is then used as the evaluation metric. This procedure is both standard and fair.\n\n\n## Q5: Generation Behavior of the Low-confidence Remasking\n\nThanks for your question. When using the Low-confidence Remasking strategy, the model tends to generate tokens on the left side of the sequence earlier in a coarse-grained manner, but the generation does not follow a strictly left-to-right autoregressive order.\n\nWe selected the first 100 samples from the GSM8K test set and analyzed the position distributions of the 10th, 50th, and 100th generated tokens. The frequency statistics are presented in the tables below. For clarity, we report only the top 10 most frequent positions for each case.\n\nthe 10th generated tokens:\n| Position | 12 | 10 | 11 | 13 | 14 | 17 | 16 | 9 | 8 | 6 |\n|----------|----|----|----|----|----|----|----|---|---|---|\n| Freq     |15% |12% |12% |11% | 9% | 7% | 6% |6% |4% |4% |\n\nthe 50th generated tokens:\n| Position | 50 | 51 | 52 | 49 | 54 | 48 | 53 | 59 | 56 | 43 |\n|----------|----|----|----|----|----|----|----|---|---|---|\n| Freq     |17% |10% |8% |7% | 6% | 6% | 4% |4% |3% |3% |\n\nthe 100th generated tokens:\n| Position | 101 | 100 | 103 | 99 | 112 | 98 | 102 | 105 | 104 | 107 |\n|----------|----|----|----|----|----|----|----|---|---|---|\n| Freq     |13% |11% |10% |9% | 7% | 6% | 6% |5% |5% |4% |\n\n\n## Q6: Performance and Efficiency on Code Generation\n\nThank you for your question. We have evaluated LLaDA on both the HumanEval and MBPP benchmarks; please refer to our response to Q3 for details. It is worth noting that our focus is on general performance, and we did not specifically design the training process for code generation. We may consider incorporating a higher proportion of code data or applying common techniques such as reinforcement learning to further improve LLaDA’s code generation capabilities.\n\n\n## Q7: Scaling Continous Diffusion\n\nThis is an excellent question. Our preliminary experiments suggest that, given the same model size and training data, continuous diffusion models fall behind discrete diffusion models. Some existing works [9, 10] in the field have also reported similar observations. We hypothesize that this may be due to a mismatch between discrete data and continuous representations. We believe this is a challenging yet highly impactful direction for future research.\n\n[9] Gulrajani et al. Likelihood-Based Diffusion Language Models. NeurIPS 2023.\n\n[10] Nie et al. Scaling up Masked Diffusion Models on Text. ICLR 2025.\n\n**If you have any more questions, we are happy to discuss them and will do our best to address them!**"}, {"review_id": "cyekx0tuT1", "reviewer": "Reviewer_vGqQ", "summary": "This paper scales up the training for diffusion language model to more than 2T pretraining tokens and 8B model size, achieving superior performance than the autoregressive counterpart across a broad range of downstream benchmarks. The artifact, LLaDA, shows strong capabilities in scalability, in-context learning, and instruction-following, overcoming the limitations of autoregressive models on certain tasks such as the reversal curse.", "questions": "1. Could the author explain how is the output length controlled during evaluation? The paper says that <EOS> tokens were injected during pretraining and removed during sampling. However, I did not understand how the length is predicted at inference time--do you set a fixed length and force the model to fill the length for different samples, or do you ask the model to generate a maximum length and cut after <EOS>? How is the output length predicted for different evaluation samples?\n2. Could you explain more about how you did conditional likelihood evaluation? What is a lower variance and upper bound in 2.4 and how they are used?\n3. Do you have any insights on why diffusion LM outperforms autoregressive LMs with the same training FLOPs? Do you think your pretraining data quality is higher than the baseline models? \n4. Do you plan to open-source your data and training code?", "limitations": "yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strenghts:\n1. The performance is impressive--it is the first work showing that the diffusion language model could performance comparable and even superior than well-established autoregressive models.\n2. Comprehensive evaluation is conducted to show the strong capabilities of the trained diffusion model on various tasks.\n\nWeaknesses:\n1. There is limited novelty on architecture design and training method. The main contribution is on scaling up the training set.\n2. The paper writing could be further improved to clarify some details (see questions).", "was_revised": true, "final_justification": "This paper present comprehensive experiments for scaling up the training for diffusion LMs, providing meaningful insights for practitioners. My questions have been addressed during rebuttal and I have raised my score accordingly.", "rebuttal": "# Response to Reviewer vGqQ\nWe thank Reviewer vGqQ for the recognition of our contributions and the thoughtful comments. Below is our point-by-point response.\n\n## Q1: Contribution\nThank you for your question. We kindly note that, in addition to the architectural design and training methods you mentioned, new paradigms, conceptual advances, or novel discoveries also reflect forms of scientific innovation. We illustrate this point using two highly influential examples: Vision Transformer (ViT) [1] and GPT-3 [2]. \n\nAs early as ICLR 2020, [1] proposed the idea of applying Transformers to the image recognition domain. ViT, published at ICLR 2021, further demonstrated that large-scale pre-training can make vanilla Transformers competitive with, or even superior to, state-of-the-art CNNs. Similarly, GPT-3 builds on the same architecture and generative pre-training approach introduced in GPT-1 and GPT-2, and shows that scaling up language models significantly improves task-agnostic, few-shot performance. **Neither ViT nor GPT-3 introduced fundamentally new architectures or training algorithms, yet both have profoundly shaped the current landscape of artificial intelligence.**\n\n\nOur findings first show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities including scalability, in-context learning, and instruction following are inherently depend on autoregressive models. We appreciate your comment, \"it is the first work showing that the diffusion language model could perform comparably to, and even better than, well-established autoregressive models\".\n\nWe will include the above discussion in the revised version.\n\n\n[1] Dosovitskiy et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021.\n\n[2] Brown et al. Language Models are Few-Shot Learners. NeurIPS 2020.\n\n[3] Cordonnier et al. On the Relationship between Self-Attention and Convolutional Layers. ICLR 2020.\n\n\n## Q2: Output Length\nThank you for your question. In your comment, you mentioned that \"The paper says that [object Object] tokens were injected during pretraining.\" We assume that the \"[object Object]\" text is a typographical error, and based on the context, it likely refers to the |EOS| token. If our interpretation is incorrect, please feel free to let us know.\n\nAs detailed in Lines 150–152, the generation length is treated as a hyperparameter that specifies the length of the fully masked sentence at the beginning of the sampling process. Consequently, the length of the sentence generated by LLaDA corresponds to this hyperparameter. To support variable-length generation, we truncate the output at the |EOS| token. \n\nWe analyze the impact of this hyperparameter, as shown in Tab. 10 (Page 24). For generation lengths of 256, 512, and 1024, the maximum performance differences observed on the BBH, GSM8K, Math, HumanEval, and MBPP benchmarks are 5.4, 0.8, 1.1, 2.5, and 1.0 points, respectively. Overall, the impact is limited and remains well-controlled.\n\n## Q3: Conditional Likelihood Evaluation\nThank you for your question. As detailed in Eq. (4) of our submission, the training loss of LLaDA has been proven to be an upper bound on the negative log-likelihood of the model distribution. Please refer to the Appendix A of [4] for the proof.\n\nFor conditional likelihood estimation, we derive two upper bounds, namely Eq. (5) and Eq. (6). Since the expectations in both equations are intractable to compute analytically, we estimate them using Monte Carlo sampling. While both estimators yield the same expected value, the one based on Eq. (6) exhibits lower variance, as it avoids the additional randomness introduced by sampling $r_t$. Please refer to Lines 694–700 for a more detailed discussion.\n\n[4] Sahoo et al. Simple and Effective Masked Diffusion Language Models. NeurIPS 2024.\n\n\n## Q4: Insight on Performance Advantage\nThanks for your question. **Both LLaDA and our autoregressive baselines (in Fig. 3 on Page 6) are trained on the same dataset**, as detailed in Lines 176–179. As for LLaDA’s performance advantages, some possible explanations are as follows:\n\n(1) When modeling the joint distribution of data, autoregressive models optimize only a single factorization order—typically from left to right. In contrast, LLaDA is trained to consider multiple conditioning directions, which may provide greater flexibility and lead to better generalization. The advantage of LLaDA on reversal reasoning is a concrete example of this property.\n\n(2) We hypothesize that removing the inductive bias of left-to-right generation in text and allowing the model to learn the generation order on its own may benefit model training. A similar observation was made in the Vision Transformer (ViT) paper [1], which showed that removing certain inductive biases from the data (e.g., translation equivariance in images) can enable Transformers to learn more effectively.\n\n\n## Q5: Open-Source\n\nThank you for your question. We will release our model weights, evaluation code, and training code. Details of our data collection process are provided in Lines 723–744. However, similar to several influential works [5, 6, 7], we choose not to release the training data to mitigate risks associated with the misuse of sensitive content or potential data biases.\n\n[5] Saharia et al. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. NeurIPS 2022.\n\n[6] Esser et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML 2024.\n\n[7] Kondratyuk et al. VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024.\n\n\n**If you have any more questions, we are happy to discuss them and will do our best to address them!**"}, {"review_id": "JTVbE5Srcm", "reviewer": "Reviewer_mRMF", "summary": "This paper develop language models based on diffusion models rather than autoregressive models. The paper examines this under scaling, demonstrating similar performance to equally sized autoregressive LMs, and improved performance on the reversal curse problem.", "questions": "- Outside of the reversal curse, what advantages are there to using diffusion models? Why might they have different capability profiles than their autoregressive counterparts?\n- .Can the authors comment on the novelty of the paper with regards to other diffusion models such as Inception's Mercury?\n- I think the paper would benefit from a more extensive related work section as some fundamental work in a similar vein is missing, e.g., https://arxiv.org/abs/2405.20519", "limitations": "Mostly yes, however, it would be interesting for the authors to comments on limitations of diffusion language models (rather than just the limitations of their own experiments (which they did well)).", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, clear, and succinct. I found it particularly easy to follow what the authors are describing. \n- There are clear advantages to diffusion models over autoregressive models (e.g., outperforming on tasks like the reversal curse). \n- The comparison from autoregressive models to diffusion models was done well, with a fair comparison based on identical model size and training data. Additionally, the evaluation of LLaDA as a tool was also strong, with comparisons presented against reasonable alternatives (e.g., Llama) despite differences in training data or exact size.\n\nMixed:\n- It's unclear how novel the approach is. The core of the method is the mask predictor and its role in the model's loss function. I have insufficient expertise in diffusion models to know whether this is a standard approach or truly novel. Nevertheless, the end result or application is seemingly novel. However, there are other diffusion language models now available. e.g., Mercury from https://www.inceptionlabs.ai/. It looks to me as those these have been contemporaneously developed -- yet they should probably be discussed in the related work section. \n\nWeaknesses:\n- When LLaDA underperforms against other models (e.g., Llama) it can be a significant amount (e.g., as seen in Table 2). On the other hand, when LLaDA outperforms other models it's often marginally. This doesn't seem to be discussed in detail. More broadly, the paper doesn't explore or speculate on why diffusion models may have different capability profiles than their autoregressive counterparts (outside of the reversal curse).", "was_revised": true, "final_justification": "This is a very strong paper. I still think it's a clear accept. The authors have engaged with comments meaningfully and offered to make alterations to improve the paper.", "rebuttal": "# Response to Reviewer mRMF \nWe thank Reviewer mRMF for the recognition of our contributions and the thoughtful comments. Below is our point-by-point response.\n\n\n## Q1: Discussion about Inception's Mercury\nThank you for your question. **The Mercury [1] technical report does not provide details on the model architecture, training loss, inference method, or other technical aspects.** Specifically, Section 2.1 of [1] briefly mentions that Mercury adopts a forward noising process and a reverse denoising process, but it does not describe the specific formulations of these processes or the loss function. In addition, with respect to the model architecture, [1] only states that Mercury uses a Transformer, without clarifying whether it is bidirectional or causal, or whether it follows the standard Transformer design or a modified Diffusion Transformer.\n\nIn contrast, our submission provides detailed information on model training and inference, and we will release our model weights, evaluation code, and training code.\n\nConsidering the novelty, we kindly note that new paradigms, conceptual advances, or novel discoveries also reflect forms of scientific innovation. Our findings first show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities including scalability, in-context learning, and instruction following are inherently depend on autoregressive models. Other reviewers, such as Reviewer YafR, also recognized our contribution and commented that our work “marks a significant contribution to the community” and “highlights the potential of non-AR approaches and broadens the design space for future language model development.”\n\n**Following your suggestion, we will include a discussion of the concurrent work Mercury [1] in the related work of the revised version.**\n\n[1] Inception Labs. Mercury: Ultra-Fast Language Models Based on Diffusion. arXiv 2025.06. https://arxiv.org/pdf/2506.17298.\n\n\n## Q2: Comparison with Autoregressive Models\nThank you for your question. In the following, we first explain the reasons behind the performance differences between LLaDA and LLaMA3, and then analyze the unique characteristics of LLaDA.\n\n### Performance differences with LLaMA3\nLLaDA underperforms LLaMA3 on certain benchmarks, which is primarily due to differences in training data scale and the use of reinforcement learning. Specifically, LLaMA3 was pretrained on 15T tokens, while LLaDA was trained on only 2.3T tokens. In addition, LLaMA3 was aligned using reinforcement learning, whereas LLaDA has not yet undergone any reinforcement learning alignment. \n\nThat said, we note that LLaDA significantly outperforms LLaMA3 on several benchmarks, such as GSM8K and Math, as shown in Tab. 1 (on Page 7). Moreover, we emphasize that **when trained on the same dataset, LLaDA is overall comparable to autoregressive baselines**, which follow the LLaMA3 architecture (see Fig. 3 on Page 6).\n\n\n### Unique characteristics of LLaDA\n\nWhen modeling the joint distribution of data, autoregressive models optimize only a single factorization order—typically from left to right. In contrast, **LLaDA is trained to consider multiple conditioning directions, which may provide greater flexibility and lead to better generalization.** The advantage of LLaDA on reversal reasoning is a concrete example of this property.\n\n\nWe further speculate that bidirectional training might also offer benefits in other tasks such as mathematical reasoning and code generation. The experimental results presented in Lines 775–821, where autoregressive sampling underperforms diffusion sampling for LLaDA, offer valuable insight into this hypothesis. However, compared to reversal reasoning, these tasks are more complex, making it more difficult to design experiments that directly isolate the benefits of bidirectionality. We leave a more thorough investigation to future work.\n\n\n\n## Q3: Advantages of Diffusion Language Models\nThank you for your question. Building on the bidirectional modeling capability of LLaDA, we would like to point out several additional advantages, beyond its strong performance on reversal reasoning as demonstrated in our submission.\n\n1. LLaDA has the potential to achieve faster sampling via parallel generation, as inspired by Fig. 5 on Page 25.\n2. As our work has demonstrated the potential of diffusion models for text generation, and considering their proven success in visual domains, LLaDA may be particularly well-suited to support future multimodal integration. \n3. Building on the hypothesis that bidirectional modeling may offer better generalization, diffusion language models could have an advantage over autoregressive models when data is scarce [2].\n\n\n[2] Prabhudesai et al. Diffusion Beats Autoregressive in Data-Constrained Settings. arXiv 2025.07.\n\n\n\n## Q4: More Related Work\nThanks for your question. Reference [3], mentioned in your comment, proposes a program synthesis approach using a syntax tree-guided diffusion model, jointly trained with a value model to guide the denoising process. It shows strong performance on inverse graphics tasks. Following your suggestion, we will include a discussion of this work in the related work of the revised version.\n\n[3] Kapur et al. Diffusion On Syntax Trees For Program Synthesis. ICLR 2025.\n\n\n## Q5: Limitations of Diffusion Language Models\nThanks for your comment that we did well in discussing the limitations of our experiments.\nRegarding the broader limitations of diffusion language models, one issue is that their architectures do not naturally support inference acceleration techniques such as KV cache. While there has been some recent work [4] exploring KV cache mechanisms for diffusion models, these approaches typically introduce small approximation errors. We will add this discussion in the revised version.\n\n[4] Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. arXiv 2025.05.\n\n**If you have any more questions, we are happy to discuss them and will do our best to address them!**"}, {"review_id": "8qvtzXxaAF", "reviewer": "Reviewer_APN5", "summary": "This paper introduces LLaDA, a diffusion language model trained from scratch. LLaDA shows comparable performances with auto-regressive models under the same experiment settings. After complete training process including Pre-training and SFT, LLaDA 8B is comparable with LLaMA3 8B. Moreover, LLaDA has advantages on some capabilities, such as the reversal curse, than auto-regressive models. LLaDA shows a promising direction for diffusion language models as general LLM.", "questions": "1. The key difference between language and image is in the variational length. Although the generation length is easy to decide for most of the benchmarks, it should still be fixed for real deployment. Is there any elegant solution for this?\n\n2. It seems LLaDA requires more FLOPs than ARM baselines. However, in Figure 5, the throughput of LLaDA is much better than ARM. Is there any optimization for ARM? After all, ARM decoding is memory-bound while LLaDA seems to be computation-bound. It's also weird that w/ KV cache and w/o KV cache's throughput are quite similar.\n\n3. To my understanding, there is a misalignment between training and inference on LLaDA. In training, all of the tokens are treated equally, while in inference, the prompt is given as a condition and always the clean token. Could you give any discussion for this?", "limitations": "Limitations are discussed in the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. LLaDA is the first strong pre-trained model comparable with well-known auto-regressive language models.\n2. The model recipe is simple and elegant, without any ad-hoc configuration and cherry-pick evaluation.\n3. Diffusion language model shows a capability advantage over auto-regressive models.\n\nWeaknesses:\nThere is no obvious weakness in the paper. However, as a new proposed model design rather than a pre-training technical report, most of the experiment part should focus on the comparison with baseline models. However, only Section 3.1 makes a fair comparison, the rest of the pair is on the comparison with other open source models. Although the performance is also strong, the **fair comparsion** is better to present the effectiveness of the proposed method.", "was_revised": true, "final_justification": "The author solves most of my concerns. I believe it is a good work.", "rebuttal": "# Response to Reviewer APN5\nWe thank Reviewer APN5 for the recognition of our contributions and the thoughtful comments. Below is our point-by-point response.\n\n\n## Q1: Comparison with Autoregressive Baselines\nWe appreciate your recognition of the fair comparison presented in Sec. 3.1. Specifically, we evaluate LLaDA and autoregressive baselines across six diverse and representative tasks, including general language understanding, mathematics, code generation, and Chinese. The comparison spans model sizes from 1B to 8B parameters and pre-training data scales from 30B to 2T tokens, covering a total of 40 checkpoints. In total, **we conducted 240 experimental configurations, enabling a broad and thorough comparison between LLaDA and autoregressive baselines across different tasks, model sizes, and data scales.**.\n\n\nAfter establishing a fair and comprehensive comparison with autoregressive baselines (in Sec. 3.1), we follow a common practice in the field by further comparing LLaDA with well-known open-source models (in Sec. 3.2). In Sec. 3.3, to highlight LLaDA’s advantage in reversal reasoning, we compare it with GPT-4o, a state-of-the-art model that far surpasses our autoregressive baseline. We believe this represents a stronger benchmark, and better highlights LLaDA’s advantage in reversal reasoning. Finally, Sec. 3.4 presents case studies.\n\n\n## Q2: Output Length\nThanks for your question. In response to your inquiry about an elegant solution, we believe that jointly modeling the response and its length offers a promising and elegant approach. Specifically, let $p$ denote the prompt, $r$ the response, and $l$ the response length. We consider adding a lightweight head to the Transformer to predict the response length, and model $p_{\\theta}(r, l|p)$ using maximum likelihood estimation as the objective function. We leave the implementation details to future work.\n\n\n## Q3: Throughput\nThanks for your question. Below, we would first like to clarify LLaDA’s throughput advantage, and then explain the efficiency differences caused by KV cache for autoregressive models.\n\n\n**The parallel decoding capability is the key reason behind LLaDA’s advantage in sampling efficiency on certain benchmarks.** As detailed in Fig. 5 (Page 25), on the GSM8K and Math benchmarks, when **each forward pass generates 4 tokens** (i.e., the number of sampling steps is set to one-fourth of the generation length), LLaDA achieves slightly better accuracy than LLaMA3 with KV Cache, while delivering 1.5× and 1.8× higher throughput, respectively.\n\n\nThe acceleration effect of KV cache for autoregressive models depends on both the sequence length and the hardware resources. As described in Lines 868 to 870, all efficiency analysis experiments were conducted on a single A100 GPU with 80GB of memory, using a batch size of 1. Both LLaMA3 and LLaDA operated with bfloat16 precision. For the HumanEval benchmark, where the average prompt length is relatively short at 132 tokens, enabling KV cache in LLaMA3 leads to a modest speedup of 1.27 times. In contrast, for benchmarks with longer prompts such as GSM8K, Math, and MBPP, whose average prompt lengths are 894, 680, and 628 tokens respectively, KV cache yields more significant improvements, with speedups of 2.09, 1.72, and 1.46 times accordingly.\n\n\n## Q4: Is there a train-inference misalignment?\nThank you for your question. **The inference setup is consistent with the SFT stage and can be regarded as a special case of the pre-training phase.** During unsupervised pre-training, the model learns to model an unconditional distribution and therefore treats all tokens equally. During SFT, it models a conditional distribution, so the prompt is kept clean and the loss is computed only on the response. At inference time, the model samples from a conditional distribution, and therefore the prompt is kept clean as well.\n\nIn comparison, this process is consistent with that of autoregressive models, which also treat all tokens equally during pre-training but **compute the loss only on the response during SFT**. During inference, they likewise sample from a conditional distribution, generating from the prompt as in the SFT stage, rather than generating unconditionally as in pre-training.\n\n**If you have any more questions, we are happy to discuss them and will do our best to address them!**"}]

Author rebuttal:

Meta-review: This paper presents a diffusion-based language model scaled to 8 billion parameters and trained on over 2 trillion tokens. The primary contribution is the empirical demonstration that a non-autoregressive architecture can achieve performance comparable to, and in some cases exceeding, well-established autoregressive models of a similar scale (like Llama 3 8B). 

All four reviewers are in agreement on the paper's acceptance and identify several key strengths including comprehensive evaluation, strong model performance, and novel capabilities (like addressing the reversal curse) 

The initial reviews raised several points for clarification. Reviewers questioned the model's latency and throughput compared to highly optimized autoregressive models with KV caching; they questioned it being called a diffusion model at all. They also asked for more details on handling variable length generation, as the model requires a fixed output length hyperparameter. The reviewers were satisfied by the author response.

For the camera-ready version, the authors are expected to integrate the detailed clarifications and discussions from their rebuttal into the main body of the paper. This includes: (1) the expanded discussion on inference efficiency and throughput comparisons, (2) the clarification of the model's theoretical relationship to, and distinction from, other discrete diffusion models and masked language models, and (3) the discussion on the current approach to variable-length generation and its limitations.

The key reason for my recommendation for oral is the paper's high impact: it establishes a viable, non-autoregressive pathway for foundation models, opening significant new avenues for research
Decision: Accept (oral)