Scaling Diffusion Language Models via Adaptation from Autoregressive Models
Authors: Shansan Gong
Shivam Agarwal
Yizhe Zhang
Jiacheng Ye
Lin Zheng
Mukai Li
Chenxin An
Peilin Zhao
Wei Bi
Jiawei Han
Hao Peng
Lingpeng Kong
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models; diffusion language models; text diffusion models; discrete diffusion models
Primary area: foundation or frontier models, including LLMs
TL;DR: We developed advanced diffusion language models by adaptation from autoregressive models, focusing on small, medium, and 7B sizes.

Abstract: Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (127M-355M-7B) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions.

Published as a conference paper at ICLR 2025
SCALING DIFFUSION LANGUAGE MODELS VIA ADAPTATION FROM AUTOREGRESSIVE MODELS
Shansan Gong"), Shivam
Agarwal"?, Shivamm
Yizhe Zhang', Jiacheng Ye', Lin Zheng'
Mukai Li", Chenxin An', Peilin Zhao', Wei Bi Hao Peng". Jiawei Han". Lingpeng Kong'
'The University of Hong Kong 2 University of Illinois at Urbana- -hhaa-hhaaa 3 Apple 4 Tencent AI Lab
sansa connect h.k.hkkkk hk, shivama 2 www.ammmeiiiiiiiss...... wwwiiiiis......... edu
ABSTRACT
Diffusion Language Models (DLMs) have emerged a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale com- pared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we pro- pose adapting these models to build text diffusion models. We demonstrate con- nections between AR and diffusion modeling objectives and introduce simple continual pre-training approach for training diffusion models. Through system- atic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform carlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (27M-3355-77B capable of generating fluent text, performing in-context learning, filling the middle without prompt re-ordering, and following instructions. https: /ttps:///itiee //gpthbb.ttbbtt/// com/ HKUNLP /ttf//iffuuu FfuLLaMA
INTRODUCTION
Large language models (LLMs) have ushered in a new era of artificial intelligence, demonstrating remarkable capabilities in generating high-quality text, in-context learning, and following complex instructions (OpenAI, 2023; Touvron al., 2023a). These advancements the scaling up of autoregressive (AR) language models. During both training and inference, these models leverage vast datasets and billions of parameters, employing strict left-to-right sequential process for memorization and generation. This approach has resulted in the emergence of intelli- gence capable of tackling diverse tasks (Wei et al., 2022a; Hoffmann et al., 2024). However, the ultimate upper limit of intelligence achievable through this paradigm remains an open question. While AR mechanisms form the foundation of current LLMs, they are not without limitations (Lin et al., 2021). Notable challenges include difficulties in future planning (Bachmann & Nagarajan, 2024; Hu* et al., 2024; Xie et al., 2024) and self-correction (Huang et al., 2024). These constraints have spurred researchers to explore alternative architectures for next-generation LLMs.
primarily rooted in
compelling direction current research focuses on the development of text diffusion models (Li et al., 2023b). Building upon the rapid evolution of diffusion models in various domains (Ho et al., 2020; Nichol & Dhariwal, 2021; Ramesh et al., 2021), innovative text diffusion models (Li et al., 2022: Lou et al., 2024) have opened up new possibilities for text generation. A unifying insight across these models is the potential of diffusion language models (DLMs) for controllable (Venka- traman al., 2024), any-order, and parallel text generation (Gong al., 2023a). Notably, DLMs exhibit promising capabilities in intermediate token correction (Ye et al., 2024b) and global plan- ning (Zhang et al., 2023), thereby addressing key limitations inherent in the AR approach.
"Equal contribution

---

Published as a conference paper at ICLR 2025
Despite the promising potential of text diffusion models, the relatively small model size limits the competitiveness of DLMs compared to AR models. Existing state-of-the-an DLMs such as Plaid IB (Gulrajani & Hashimoto, 2023) and SEDD (Lou et al., 2024) are relatively small in size (127M- 1B parameters) and under-trained, with less than 400B tokens of training data. This substantial gap in scale prevents fair comparisons with larger AR language models on many advanced capabilities and tasks, such as chain-of-thoughh reasoning abilities on complex mathematical benchmarks. Re- cent approaches (Ye et al., 2023) attempt adapt LLaMA models DLMs based on masked language modeling (He et al., 2023). However, they find that the base model capabilities are lost during their adaptation stage. Pre-training at such a scale is extremely resource-intensvve and the challenge is even more pronounced for diffusion models. These models lack the computational optimizations that have been developed for LLMs (Samragh et al., 2024) and require significantly more resources than their AR counterparts, as noted by Guirajani & Hashimoto (2023).
Given these scaling challenges, pre-trained LLMs emerge as an invaluable resource that we can leverage, considering the extensive computational efforts already invested in their development. This strategy aligns with recent trends where new models are scaled up or adapted to new architectures using existing LLMs (Wang et al., 2024; Zhang al., 2024c). However, building DLMs through adaptation from AR models is non-trivial due to fundamental differences in their language modeling objectives. Two key distinctions present significant hurdles. First, AR models employ causal mask- ing to prevent future information leakage, whereas diffusion models utilize bi-directional attention masks. Second, an AR LM processes clean inputs predict subsequent tokens at each step, while a diffusion model operates on noisy inputs to predict their denoised versions.
To overcome these challenges, we propose a simple adaptation approach that bridges these discrep- ancies. We unify their modeling objectives ($3.2) ) and address the architectural differences by break- ing the causal masking bias in AR models through attention mask annealing ($3.3). Additionally, we inherit the shift operation from AR models ($3.3). This streamlined adaptation recipe enables us to construct a pre-trained DI.M that can effectively compete in the arena of L.MMs. Building on this approach, we leverage the Wen (Penedo et al., 2024) and SlimPajama (Soboleva et al., 2023) pre-training corpora to continue training small and medium- sized DLMS based on GPT2 (Brown et al., 2020), and further train up 7B model based on LLaMA2 (Touvron et al., 2023b).
Our experiments provide a comprehensive comparison between AR LMs and DLMs across lan- guage modeling, reasoning, and infilling tasks. The evaluation encompasses diverse settings, in- cluding zero-shot, few-shot, and finettunigg scenarios, addressing the limitations of relying solely on perplexity in previous works (Shi et al., 2024). Our contributions and empirical findings include:
We demonstrate that by narrowing the gap between AR models and DLMs, it is possible to con- vert 127M-7B AR models (GPT2 and LLaMA2) into DiffuGPT and DiffuLaaMA with training on less than 200B tokens. Notably, DiffuGPT outperforms GPT2 in most tasks.
We adapt 7B AR models to DLMs, greatly expanding the expertise compared to smaller-sized diffusion models. DiffuLLaMA emerges as the state-of-the-an art DLM, exhibiting in-context learn- ing, code generation, and strong infilling capabilities. Its generation speed competitive with AR counterparts for unconditionally generating 1024 tokens using 256 diffusion timesteps.
We provide comprehensive benchmark for DLMs and release our adapted diffusion models (127M, 355M and 7B) along with open-source adaptation code, efficient fine-tuning scripts, and evaluation toolkits.
2 PRELIMINARY AND NOTATION
Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Song al., 2021b) are latent variable generative models characterized by a forward and a reverse Markov process. We denote Xa ~ (xata(Xp)) as the variable following the data distribution, and ~ ((x)) as the noisy variable of Xy at time t, where the maximum time is T. The forward pro- cess q(xir Xx)) = IT= q(x, xx-1) corrupts the initial data Xe into a sequence of increasingly noisy variables X1.7. Accordingly, the backward Markov process models the joint probability as Po(XorT) = Pa (x7) II Po (x,-1 (x), which gradually denoises x, to reconstruct the original data Xg- Parameters 0 are learned by minimizing the negative log-likelihood of xo. which can be opti-
2

---

Published as a conference paper at ICLR 2025
Language Models
Adaplation
Adapted Diffusion Language Models
Train
Predict
Logilr
Labala
Causal musk
Attention aonculing
Fill
Logee
Predict
Lubeb
Ati
LN CE
Aleenioo weighs
Misk states
Figure 1: The overview of our approach to adapt autoregressive (AR) models to diffusion models. Left: The shift operation in AR models enables the output layer h; to approximate the distribution of next tokens Fi+1 in hidden representations through the cross entropy (CE) loss. Middle: We remove the causal mask gradually during training eventually making
model bi-directional. Right: inside the diffusion models we shift the logits to compute the loss with the next token (i.e., the loss on h2 would be with respect to 1++1). while perceptually, the diffusion models still functioning as recovering the original signals (since h; corresponds to 2441 in AR loss).
mized through the evidence lower bound (ELBO),
- llo Po (xx) I (8((x100)] Po [1]] + DKL (xx.(((xr/x))))/o((x + Lr;
Po (xo
(1)
For continuous text diffusion (Li et al., 2022; Gong et al., 2023b), at each forward step, perturbations are applied according to q(x, x1 = N(x: vI -. B,I), where B, E 1) represents different scales across
with LT LT = Eytx, (x.) [DKT (q(x;-1 1xx, xo) (x
time steps such that xy _V(O. I). In the case of discrete denoising models (Ho al., 2020; Austin et al., 2021; Zheng et al., 2024a), the forward process is defined as a categorical distribution q(x, (xx)) size K, Q E [o, 1]
Cat m:;; Q. .;;-1), where each x, 10 is one-hot vector with vocabulary is the transition matrix, and each entry QQlii denotes the probability of transition from the state i We build on the formulation of absorbing discrete diffusion (Austin et al., 2021), I), which specifies Q. (1 I + 3, 1m We denote 1 as an all-one vector of size K and m as the one-hot encoding of a special [MASK] token in the vocabulary. Therefore, the transition matrix, Q: indicates that with probability - 3r. 2; remains unchanged; otherwise, it transitions to m. becoming absorbed into [MASK]. Letting Q, = I Q = a a+) 1m
and ar II: (1 3r), the distribution of æ, conditional on 25g given by
(2) since wo is a one-hot vector and thus 1 æ = We expect ar to approach such that the full noise data ær equals m with probability 1.
((xx(<)) Cat(æ;; Q xoo) a, + (1 a. am)m1 x 0g + (1 a;)m,
The discrete time representation of E [], restricts o, to fixed noise ratios. To avoid this bias and enable sampling from any noisy representation, span any point within [0, 1] (Kingma et al., 2021; Shi et al., 2024; Zhao et al., 2024; Ou et al., 2024). Continuous-time sampling is equivalent dividing [0, 1] into T intervals and where T - xc. For any 0 < s < I < 1, the forward process generalizes to g(x, x,). We will use this continuous-time notation in the following sections.
continuous-tiin time sampling, allowing to
MODEL
We begin by formulating the continuous-time time discrete diffusion process ($3.1) and establishing connection between the discrete diffusion and autoregressive objectives ($3.2). Based on this equiv- alence, we propose adaptation approach ($3.3) and sampling algorithm ($3.4) for diffusion models adapted from AR models. The whole process is illustrated Figure 1.
3.1 CONTINUOUS-TIME DISCRETE DIFFUSION PROCESSES
Following Eq.2 and ((0:/20)) Ex, q(,,, ar, )4(2x(20)) the forward transition distribution be- tween arbitrary points < I. can be derived as
qxx,)x,) Cat( Q.(
Ot xx (1 m
(3)
as
a2

---

Published as a conference paper at ICLR 2025
with Q,1= = Q. = _ 1mm The corresponding backward transition distribution conditional on zo is also available closed form,
a,
q( xo) (2, (2,
G. a.-as 2o +
1-as m I-
if xt m,
(4)
20
if xt + m.
In discrete diffusion processes, we aim to approximate the backward transition distribution (a Wo) using a denoising model po(a, fo(x)), where fo(#)), an approximation of To, is usually the output of neural networks such as a transformer (Vaswani et al., 2017). We can define the denoising model to have a similar form of backward transitions as po x2)) = T-a; (x)) + 1-a.m. m. According to the training objective in Eq.l, the KL-divergence of Cr at each step can be simplified to a reweighted cross-entropy function,
DKL (q(x, x2, (o) ( [x)) 0gg at mm log fo())))
(5)
1 - az
where da.6 is the indicator function for a = b. we take the limit and let T + oo, the first two terms of Eq.1 will approach 0 and some constant, respectively. Thus the evidence lower bound (ELBO) effectively becomes Lr and
1 at Ote
Jim CT CT= T-00
Ee(xe (x)) ( m0 log fe( dt.
(6)
The full derivation is listed in Appendix A.2. The same form of ELBO which is invariant to noise schedule but related to the signal-to-noiss ratio (SNR) is also introduced in Kingma et al. (2024). Following Austin et al. (2021), we choose the noise schedule a, = 1 - f. then T- :. The previous discussion focused on the single token It and can be applied independently to text sequence of N tokens x, = [ .... "."). During training, we not compute integral loss in Eq.6 for efficiency consideration; instead, we sample for each data point. The final loss at is
(2021); Shi
C C
1 ( Bobxa (Nc0)
[-
Σ
8x; m mx(x))
T log (x )
where fe(x )n denotes the whole input sequence is fed into the transformer model and the n-th output token indexed.
3.2 UNIFYING LANGUAGE MODELING ORIECTIVES
The training objective of autoregressive (AR) language models is the negative likeliiel of each ground-truth token provided the preceding tokens,
N
C
Σ (xx) log fo 1mm (x-m--))m- )-1-
(8)
Comparing Eq.8 against Eq. we note that while both take the form of cross-entropy functions, Eq.7 includes additional reweighting term 1 and an indicator function ōxp They result from the definition of discrete diffusion processes ()..1). The reweighting emphasizes smaller f where x; contains fewer masked tokens, and this can be regarded the importance sampling (Nichol & Dhariwal, 2021). The indicator specifies which tokens are masked for prediction. The AR training objective Eq.8, on the other hand, constrains the context be unidirectional via attention masking and shifts the targets so that each token predicts the next token instead of itself. These discrepancies form the basis of our adaptation framework, which detailed in $3.3.
In fact, an alternative way to understand AR modeling, through the lens of diffusion models. is to consider a diffusion process where the forward pass deterministically masks right-to-left and token- by-token (Austin et al., 2021; Hoogeboom et al., 2022). This yields backward process generating one token at a time from left to right, running with T = N denoising steps in total. As discussed in Austin et al. (2021), the loss objective of this diffusion process is equivalent to standard cross- entropy (Eq.8) commonly used to train AR language models. This crafted diffusion process for AR models represents special case of discrete diffusion ($3.1), yet it is limited to unidirectional con- text and sequential token generation. In contrast, general discrete diffusion processes can leverage bidirectional context and support parallel generation in arbitrary orders.
4

---

Published as a conference paper at ICLR 2025
3.3 ADAPTATION
Building on the connection between AR modeling and discrete diffusion processes, we construct an adaptation recipe next. Figure shows an overview of our adaptation approach. We use attention mask annealing, shift operations, and emeeemmbd free architecture to narrow the differences between AR and DLMs.
Algorithm Adaptation Training
Algorithm 2 Sampling
1: Input: network fo initialized by existing models, training corpus Pdata (ar) :N)), mask token m.
Input: Trained diffusion model fo. sampling al-
gorithm T. mask token m. start token s.
2: Output: model parameters 0.
2: Output: generated sample Zo.
3: repeat
3: Initialize
4: Draw w Piataa and set labels c- wd
4: for T...... do
Sample E Uniform(O, 1)
5: Forward logits < fo(a)!")
6: Sample -o!-V ~ (
6: Sample #.. (ategorical((((ooits))
7:
Anneal the attention mask alln mask
7:
for 1,..., do
Forward logits <- So (wt with mltks
8:
ai-1 = q(>) |x mū) Eq.4
9: Right shift logits by one position
9: end for
10: C. = 1 2m..m. CE(logits, labels) b Bq.7
10: Right shift w [s,
11: Backprop with L, and update 8
11: end for
12: until end training
12: Return x0
Attention Mask Annealing The prediction of the n-th token, given all preceding tokens, fo(x!"-1) is usually implemented by causal attention masking in transformer- AR language models. As shown in Figure I, causal attention masks set all entries in the upper triangle of the self- attention matrices to zero, so each token cannot attend to its respective future tokens. Such causal masking prevents the model from learning right-to-left dependencies for more general diffusion pro- cesses. To address this limitation while preserving left- left-to-right conditionals during adaptation, we introduce an incremental annealing process from causal masks full attention matrices. During annealing, the causal mask is not immediately removed; instead, it is retained at a controlled ratio, as shown in the middle part of Figure At each training step, we sample the amount of context from the right side and progressively increase this amount till we obtain the full attention mask.
Shift Operation AR models also apply a shifting operation, where the target output is the input sequence shifted left by one position. In other words, the prediction target of the 1)-th token is the n-th token, contrasting with typical diffusion models that try to predict masked tokens at their original positions. When initializing text diffusion models with AR model parameters, the model would tend to output the hidden representations of the shifted input sequence. If we continue to optimize the cross-entropy objective based on the original token positions, the model struggles to adapt due to misalignment between input and output. Instead, we maintain the shift operation (Algo.1, line 9), treating the output logits at each position as corresponding to the next token. When calculating the objective, we align prediction targets that the diffusion model learns to recover the original signals. This process is illustrated in the right panel of Figure 1.
Time- tanne--meedding-iiee Architecture Many diffusion models for text generation (Li et al., 2022; Dieleman et al., 2022; Guirajani & Hashimoto, 2023; Lou et al., 2024; Shi et al., 2024) incorporate time embedding layers to represent the information of current timesteps t, which can explicitly be challenging
indicate the noise scale of the input noisy data. While inferring these timesteps for image diffusion models (Ho et al., 2020; Li et al., 2024), some discrete text diffusion models (He al., 2023) assert that timesteps can be easily learned implicitly based the number of mask tokens. Since AR models are not equipped with time embedding layers, we also choose not to use the time embedding, resulting in no additional parameters compared previous diffusion models.
3.4 SAMPLING
Following Shi et al. (2024), we initialize æy with all [MASK] tokens and then sample tokens cording to the time reversal g( [ Φp) in Eq.4. At each timestep, if æ, is a mask, it will jump to the predicted æg at time with probability as-ax, T-a: After T iterations, the model generates the full sequence. Since our adapted models are trained with the shift operation, each sampling iteration,
5

---

Published as a conference paper at ICLR 2025
shift back the generated sentence and prepend start token before the next forward pass (Algo.2, line 10). Usually larger T requires more interactions of computation, and can yield texts in higher quality, and this trade-off can be controlled easily through T. Through experiments, output generated by diffusion models is diverse and scattered. Therefore, for conditional generation tasks, we improve the sampling procedure to ensure that only tokens with high probabilities from neural networks are denoised (Ghazvininejad et al., 2019; Chang et al., 2022; Zheng et al., 2024a), so that the model could predict tokens mostly relevant the input. In addition, existing sampling techniques for AR language models, including top-k and nucleus sampling (Holtzman et al., 2020), can be seamlessly applied to diffusion models as well.
find that the
4 EXPERIMENT
4.1 ADAPTATION SETUP
DiffuGPT We use the 30 billion tokens' random split from the Wen dataset (Penedo et al., 2024), an improved corpus than Open WebText (Gokaslan & Cohen, 2019) used in prior DL (Lou et al., 2024), continue training GPT2 base (Radford et al., 2019). We use sequence packing. logits shifting, and 10K-step attention mask annealing to transform GPT2 to DiffuGPT.
Laff We continue pre-training LLAMA-2-7-IIP (Tou- vron et al., 2023a) on a mixture of SlimPajama (70%) (Soboleva et al., 2023) and Starcoder (30%) (Li et al., 2023a) data follow- ing TinyLLaMA (Zhang et al., 2024a). We randomly sample 65 billion tokens from this mixture and use sequence packing with context length of 2048. For efficient implementation we enable
10
DiffGPT-12MM
DIT-GPT--GGM
MA-
Training
flash-attention 2 (Dao, 2024) and directly use bi-directional at- tention without attention mask annealing.
For both adaptation settings, we employ full parameter finetun- ing with bf16. Please refer to Appendix B.2 for details. We plot the training loss curve in Figure 2. We train DiffuLLaMA on 60B tokens and achieve lower loss compared to 127M and 335M models, suggesting a scaling trend similar to that of AR LLMs (Kaplan et al., 2020). We also note that there is still scope for training more, since the model does not show signs of saturation.
Tralning Tokeus (Billio)
Figure 2: Training loss over to- kens for various model sizes of our adapted diffusion models.
4.2 EVALUATION SETUP
Previously developed diffusion language models (Gulrajani & Hashimoto, 2023; Lou et al., 2024; Shi et al., 2024; Ou et al., 2024) evaluate model performance using zero-shot perplexity on bench- mark datasets. However, this metric alone does not fully capture a model's capabilities for several reasons. First, lower perplexity does not always correlate with human-like content, even in autore- gressive models (Kuribayashi et al., 2021). Additionally, the loss from text diffusion models only indicates an upper bound on negative log-likelihood. While Kingma et al. (2021); Shi et al. (2024) demonstrate that the ELBO is invariant to the noise scheduler. discrepancies between continuous diffusion. discrete diffusion, and autoregressive loss still hinder fair comparisons across different model types. Given the ample evaluation benchmarks (Gu et al., 2024) for LLMs, we propose a more comprehensive evaluation for diffusion models.
Tasks and Metrics We consider TriviaQA (Joshi et al., 2017) to test the reading comprehension of models and last word completion task Lambada (Paperno et al., 2016) to test how models cap- ture long-range dependencies in text. These two tasks are measured by exact match accuracy. We also test for common sense reasoning tasks HellaSwag (Zellers et al., 2019), Winogrande (Sak- aguchi et al., 2021), SIQA (Sap et al., 2019) and PIQA (Bisk et al., 2020), all of which involve multiple-choice questions assessed by accuracy. On grade school math problems GSM8K (Cobbe al., 2021), we follow Ye et al. (2024b) in finetuning setting using the augmented symbolic data to test the COT (Wei et al., 2022b) math reasoning abilities of diffusion models. Following Shen
'This is the total number of tokens used; however, our effective training tokens exceed this count, meaning that we train for more than one epnch.

---

Published as a conference paper at ICLR 2025
Table 1: Comprchensive evaluation of different diffusion language models and the same size pre- trained autoregressive models. There are 3 types of these models: AR for autoregressive, DD for discrete diffusion and CD for continuous diffusion. For the infilling task, we use ROUGE-1/2/L. score; for other tasks, we use the accuracy (%) metric. indicates we finetune GSM8K on models; other tasks are all zero-shot setting. Numbers the () indicate that AR models are only given pre- fix for infilling tasks. We bold the best performance among diffusion language models and underline results that surpass their base models.
QA
Word
CommonSense Reasoning Wino. SIQA
Math GSM8K
Infilling
Model
Size
Type
TriQA
Lamb,
HSwag
PIQA
ROCStorics
Codc
GPT2-S
127M
AR
4.0
25.9
29.9
48.5
35.7
62.1
44.8
(7.8/0.8/7.4)
(1.6)
SEDD-S
170M
DD
1.5
12.4
30.2
50.1
34.4
55.6
45.3
11.9/0.7/10.9
0.7
DiffuGPT-S
127M
DD
2.0
45.0
33.4
50.8
37.0
57.7
50.2
13.7/1
0.3
GPT2-M
355M
AR
6.7
37.7
38.3
50.7
37.7
67.4
45.6
(8.6/0.9/8
(2.6)
SEDD-M
424M
DD
1.8
23.1
31.5
49.0
35,4
56.1
53.5
13.1/1.4/12.2
0.5
DiffuGPT-M
355M
DD
3.8
60.5
37.2
52.6
39.0
59.6
61.8
18.7/2.7/17.0
2.9
PlaidIB
L.3B
CD
1.2
8.6
39.3
51.3
32.3
54.5
32.6
12.1/1.1/11.2
0.1
LLaMA2
7B
AR
45.4
68.8
74.9
67.1
44.8
78.3
58.6
(11.6/2.1/10.
(1.7)
DiffuLLaMA
7B
DD
18.5
70.9
58.7
56.4
43.2
63.3
63.1
23.3/5.5/21.2
15.5
et al. (2023), we also test the story infilling tasks using ROCStories (Mostafazadeh et al., 2016) and evaluate using ROUGE score (Lin, 2004). To test the code infilling, we adopt Humaneval (Bavar- ian et al., 2022a) single line infilling task, which is evaluated by pass@1 rate. We evaluate Dif- LuLLaMA's math reasoning and in-context learning ability by evaluating on MAWPS (Koncel- Kedziorski et al., 2016) consisting of math word problems and SATMATH from AGI-eval con- sisting of math problems from SAT exam (Zhong et al., 2024). We base our implementation on Im-eva I luatii con-harness (Gao et al., 2024) and re-implement all tasks across models to en- sure a fair comparison.
Implementation Details For pre-trained diffusion language models, we mainly use continuous dif- fusion (CD) model Plaid IB (Gulrajani & Hashimoto, 2023), discrete diffusion (DD) model SEDD (Lou et al., 2024) with different sizes as baselines. MD4 (Shi et al., 2024) and RADD (Ou al., 2024) are based on and compared with SEDD, so we mainly compare SEDD. For autoregressive (AR) baselines, we consider the base models from which our models adapt. We implement infilling tasks for AR models by feeding the prefix and cutting off the generation length using the oracle length, considering that these AR models are not supporting infilling. For the sentence completion task, T' is the exact number of ground truth tokens for DD and 32 for CD. For 4 multi-choices tasks from commonsense reasoning, we compute the loss (Eq.6) of each choice (averaged by token) and choose the one with lowest loss (perplexity). For GSM8K finetuning, we use parameter-efficiee LoRA tuning (Hu al., 2022) for DaMA The decoding T are set 32 by default. The detailed settings are in Appendix B.3.
4.3 LANGUAGE MODELING CAPACITIES
Benchmark performance According to Table 1, the results on diverse tasks demonstrate that our adapted diffusion models achieve the state-of-the-ant results among all existing diffusion language models (DLMs). We observe that diffusion models with larger parameters show improved perfor- mance, likely due to better base AR models. Laf'u performance still falls short of the ILLMA2 model. This drop in performance is likely because aaa is trained on small subset of SlimPajama and Starcoder data. We believe more training tokens can help improve these numbers. TriviaQA and PIQA are significant challenging for DLMs, probably because they require specific physical knowledge, such as the capital of a city the boiling point of water; while models are trained on 30B-70B tokens, which may be insufficient to preserve the general knowledge in the original LMs (Ke et al., 2023).
In tasks that require more extensive global reasoning, such as complex mathematics and coding, DLMs consistently exhibit better performance compared to AR models that rely solely on left-to-
7

---

Published as a conference paper at ICLR 2025
right modeling capabilities. Remarkably, DLMs demonstrate their strengths in infilling tasks. Reg- ular LLMs like LLaMA2 are not trained for filling-in-the-men in-the-middle (FIM) tasks like those in Roziere et al. (2023), making them incapable of handling infilling. Considering this, we do not provide the suffix information to the model, which might result in an unfair comparison. But the FIM requires re-arranging the order of pre-triningginference sequence with special tokens (Zheng al., 2024h), while diffusion training naturally supports this in its objective modeling.
Tasks in Table mainly measure conditional modeling abilities, where Plaid 1B performs unsatisfac- torily for conditional generation tasks even though with IB parameters. We attribute this result to the gap between the continuous diffusion modeling and discrete text representation; contrast, discrete diffusion models align more closely with AR modeling, naturally supporting conditional generation. Despite this, as illustrated in Figure 3, Plaid 1B demonstrates its strength in unconditional genera- tion, highlighting its language modeling capabilities as a generative model. These findings reveal that the previous evaluation based on the perplexity of test data is too general to accurately assess the model's 's true capabilities, while our evaluation offers more nuanced benchmark.
Unconditional Generation We evaluate the quality of text unconditionally generated by DLMs in Figure 3. The perplexity is measured using GPT2 large, consistent with the prior work (Lou et al., 2024), where the data of MD4 (Shi et al., 2024) is soureed from its original pa- per. To make sure low perplexity is not brought by re- peated content, we assess the distinct 2-gram diversity of the generated text. Our model achieves low perplex- ity while maintaining a high level of diversity, validating that the DiffuGPT series excels in fluent text generation. As the number of decoding steps increases, thereby ex- tending the test computing time, the fluency of uncondi- tional generation improves. Similarly, increasing model size also contribute to better performance. An increase in generation perplexity is often associated with slight de- crease in diversity, which is a common phenomenon. No- tably, DiffuGPT outperforms both SEDD and MD4 mod- els. particularly at lower step counts (c.g., 64 steps), while as the continuous diffusion models, Plaid 1B needs more decoding steps to generate more fluent texts. DiffuGPT thus exhibits significant advantage on less sampling time. We outline the decoding hyperparameters and show the diversity changes across different settings in Appendix C.I, which also includes generation cases.
1.0
240
Genorrttvv Perplexity
0.9
D Iect 2- mmm
120
Bura-00 14:5-000M
60)
0.7
30
V
20
32
64
128
512
10220.0
0.6
Decoding Steps
Figure 3: Quality evaluation for un- conditional generation, with perplexity measured by GPT2 large and distinct 2- gram diversity.
4.4 ANALYSIS ON DIFFULLAMA
We validate that increasing the size of adapted
Table 2: Performance on math/QA benchmarks
DLMs significantly enhances the performance of downstream tasks in Table I. Further, aim to assess if the 7B model demonstrates in- context learning and reasoning capabilities sim- ilar to AR LLMs. Table 2 presents the exact match accuracy between gold labels and predic- tions generated by DiffuLLaMe across zero- shot (ZS), few-shot (FS), and FS with chain- of-thought (CoT)) scenarios. Besides, we de- ploy the seff-consistency approach (Wang al., 2023), considering that small DLMS can indeed benefit from this technique (Ye et al., 2024b). We use majority vote to choose the best answer from 3 individual predictions, and also report the hit rate @k k with k = 3, which measures whether any of the k predictions include the correct swer, serving as a reference for the model's 's upper bound. For in-context learning (ICL) evaluations, give 4-shot math tasks and 2-shot on TriviaQA.
(t). We compare of DiffuLLaMA with zero-shot (ZS), few-shot (FS), self-consistency (SC), hit@k and chain-of-thought (CoT) prompts.
Settings
MAWPS SATMath TriviaQA
LLaMA2
63.5
24.5
45.4
DiffuLLaMA-ZS DiMAL.L.MA-FS
9.7
<1
18.5
31.3
23.6
20.9
DiffL.LaMA-SC Difful..aM--G
33.1
27.7
26.0
40.8
57.7
34.1
DiffuLLaMA-Co1
28.7
9.5
8

---

Published as a conference paper at ICLR 2025
The performance improvement from zero-shot to few-shot settings suggests that DiffuLLaMA can learn from ICL examples, particularly in following to the format of answers as we observe. We hy- pothesize that the adapted model retains some of the abilities from the base AR model. We randomly select the ICL demonstration here and anticipate that advanced ICL. strategies in LLMs (Wu et al., 2023) could yield potentially higher results. The self-consistency offers LMs with an effective ap- proach to test-time scaling (Snell et al., 2024), and DiffuLLaMA shows that it can also leverage this method. Furthermore, we report the hit rate results generated candidate answers, highlighting the model's 's potential to produce the correct answer. This reveals that the current model exhibits high un- certainty about its responses, leading to temporarily suboptimal performance. We also observe that adding step-wise solutions in the in-context example (CoT) leads to a drop in performance, likely due to the absence of instruction tuning, similar to the findings LLMs (Ouyang et al., 2022). We will leave instruction tuning as the future work as Ye et al. (2023) show that text diffusion model can benefit from instruction tuning. In summary, we show the potential capabilities of Laff which motivates us to further investigate the scaling of diffusion models.
4.5 DISCUSSIONS
Ablation Test on GSM8K-symbolic Direct ablation on adaptation training is costly; hence, we conduct pre- liminary experiments to determine the adaptation recipes. Following Ye et al. (2024b), we finetune models on the augmented GSM8K symbolic dataset using various base models and training objectives. The models are either trained from scratch (random initialization) or initialized with GPT2-S/M weights. Training objectives includes autoregressive training with a causal mask, continuous diffusion loss (CD), and discrete diffusion loss (DD). As shown in Table 3, different training objectives yield com- parable results when training from scratch. However, when using GPT2 as the base model, the CD loss performs worse than both the DD and AR losses. We attribute this the better alignment of DD and AR losses as discussed in $3.2. Previous contin- uous diffusion models (Dieleman et al., 2022; Guirajani & Hashimoto, 2023) has reparameterized the estimation of embeddings into the CE loss. However, adapting diffusion models from an AR model in continuous space necessitates additional projection from the embedding to a categorical distribution, increasing the difficulty of adaptation.
Table 3: Ablation test for adaptation ap- proaches on GSM8K symbolic dataset. CD is for continuous diffusion and DD is for discrete diffusion.
Base models
Random
GPT2-S
GPT2-M
Autoregressive
30.5
44,8
45.6
27.9
19.2
20.2
DD-w/o shift
33.5
34.5
DD-w/o anneal
43.3
47.2
28.0
45.4
49.7
For DD loss, removing attention mask annealing and shift operations both degrade performance, indicating the efficacy of our approaches. The mask annealing has minimal impact, so we choose to omit it for 7B adaptation to simplify implementation using flash-attention 2.
Direct DD loss finetuning on GPT2 achieves accuracy of 45.4 and 49.7 for small and medium models, respec- tively, outperforming GPT2 AR finetuning. However, fine- tuning from already adapted diffusion language models (DiffuGPT) yields accuracy of 50.2 and 61.8 (Table 1).
Single Batch Decoding Time (Sec)
40
MAMAD
INGEuLLabA T-312
DeFalLaMA
30
DiLaLAMMA T--4
This demonstrates the superiority of DiffuGPT as the cur- rent best diffusion base model at this size and highlights that better base model leads to improved results. Even with the
20
iusrtsing decrensing T
same DD loss, DiffuGPT's finetuning converges faster and achieves lower loss as shown in Appendix C.2.
10
Inference Speed AR models usually utilize key-value caching (incremental decoding; Ott al. 2019) to enhance throughput during decoding. However, due to the nature of sequential token generation, they are highly memory-bound and cannot fully exploit modern accelerators (Chen et al., 2023). In contrast, diffusion models, despite not having a concept of caching and requiring self-attention with fewer iterations than the sequence length and exhibit less memory- bouund behavior. Their per- formance can be further boosted with hardware-aware optimizations like flash-attention (Dao al.,
512
1024
2048
Generation Length (Tokens)
Figure 4: Single batch decoding speed (seconds) for different models using flash-attention 2.
the entire sequence at each iteration, can operate

---

Published as a conference paper at ICLR 2025
2022: Dao, 2024; Shah et al., 2024). In Figure 4, evaluate the decoding latency with batch size using flash-attention 2 and illustrate that our DiffuLLaMA achieves better inference efficiency using T = 256 when generating sequences of length 1024 or longer. This underscores the significant po- tential of diffusion models for efficient inference. Further decreasing T can lead to faster decoding but may sacrifice quality. Additional latency comparisons provided in Appendix C.5.
5 RELATED WORK
Continue Pre-training Continue pre-training is commonly used in adapting an existing language model (L.M to a domain-specific LM (Ke et al., 2023) or enabling new abilities of I.M, such as for longer context (Chen et al., 2024) code generation (Xu et al., 2024). Pre-training LMs is non- trivial and expensive (Samragh et al., 2024), thus in exploring of new architectures of LMs such as Mamba (Gu & Dao, 2023) and gated attention, Wang et al. (2024); Zhang et al. (2024c) choose to transfer from LMs to save the training cost. However, all these continue pre-training works follow the autoregressive (AR) language modeling, while adapting LMs into diffusion language model is more challenging due to discrepancies between their modeling objectives.
Text Diffusion Models Diffusion models have demonstrated significant diversity and controllabil- ity in image generation (Ho et al., 2020; Song et al., 2021a; Ramesh et al., 2022). Building on this success, line of research (Li et al., 2022; Gong et al., 2023b;a; Dieleman et al., 2022) build continuous diffusion models for text generation tasks. Among them, Lin et al. (2023) experiment with a pre-training and finetuning framework under a small scale; Gulrajani & Hashimoto (2023) highlight the scaling law of continuous diffusion models, revealing that the compute-optimal re- quires longer training than their AR counterparts. To address the discrete nature of text, Austin et al. (2021); Hoogeboom et al. (2021); Zheng et al. (2024a) incorporate an absorbing [MASK] state as noise, laying the foundation for discrete diffusion models, which are further developed by Lou et al. (2024); Shi al. (2024); Ou et al. (2024); Zhao al. (2024); Sahoo al. (2024). By connecting text diffusion models with pre-trained masked language models (MLMs; Devlin et al. 2019), Ye et al. (2023); He et al. (2023) initialize discrete diffusion models using MLMs. Besides, the unifi- cation between diffusion and AR generation is also discussed in image generation (Li et al., 2024). However, the adaptation of diffusion models from AR LLMs remains unexplored.
Non-autoregressive Generation Non- non-autoregressive (NAR) models, introduced by Gu et al. (2018), break free from the left-to-right generation constraint, allowing for new capabilities like planning with future tokens (Wu et al., 2024). Current diffusion language models are notable part of the NAR family (Gong et al., 2023b). Given the challenges of developing NAR models, researchers often seek to find a trade-off. For instance, SSD-LM (Han et al., 2023) leverages diffusion models to iteratively generate text blocks, facilitating a semi-NAR generation process. Similarly, CLLM (Kou et al., 2024) enhances LLMs by enabling the parallel generation of tokens, thereby improving decoding speed. FiLM (Shen et al., 2023) adapts language models to generate tokens in any order, which is particularly useful for infilling tasks. Guo et al. (2020) trains NAR using curriculum for the attention mask on translation tasks with seq2seq labels. Additionally, Gloeckle et al. (2024) focus on training models to achieve better and faster multi-token predictions as they scale up. These NAR approaches provide compelling alternatives to traditional AR LLMs, yet few have thoroughly explored training large NAR models on large-scale unlabeled data.
6 CONCLUSION
Building on existing DLMs, we present a recipe for building DLMs by continuing training on off- the-shelf autoregressive LLMs. Our adaptation technique involves using 1) attention mask annealing to enable bidirectional modeling and 2) shift operation to allow similar training dynamics like AR models. By unifying the language modeling objectives of autoregressive and diffusion models, we train diffusion models up to 7B parameters. Through experiments on common sense reasoning, lan- guage modeling, math reasoning and code generation, we show that DiffuGPT and DiA have better performance compared to existing DLMs. We find that DiffuLLaMA is capable of following in-context demonstrations to some extent math problems. In the future, we aim to in- struction tune our DLMs and explore inference time planning methods. We release DiffuLLaMA and DiffuGPT for further exploration of diffusion models as alternative language modeling method.
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
Shansan Gong: Project lead, methodology development, DiffuGPT training and model evaluation, major writing. Shivam Agarwal: Methodology exploration, discussion, DiffuLLaMA training, writ- ing. Yizhe Zhang: Discussion, DiffuLLaMA training. writing suggestions. Jiacheng Ye: Initial methodology exploration. Lin Zheng: Discussion, writing. Mukai Li & Chenxin An: Discussion, writing suggestions. Others: Mentorship and supervision.
ACKNOWLEDGMENTS
Research supported in part by US DARPA INCAS Program No. HR0011-21-C00 and BRIES Program No. HR001 -24-3-0325, National Science Foundation IIS-19-56151, the Molecule Maker Lab Institute: An AT Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I- GUIDE) by NSF under Award No. 2118329. This work used Delta AI University of Illinois Urbana- TabannaChaapaigee through allocation CIS230229, CIS240488 from the Advanced Cyberinfrastruce ture Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.
This research was supported in part by the joint research scheme of the National Natural Sci- ence Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number HKU714/21.
This work was also in part supported by research awards from Apple and the Allen Institute for AL.
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 1798 11981-17993, 2021.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In Forty-first International Conference on Machine Learning, ICML, 2024.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint 2tt.:2207.14255. 2022a.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022b.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-1,, 2020, virtual, 2020.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In 2022 IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11305-11315, 2022.
11

---

Published as a conference paper at ICLR 2025
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- -an--Bttiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Lon- gloRA: Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, 0bs/2110.14166 2021.
Tri Dao. F:ashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICIRR), 2024.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficiet exact attention with -wwarreessss In Advances in Neural Information Process- ing Systems, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proc. of NAACL-HLT, pp. 4171-4186. Association for Computational Linguistics, 2019.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond. Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffu- sion for categorical data. arXiv preprint arXiv:2211.15082 2022.
Kanishk Gandhi, Denise J Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (sos): Learning to search in language. First Conference on Language Modeling, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel de- coding of conditional masked language models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 61 2221,, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: N..11853/v/DDD91
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning, 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http::/99kt///oo0000..got OpenWebTextCorpuss 2019.
io/
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq-v2: Bridging discrete and continuous text spaces for accelerated Seq2Seq diffusion models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9868-9875. Association for Computational Linguistics, 2023a.
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR, 2023b.
Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00052,, 2023.
12

---

Published as a conference paper at ICLR 2025
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. Non- autoregressive neural machine translation. In Proc. of ICLR. OpenReview.net, 2018.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi.
Olmes: A standard for language model evaluations, 2024.
Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood-base diffusion language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. Fine-tuning by cur- riculum learning for non-autoregressive neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7839-7846, 2020.
Yiduo Guo, Jie Fu, Huishuai Zhang, Dongyan Zhao, and Yikang Shen. Efficient continual pre- training by mitigating the stability gap, 2024.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. SSD-LM: Semi-attoregressive simplex-bar -base diffusion language model for text generation and modular control. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pp. 1 159,,, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 2116655//1/222333 aac-llnn..67.
Zhengfu He, Tianxiang Sun, Qiong Tang. Kuanning Wang, Xuanjing Huang. and Xipeng Qiu. DiffusionBERT: Improving generative masked language models with diffusion models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 4521-4534, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v 000186553/112222..2 1/2013..c1212/o/00 ac4.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. Proceedings of the 36th International Con- ference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871085 10088.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neurul Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 12454-12465. 2021.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Rep- resentations, 2022.
Edward Hu* Moksh Jain*, Eric Elmoznino, Younesse Kaddar, Guillaume Lajoie, Yoshua Bengio, and Nikolay Malkin. Amortizing intractable inference in large language models. In International Conference on Learning Representations, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2025
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference Learning Representations. 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In The Eleventh International Conference on Learning Representa- lions, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Rik Concel-Kedziorste Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Ram- bow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 1152-1157. San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. (0.18653/v1/N16 1136.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. In International Conference on Machine Learning, ICML, 2024.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. Lower perplexity not always human-like. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers). pp. 5203-5217, Online, August 2021. Association for Computational Linguisties. doi: 00.18653/v1/2022 acl-llogg.455.
Raymond Li, Loubna Ben allal, Yangtian 7i, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia I.I, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro. Nicolas Gontier, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lee, Nadav Timor, Jennifer Ding, Claire S Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavit,, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. Reproducibility Certification.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.1 1838, 2024.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. In Conference on Neural Information Pro- cessing Systems, NeurIPS, 2022.
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion models for non-autoreeressivve text generation: survey. In Proceedings of the Shiony-Second International Joint Conference on Artificial Intelligence, IJCAI '23, 2023b. ISBN 9598919-6722--3-4
Chin-Yews Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
14

---

Published as a conference paper at ICLR 2025
Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of au- toregressive models and their alternatives. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 5147-5173. Association for Computational Linguistics, 2021.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: a pre-training approach with continuous paragraph denoise. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, org, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Reprexentations, 2019.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios the data distribution. In International Conference on Machine Learning, ICML, 2024.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, San Diego California, June 2016. Association for Computational Linguistics. doi: 10. 18158//11/11111100 1098.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. PMLR, 2021.
OpenAl. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023.
Myle Ott, Sergey Edunov, Alexei Bacvski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Walced Ammar, Annie Louis, and Nasrin Mostafazadeh (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) pp. 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ v1/N19-4009.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint 2rX.v:2400.033736 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.186533v1/P16-1111
Guilherme Penedo, Hynek Kydliček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEFE, 2020.
15

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zeoo-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 8821 PMLR, 18-24 Jul 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv preprint, 0os/2204.00125.. 2022.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint 2ii.:230..12955 2023,
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. URL https: /openeeveeeeeee net htt///ooommiaaaaaAAA--A
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, August 2021. ISSN 0001-0782. doi: 10.1 14534444347438..
Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. Scaling smart: Accelerating large language model pre-training with small model initialization. arXiv preprint arXiv:2409. 12903, 2024.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common-
sense reasoning about social interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4463-4473 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.19653/
Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. arXiv preprint arXiv:2407.08600 2024.
Tianxiao Shen, Hao Peng, Ruoqi Shen, Yao Fu, Zaid Harchaoui, and Yejin Choi. Film: Fill-in language models for any-order generation. arXiv preprint arXiv:2310.09900 2023.
Jiaxin Shi, Kehang Han, Zhe Wang, Amnad Doucet, and Michalis K Titsias. Simplified and gener- alized masked diffusion for discrete data. arXiv preprint arXiv:2406.00 04329, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 4rXiv:2408.03314. 2024.
Daria Soboleva, Faisal Al-Khate,b, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023.
Jascha Sohl-Dickstein. Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In Francis R. Bach and David M. Blei (eds.), Proc. of ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2256- 2265. JMg,. 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proc. of ICLR. OpenReview.net, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bor Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1 11895-11907 2019.
16

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, wtbp//302.1397..... 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 23988892. 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marein Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, et al. Amortizing intractable in- ference in diffusion models for vision, language, and control. arXiv preprint arXiv:4405.0097, 2024.
Junxiong Wang, Daniele Paliotta, Avner May. Alexander M Rush, and Tri Dao. The mamba in the Ilama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408. 15237, 2024.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Perey Liang. Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. thain-of-though prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances in Neural Information Processing Systems, volume 35, pp. 24824 24837. Curran Associates, Inc., 2022b.
Wilson Wu, John Xavier Morris, and Lionel Levine. Do language models plan ahead for future tokens? In First Conference on Language Modeling, 2024.
Zhiyong WU, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learn- ing: An information compression perspective for in-context example selection and ordering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 1423-1436, Toronto, Canada, July 2023. Association for Computational Linguistics. doi:
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. ArXiv, abs/2402.001622, 2024.
Yiheng Xu, Hongjin SU, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representations, 2024.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2024a.
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusson of thoughts: Chain of-thugt reasoning in diffusion language models. arXiv preprint arXiv:2402.07754. 2024b.
17

---

Published as a conference paper at ICLR 2025
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language models can perform many tasks with scaling and instruction finetuning. ArXiv preprint, ftos/2308.122111 2023.
instruttionfinetuningg
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024b.
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua M. Susskind, and Navdeep Jaitly. PLANNER: Generating diversified paragraph via latent language diffusion model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023,
Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang. Wei Bi, al. Gated slot attention for efficient linear-time sequence modeling. arXiv preprint arXiv:240.007446 2024c.
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discrete&ccomtinuus--tinn discrete denoising diffusion. arXiv preprint xxXiv:2402.03701, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In Conferenec on Language Modeling. COLM, October 7-9, 2024, Philadel- phia, PA, 2024a.
Lin Zheng, Jianbo Yuan, Zhi Zhang, Hongxia Yang, and Lingpeng Kong. Self-infillnng code gener- ation. In Forty-first International Conference on Machine Learning, 2024b.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299-2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18.58v/3/v 1/2024. windings-naaaee naacl. 149.
18

---

Published as a conference paper at ICLR 2025
A OBJECTIVE DERIVATIONS
This section provides detailed preliminary and loss derivations of $2 and $3.1 in the main paper.
A. BACKGROUND OF DIFFUSION MODELS
We denote X Pdtta (xo) as the variable following the data distribution, and x, g(x;) as the noisy variable of Xo at time i, where the maximum time is T. The forward process
(xxr Xo) = II ((x x
(9)
t=1
corrupts the initial data Xy into a sequence of increasingly noisy variables X1.7- Accordingly, the reverse Markov process models the joint probability
po (X po (xT)
II po (x)- 1=1
(10)
which gradually denoises X2 to reconstruct the original data Parameters @ are learned by min- imizing the negative log-likelihood of Xo. which can be optimized through the variational lower bound (VLB):
log po xo) S (xxx11x)) (x0) laolsiiilllooooooooo (x + (x)l(()))/))))lxxxxx))) + Cr.
(11)
with Lr
Σ
(12)
For continuous text diffusion (Li et al., 2022; Gong et al., 2023b), at each forward step, perturbations applied according to
[(x: -N(x; V
(13)
where 8, (0. represents different scales. In the end, Xy (). In the case of discrete denoising models (Ho et al., 2020; Austin et al., 2021; Zheng et al., 2024a), x, follows a categorical distribution which naturally aligns with discrete text data. Let be the one-hot encoded sample of variable x and x ~ Cat(ar;;)) represent a categorical distribution over vector x with probabilities given by p. Here, K represents the vocabulary size, (ei.-... ex and eg. E {0, 1) is the one-hot encoding of the k-th word category. The forward process can be formulated through transition matrix Q, E00, 1] [ such that
g(x,|x,-1) = Cat(a,; Q, ax:1): = (1-3))] 3.1e
(14)
with 1 as all-one vector of size K and we assume eк the special [mask] state, also defined as the absorbing state in discrete diffusion m. Each entry in [Qrlij denotes the probability of transition from the state e; ej. and thus the previously defined Q: means with probability B,, æ, will stay unchanged and otherwise it will jump to the mask state ex.
Starting from Fg. the t-step marginal distribution and the posterior previous time is respec- tively
q(x.(a0) = Cat(x;;; Q, zo); g( [2,, zo) q(3;, [1 wo)g(w; (x, (o)
(15)
(x, (o)
where cumulative products Q, - IT Q azl + (1 ai 1m and II' (1 Be). We expect or approaches such that the full noise data Ør equal ex with probability 1. In the following sections, we primarily takes the discrete diffusion formulation.
A.2 LOSS DERIVATION
Previous discrete time E [0, T] restricts x, to fixed time points whereas the continuous time sam- pling allows for more flexibility covering any point in the range (Kingma et al., 2021; 1; Shi et al.,
19

---

Published as a conference paper at ICLR 2025
2024: Zhao et al., 2024; Ou et al., 2024). In this case, t runs from 0 to 1, corresponding to dividing [0, into T intervals and let T - oc. For any two arbitrary time points, I s < t I 1, the for- ward modeling can be generalized from (xxx to (x.) We uniformly adopt the notation of continuous-time in following sections.
Following the previous definition, after simplification, we have q (2.a 0g30 0z m, refer- ring to the probability of transition to absorbing mask state. Given g( (2)) = g( xz, (2x 20), can derive the transition distribution between two arbitrary times s and t:
q( x..) Cat Qo) with Quit Q2 Q
I + (1
11m T.
(16)
02
Similarly, after simplification,
02 Ds + (1 02
Ofg as
m.
(17)
Following Zheng et al. (2024a); Shi et al. (2024) and extend the formulation to continuous time, we have the backward transition probability:
g( a00) (((xx((ss)u((r,) g(2;; (20)
1-(1-a))
1-a, 1-a1
=
a, a0 1-01
Dt = I2 mm, (18)
1 1-
oz ).a
if xr m f x...
I-0:
I-ae
For æt j m. the g( (2 v.) will stick the observed data. For 2t m, we get the simplified
0t
- 02s
ga,, De. 2o) 1 Oz+
+
Ox m.
(19)
In diffusion process, the generative model aims to approximate the reverse transitions using de- noising model po(0, fe(z)) ~ q(0,/=,, Øo). where fo (æ,) represents the probability vector obtained from the softmax applied the logits generated by the neural network, usually using trans- former networks (Vaswani et al., 2017) in text domain. We can similarly have
0g Oit dkg
fo( + O's m. Ckg
(2 Po (0 [x,)
(20)
19 and Eq.20, the KL-divergence loss is optimized by Given 19
a. I-ac 0,
(xu(zullfo(ze)) for H = m: for 2t # m.
(q xe. xo) lpe ( ax [<=[(==)))
(21)
We can use the indicator function om unify the conditional cases. In addition, given Vo, we have DKL (xx)) = log (ax) which corresponds to the cross-entropy widely used in the classification. Therefore, we have
DKL Iz, zo)) (x.11=<() = Gg at Se: a
L. log fe(x))
(22)
- at
Following Eq. 12, if we set a small timestep A, 1 = = E (0.1),
CT Σl
ag - at
Se. my log fo)
(23)
(1 - 5) - at)
By taking the limit as T - 00, we have af = o and the sum is transformed into an integral:
1 of ar
lim LT T
E 1 16 ay log fo (ax)) dt.
(24)
Also, the first two terms in Eq.11 evidence lower bound (ELBO) of - log po (Xn) as Eq.24.
+ and constant, respectively. Thus we can formulate the
The same form of ELBO which is invariant noise schedule but related to the signal-to-noise ratio (SNR) is also introduced in Kingma et al. (2021); Shi et al. (2024). Following Austin et al. (2021);
-a! Zheng al. (2024a), we choose the noise schedule a, 1 L, then T-c:
20

---

Published as a conference paper at ICLR 2025
The previous discussion focused on the single token Dt, and can be easily extended to text sequence The final loss of the whole sequence is
of length N represented as X = [x] 2
xt
N
1 CN = BBuycsslau (sea)
8xp mm (x0)Tlog log fx(x )n
(25)
where [o( )n denotes the whole input sequence fed into the transformer model and the n-th output token is indexed. During training, we sample / for each data point to optimize the expectation in C!:N instead the integral Cr, while for evaluation, we use integral Cr.
B IMPLEMENTATION DETAILS
B.1 TRAINING DATA
DiffuGPT Previous diffusion language models such as Plaid IB (Gulrajani & Hashimoto, 2023),
SEDD (Lou et al., 2024) and MD4 (Shi et al., 2024) use OpenWetTeext WebText (Gokaslan & Cohen, 2019) to
pre-train from scratch, referring to GPT2 (Radford et al., 2019). We choose the advanced Wibe corpus (Penedo al., 2024), which also derived from Common Crawl. We randomly sample 30 billion tokens from subset sample 1anee-100 1000B.
Laff Following Zhang et al. (2024b)* we construct the training data for Daff by mixing SlimPajama (Soboleva et al., 2023) and Starcoder data (Li et al., 2023a). We randomly sample 65 billion tokens in the ratio of 7:3 from SlimPajama and Starcoder, respectively. We use sequence packing and pre-tokenize the dataset for efficient computing.
Data Selection Consideration Our dataset selection aims to align with their respective pre- training objectives. Since we train a relatively smaller number of tokens and do not intend to introduce new capabilities, we prioritize maintaining continuity with the models' pre-training dis- tributions to minimize distributional shift. For GPT2-based DiffuGPT, we the FineWeb dataset, which closely resembles WebText (the dataset used in GPT2's pre-training). Considering that SEDD (Lou et al., 2024) iterates Open WebText for more than epoch and the total training amount is around 200B tokens, we also iteratively train DiffuGPT on 30B FineWeb data to more than 100B training tokens. In contrast, LLaMA2 is pre-trained over 1T tokens within one epoch on web and code data. To aligm with this, we follow TinyLLaMA (Zhang et al., 2024a) and use a mixture of SlimPajama and Starcoder data, designed to reflect LLaMA2's pre-training data.
B.2 MODEL OPTIMIZATION AND HYPERPARAMETERS
DiffuGPT We implement DiffuGPT using LLaMA-Factors with DeepSpeed Zero-2 paralleliza- tion (Rajbhandari et al., 2020). The hyperparameter setting compared with previous work is listed in Table 4. The global batch size is calculated by multiplying the single GPU batch size, the number of gradient accumulation steps, and the number of GPUs, where we use 8 A100 S0G. We use learning rate of 3e 4 with cosine scheduler. The warm up steps set to 2K and attention mask annealing steps are 10K. As shown in Table 4, our effective training tokens are less or equal than SEDD and MD4, while DiffuGPT exhibits better performance according Table and Figure 3.
DiffuLLaMA For a more efficient pre-training, we implement DiffuLLaMA using huggingface We use DeepSpeed Zero-3 parallelization with CPU offloading (Rajbhandari al., 2020) to effi- ciently scale DiffuLLaMA to multiple GPUs and nodes. Furthermore, we use flash-attention 2 and fused cross-entropy loss for optimized GPU memory usage and compute time (Dao, 2024). With these settings, we get set batch size of 60 per GPU with context length 2048 on GH200 96GB GPU. We use AdamW (Loshchilov Hutter, 2019) to optimize our models with a constant learning rate of 2e - 5 and accumulate gradients every 4 steps. We train our model for 65 billion tokens on 16 4xGH200 nodes.
htttrpa///annnnnsteecco////tttte Fineweb 'nttps: Shttps://gt thub.
21

---

Published as a conference paper at ICLR 2025
Table 4: Training settings for different diffusion language models.
Models
Training steps
Global batch size
Context length
SEDD (Lou et al., 2024)
400k
512
1024
MD44 (Shi et al., 2024)
1000k
512
1024
DiffuGPT-S
1000k
256
512
DiffuGPT-M
160k
1280
1024
Tokenizer During adaptation, we do not change the tokenizer of the base model. In theory, we should expand the original vocabulary by adding additional dimension to include a special token as [MASK] token. However, considering practical issues on implementation, we can alternatively select existing word from the vocabulary to serve the [MASK] token. It is preferable that this chosen word has a particularly low frequency of occurrence in corpus. For DiffuGPT-S we use tckenid=1 054 and for DiffuGPT-M we set new [MASK] token with tokenid=50255 For aaM, we set token cokenid=8111 11
B.3 EVALUATION DETAILS
Generation tasks For the TriviaQA and Lambada sentence completion tasks, we generate n- tokens for continue-writing In triviaQA, we set to the oracle length plus an additional 10 to- kens, and we only evaluate the first 2000 cases in this dataset for efficiency. For Lambada, which requires the completion of the last word, we set to oracle length of that word's tokens, which might be larger than based on the tokenizer. For DLMs, we set the diffusion timesteps T to the required generation length. For AR baselines, cut off maximum MAWPS, we integrate our model into weath-evallate ion- harness".
tokens. For SATMATH and
CommonSense Reasoning tasks questions with 4 options. Instead of open generation, we calculate the diffusion loss for each Questi iesssiinnchoico pair using 25. lower loss (perplexity) indicates the model thinks that choice most suitable. This approach is commonly employed in ICL of LLMs (Wu et al., 2023). We also use this for AR baselines.
The 4 commonSense reasoning tasks are multiple-choices
Finetune GSM8K-symbolic The setting of finetune GSM8K-symbolic dataset is following Ye et al. (2024b)', which enables the diffusion model to perform chain-of-thought reasoning. For Dif- LoRA Tuning (Hu et al., 2022). We set rank
fuLLaMA, we use saaunete--fffcciet--neeeeee and enable the finetuning of the word embedding layer, with 151 million (2%) parameters involved. For this task, we use T 64 for the decoding of DLMs.
Infilling tasks For ROCstories, where each case is 5-sentence story, we setup evaluation refer- ring Shen et al. (2023).The model is tasked with infilling the third sentence based on the first two and last two sentences. We evaluate the first 1000 cases in this dataset for efficiency. For code infilling, we use humaneval-single-ine infilling and their evaluation toolkit, which contains 1033 test cases. We implement infilling tasks for AR models by feeding the prefix and cutting off the generation length using the oracle length, considering that these AR models are not supporting in- filling. We also try to feed the suffix information using the instruction like Given prefix and suffix p please infil 11 the mi middle e, however, I.LMM22 can not follow this instruction. For AR LLMs, to perform infilling tasks requires additional FIM training (Roziere et al., 2023) or carefully instruction tuning.
Unconditional Generation For unconditional generation in Figure 3, we set the temperature of top-k to 0.98 and top-p to 0.9 for the medium-sized model, while using top-k of 1.0 and top-p of 0.9 for the small model. We generate 64 samples and evaluate the perplexity using the GPT-2 large model. aligning with Lou et al. (2024); Shi et al. (2024).
httt://///tttooo.oo//////n////t/ "tttps: portps://ot // thub. wttp:b/cccoom//..c.c///oo of "https: // thub. whtp.ccomm/peeaa//aaaneeeeeeeeee
22

---

Published as a conference paper at ICLR 2025
C
ADDITIONAL RESULTS
C.I UNCONDITIONAL GENERATION
The generation quality is different for different hyperparameters, shown in Figure Lowering the temperature increases fluency but reduces diversity, leading to noticeable repetition in sentences.
203
1.0
Generatve Pepliatty
200-
Durlank
T129 dist2
T1024 ppl
T1024 dina
Pefesl vappleg
topp
tappi Stppnn.
Stopl
topll Appel.pppp0 Aoopl
02
Figure 5: The unconditional generation quality for different diffusion time steps and sampling algorithms. We annotate the temperature of top-k sampling and top-p sampling.
We randomly selected samples generated by our DiffuGPT-M models with 1024 tokens for various T. as shown in Table 12, Table 13, and Table 14, Lower T values result in less fluent text.
C.2 ABLATION ON GSM8K-SYMBOII SYMBOLIC
Using the same discrete diffusion loss, if we direct finetune on GPT2 achieves accuracy of 45.4 and 49.7 for small and medium models, respectively. In contrast, Finetuning from DiffuGPT yields accuracy of 50.2 and 61.8 (Table 1). Comparing with GPT2, DiffuGPT, as the base model, converges faster and attains lower loss, shown in Figure 6. This indicates that better base model leads to improved results and also demonstrates the superiority of DiffuGPT as the current best diffusion base model.
Training with the initial weightings of GPT2, we evaluate three loss functions: DD, DD (no shift), and DD (no annealing) when finetuning on GSM8K-symbon data. The corresponding loss and accuracy are shown in Table 5. Additionally, results for DiffuGPT and DiffuLLaMa are presented. All results highlight the negative correlation between the loss and accuracy.
Table 5: The training loss (ELBO) and test accuracy the GSM8K-symbole dataset.
Models
Loss (ELBO)
Acc
GPT2-M DD
0.015
49.7
GPT2-M + DD (no shift)
0.028
34.5
GPT2-M DD (no anneal)
0.019
47.2
DiffuGPT
0.009
61.8
DiffuLLaMA
0.003
63.1
C.3 ADVANTAGES OF DLMS
To explore the self-correction advantages of DL.Ms noted by Ye et (2024b), we perform quali- tative analysis and find similar self-correction capability in DiffuGPT. Our observation of the final steps of sampling trajectories, as shown in Table 6, indicates that DLMs refine intermediate numbers without adhering left-to-right constraint.
For global planning, we follow Ye et al. (2024a) to finetune DLMs on counting down (CD) datasets. CD is mathematical reasoning challenge and a generalized version of the game 24, which many AR models struggle with (Gandhi et al., 2024). We compare DiffuGPT with other AR basclines with different model sizes in Table 7, demonstrating the advantages of DL.Ms
23

---

Published as a conference paper at ICLR 2025
Small sized model
Medium sized model
1.0
GPT2
GPT2
0.8
DiffuCPT
DiffuGPT
0.6
Training
0.4
0.2
0.0
50000
10000
20000
Training Steps
Training Steps
Figure 6: Finetune GSM8K data with discrete diffusion objectives, using a base model either GPT2-S/M or DiffuGPT-S/M. DiffuGPT converges faster and attains lower loss.
Table 6: Case study to show self-correction capacity of DiffuGPT. I/T refers to the current decoding step over the total diffusion steps. The incorrect rationales marked in red.
Steps (t/T)
DoT rationales
9/32
<<3<15=45>> <<4.45-180>>
<<10000000>> Fi 00
8/32
<<3+15=45>> <<l.:5+180>>><1111100001 0000000000000 #### #### 000
7/32
<<3+15=45>> <<d<45=80>>> <<180+400=400>> #### 480
6/32
<<3x15-15>> <<<<45=100>> <<180+300=500>>> #### 580
5/32
<<3x15=45>> <<4x45=190>> <<444==10>>><<11001>11 <<180+300=580>>>
#### 580
4/32
<<3+15=45>> <<q=45=180>> <<180+300=480>>
#### 580
3/32
<<3+15=45>> <<4<4>=180>> <<180+300=480>>
#### 580
2/32
<<3+15-45>>
<c4.45-180>>
<<180+300-480>>
444# 480
1/32
<<3x15-45>>
<<d+45-180>>
<<1804300-480>>
444 480
For infilling tasks, we attempt to query the LLaMA model with the prompt given the <prefix> and <suffix>, please answer the <middle> part, which includes both prefix and suffix information. However, this approach is no better than simply completing the prefix, likely because the LLaMA model needs tuning for filling in the middle (FIM; Bavarian et al. 2022b). Additionally, Bavarian et al. (2022b) notes that using AR models for infilling presents challenges, such prompting difficulties and repetition. In contrast, DLMs are naturally suited for this task, as they are trained to handle masked inputs, which a key advantage.
Additionally, conduct controlled experiment by training both AR and DLMs 100M tokens from the Starcoder dataset, using CodeLLaMa as the base model and evaluating performance on HumanEval infilling- We finetune Codel LaMA autoregressively with FIM in both suffix-prefix- middle (SPM) and prefix-suffix-mddde (PSM) formats. Our results in Table show that Diffu- CodeLLaMA outperforms PSM, suggesting that prompt format affects AR models but not DLMs. We believe that training on more than 100M tokens, which is relatively small, could enhance perfor- mance.
Table The finetuning results (accuracy) on the CD4 dataset.
Models
Size
CD4
GPT2-scratch
85M
45.8
LLaMA FT
13B
51.1
SoS (Gandhi et al., 2024)
250M
54.2
DiffuGPT
355M
87.5
24

---

Published as a conference paper at ICLR 2025
Table 8: Models finetuned on 100M tokens of Starcoder and their results on HumanEval Infilling.
Models
Pass@1 HumanBval Infilling
CadALLaMA FT (FIM-SPM)
0.80
CodeLLaMA FT (FIM-PSM)
0.74
(Ours)
0.76
C.4 CONTINUAL PRE-TRAINING AR MODELS
We conduct continual pre-training of GPT2 on the same corpus under the same settings as DiffuGPT. However, the zero-shot performance, shown in Table 9, indicates no improvement. This may due to the stability gap introduced by continual pre-training (Guo et al., 2024), leading to performance degradation. Additionally, since our used corpus is similar to the one used for GPT2's initial pre-training, continual pre-training may offer limited new knowledge.
Table 9: Performance of different models on various tasks.
Models
HSwag
Wino
SIQA
PIQA
Code
GPT2-M
38.3
50.7
37.7
67.4
2.6
GPT2-M (continue pretrain)
36.7
49.4
37.9
66.7
2.6
DiffuGPT-M
37.2
52.6
39.0
59.6
2.9
C.5 DECODING SPEED TESTING
We evaluate the inference time of LLaMA2 and DiffulLaMA for unconditional text generation across various lengths. Our tests include vanilla attention, flash attention 2, and the torch version of flash attention SDPA, as shown in Table 10.
Table 10: Single batch inference time for different attention implementation and generation lengths.
Length
Attention
DiffuLLaMA (sec)
LLaMA (sec)
512
flash-attention 2
12.5
9.2
1024
SDPA
13.2
16.3
1024
flss-attennion 2
13.3
17.5
1024
vanilla
16.2
17.2
2048
SDPA
28.5
29.5
2048
flash-attention 2
23.5
35.7
2048
vanilla
38.1
32.8
Yet smaller T leads to faster generation, in downstream tasks like multiple-choices, this may slightly impact accuracy. Examples are provided in Table 11.
25

---

Published as a conference paper at ICLR 2025
Table 11: Performance of DiffulLaMM models on various tasks.
Models
HSwag
Wino
SIQA
PIQA
DiffuLLaMA T=32
58.7
56.4
43.2
63.3
DiffuLLaMA T-8
47.1
52.6
41.9
57.1
Table 12: Generation examples of DiffuGPT-M (7 = 10024.
If you'l 're considering applying the JBCC school, I'm confident you will find something there!
What exactly do the schools have required?
HAVE DREAMED (I know it's not in my DNA). In fact, I have my life time.
What do you need anyway?
.MA or PhD degree in non-religious education and/or MA or PhD degree.
Our Schools ensure that you have a strong academic background in the non-religious field of your choice and a passion for teaching in literature, research, writing, reading, English as a second language, or teaching.
The concentration may help provide world-class emphasis offer unique opportunities for teaching of new skills or in dynamic contexts.
Practical Program coursework enables students to study in specific fields and areas in their career, this may an essential part of education you work as a tutor or your goal will include working an office, high school social science teacher, classroom. What makes JBCC unique?
The National International Baccala Practical Programs are located locally and offer JBCC's scope and the preparation to complete an advanced master's degree.
How do students go to graduate from JBCC?
Get in now!
Congratulations for applying for the JBCC program. Become a part of this community. What does JBCC achieve?
JBCC prepares leaders to be a catalyst for an educational and emotional enriching environ- ment for service. Leaders are within their capacity to call others service. Want to know more?
To find additional information, please use this form. You can reach us to find your application materials for JBCC here.
Christine Callender
Educator and Director
School of Christian Education Hello there everyone! Welcome to our tutorials section. We have everything you need know what and how make quality t-shirts apparel especially for those who are new to t-shirts. If you not sure, they are very popular item.
As you might imagine, there are wider audience of people than others.
- T- shirt how to make yourself?
T-shirt to wear?
What can I use to make my shirt?
4. Do need reuse my tee shirt?
There is no need to reuse your tee shirt. This means you can spend few minutes putting it on another pair of clothing. Polyester is great for quick clean, allowing you to get the stains on the front of your tee shirt, without damaging them. Since polyester is very breathable, which means able to be used both your skin and anything else
26

---

Published as a conference paper at ICLR 2025
Table 13: Generation examples of DiffuGPT-M (T' 256).
Use of the Service: Cookies may enable your internet site or your computer or device to access information using our Services, so as to allow us to work together to provide the website that you use. Cookies may provide Personal Information in certain cases.
Google Digital Advertising Cookies
Advertising partners may use internet analytics with information on your websites visit, how far you come, through our different advertising networks order to tailor them you. This may provide your personal information to our advertising partners and may be shared with third-party advertisers.
Google may also use tracking cookies that analyse visitor preferences, such as the type of device used, visits to pages the user has access to most frequently, or how frequently visitors visit particular pages, analyse how people find what sort of information most interest to them. They can keep track of all such visits as they collect more information, using cookies and analytics improve the content of website.
For more information on how to control Google's cookies on our websites, see https://geo.com// cowm///sssss//sssseeeeeeeseseees learn moreLogan has been very popular among travelers in the area since 1845. The early explorers had place and that's the best place to stay after a long jour- ney. Fast-forward the years, and the ones that stayed in Logan migrated before even crossing over into the mountains to the north. These changes have made from a midwestern city to a postmodern city of sorts where everyone lives.
This is what global locations mean. It's easy enough to get lost after day spent traveling, but every city offers different experiences and fun when it comes to being on the road.
The most important parts of Logan experience are the historic districts of Logan Square. Discover the history of so many different places and gather with people who enjoy exploring something new, even if with children pets in Discover the Broadway complex of Logan Square and the rest of the neighborhood, as it's a charm sprinkled with charm.
Sitting the North of Telegraph Hill one mile from Loomers you will find the Logan mountain trail. The 18-mile-long trail is one of the finest in the state and enjoy a day on your bike or biking, surrounded by all sorts of cafes, unique eateries and galleries. Lawsony Park has the Home Park Beer Cellar and the Home Park Bike Shop, and great gift shop for that. Explore the neighborhood on Memorial weekend and warm up with delicious bite from Logan Friendly Brews.
week on Memorial Weekend is a new tradition for the area of Loomers. Here's the excite- ment of heading the road on one of the weekends of the year!
The summer at Logan is one of the most popular times of year to explore the quirky neigh- borhoods. Our parks are vibrant and busy. We are downtown and restaurant and bars stay open to have a drink and coffee. Stop in at the rooftop patio to enjoy a really nice evening with quick bite. And every week Memorial Weekend, you'll 'll find really nice indoor pool!
For kids with weekend, Logan has a playground and good for two and on August and is the Logan Fun Day. A few spots are available (and we're re always open to both men and women) and your group allows you just catch up and season with friends while enjoying pint of beer.
John Moody Brews is about bring their Labor Day fun to the streets and your backyard on Labor Day Weekend. Logan Friendly Brews has brewery cruise, street vendors and live music on Thursday night and they'll II have giveaways on Sunday. It's a fun
27

---

Published as a conference paper at ICLR 2025
Table 14: Generation examples of DiffuGPT-M in 1024 tokens (T 32).
t work in near-zero conditions.
In the meantime, employees are planning to avoid travel, and lack of traveling. The United States will not pay or pay for travel, Grand Tech said. But the company plans to continue to take its staff to Asia and other destinations. The plans for the next trip will vary. The Philippine office is investigating the outbreak and is considering further expanding to more but still non-fue areas. There may also be more traveling than they had previously been doing before.
The company reports, confirmed first by ETOY's Sulayo Suan, denied to local media. Grand Tech has its headquarters in Cincinnati, Ohio. Delta variant virus caused a plant disaster nearly two months ago. The the genotype virus of the virus hit an employee while on job there in April.
The woman reported that she needed to remain home confinement until August. But suc- cumbed continued serious illness to test results for weeks.
The Delta variant is currently targeting companies in the European Union, with the virus occurring in Italy. French workers have also been reported suffering from the outbreak. Employees have been traveling to work due to some journeys to Asia and respiratory com- plications to travel Asia.
The company also reported significant risks to their herbal products containing Chinese plant material, including some hogweed. Hogweed alone responsible for the loss of 1 U.S stem to each year, Grand Tech said.
The U.S. is offering an ongoing vaccination program available to 72 million all Americans, including those most at risk from the virus, State Street announced. Globular epilepsy is genetic condition that is at risk.. ORIK can occur as high as 15% of people living in Indonesia. However, can discuss information that exists that ORIK. There no factors that could McanESK 2 is class II agent with teratogenic activity determined by the MSL. We propose not treatment of ORIK on patients that exceed the 2.0 threshold. Both biological factors predispose risk of the. X at least[SMRC IX. Non-Mouse type 8a(112).
Globbull
About both factors that promote the development of ORIK.
To specify reasonable regulation that would affect p.ss. The United States Juvenile Court as an Authority for Administration of the Bureau of Corrections by Arthur Whyte, Jr. Chair: John Bronz Board: Buck Morgan Jr. Rep: John Little Reader: David Gervis. THURY Fisher, WOOD and her son.
DATE: March 23, 1959
REV: July 3, 1949
By resolution which is passed: Either person uses nothing more than a boat within a warehouse, apartment or barn. Department of the Interior Department of Labor system of it existed as a whole under the laws of Michigan three. through December one of such would not. Each $100 person shall be fined and the division shall collect all damages of each two hundred dollars and dollars' the total of such amount and enter a. Rights reserved: also trustee thereof may order the delinquent bonds. If person convicted if that has committed any act of attachment. Assisting duties shall deemed to have ceased within fifteen days, then the Commissioner to be appointed shall pay the district the sum of four percent of the value of the money recovered from these costs. Pursuant in this section shall be revoked without notice. Revocation: also the equivalent instructions accompanied with. Fifteen days of the entry of such order. The blight place known to be in a person has an occupancy.k enables.
He can throw in about picture. How he loves it so much he is the greater these online webit himself right from scuba dive and wishes to be removed by the government of America, that's a much sleazier story. From time social circles, the members of MoolahFast, dating site focusing on the planet of different scuba divers.
28


Reviews:
[{"review_id": "qERg4nZ8LJ", "reviewer": "Reviewer_XhLb", "summary": "The paper studies diffusion language models (DLMs) and how they could be a good alternative to autoregressive (AR) LMs for generating text. It highlights the lack of fair comparisons on language modeling benchmarks and the challenges of training DLMs at scale. The authors propose adapting open-source AR models to build text diffusion models. By demonstrating the connections between AR and diffusion modeling objectives, they introduce a simple continual pre-training approach for training these models. The study shows that popular AR LMs can be converted into diffusion models (DiffuGPT and DiffuLLaMA) using less than 200B tokens for training, outperforming earlier DLMs and being competitive with their AR counterparts. The paper contributes a suite of DLMs capable of generating fluent text, performing in-context learning, text in-filling and following instructions.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a simple and effective approach to adapting AR models to build DLMs, bridging the gap between the two modeling paradigms, while the proposed continual discrete diffusion pre-training approach for training diffusion models is practical and effective.\nThe proposed models, DiffuGPT and DiffuLLaMA, show competitive performance with AR models and outperform earlier DLMs in generating fluent text, perform in-context learning, filling in the middle without prompt re-ordering, and following instructions. \nThe writing, organization and presentation of the paper are also good; I enjoyed reading the manuscript.", "weaknesses": "Overall, I appreciate the achievement of well-performing (discrete) diffusion LMs at scale, but I do have some concerns regarding novelty and evaluation. \n\n**About building DLMs by scaling and adapting from pre-trained large-scale LMs:**\nBuilding (discrete) diffusion LMs from pre-trained Masked LMs has been studied in [1], while scaling and instruct-tuning (discrete) diffusion by adapting large-scale pre-trained LMs (mainly Masked-LMs while they also attempted to use AR-LMs/Llama1 to test reasoning) has also been studied in [2]. While this paper makes a solid step forward and effectively improves the performance of DLMs by leveraging AR LLMs, the authors should have thoroughly discussed the connections and major differences of their proposed approach with these prior works in the introduction or at least method/related work sections. Can the authors elorabrate the substantial differences and new contributions introduced by this paper?\n\n**About progressively adapting AR text generation models into non-AR models:**\nTwo core techniques introduced in this paper are \"Attention Mask Annealing\" and \"Shift Operation\" to effectively reprogram an AR LM into a DLM. However, a similar approach has also been proposed in [3] in the case of non-autoregressive machine translation, which should have been discussed in this paper.\n\n**Some claims are not supported by evaluations, hence the true advantages of DLMs are not convincing:**\nThe paper introduces the motivation of pushing the limits of DLMs by arguing that \n> Notable challenges include difficulties in **future planning** (Bachmann & Nagarajan, 2024; Hu* et al., 2024; Xie et al., 2024) and **self-correction** (Huang et al., 2024). These constraints have spurred researchers to explore alternative architectures for next-generation LLMs.\n\nand accordingly,\n> Notably, DLMs exhibit promising capabilities in **intermediate token correction** (Ye et al., 2024) and **global planning** (Zhang et al., 2023), thereby addressing key limitations inherent in the AR approach.\n\nHowever, there is no clear evidence in this paper, not even qualitative ones or case studies, to showcase that DiffuGPT and DiffuLLaMA possess such promising capabilities of self-correction and global planning. Only evaluating on language understanding benchmarks with weaker performance compared to AR-LLMs (Table 1) cannot convince the community why we need really DLMs as an alternative to AR LMs and as a new LLM paradigm (I think this is the most important question every DLM paper needs to address). Plus, the in-filling performance of AR-LMs was unfairly assessed as they are only provided with prefixes, according to Table 1's caption. Thus, the paper still fails to demonstrate the unique, distinct advantages of DLMs over AR-LMs. I strongly suggest the authors provide more evidence to differentiate DLMs' strengths in these regards, which can also support the motivations of this work in the introduction.\n\nOverall, I think this manuscript stands a good contributions to DLM research, and I believe it could be greatly improved if authors can address the aforementioned concerns. \n\n----\n\n[1] DiffusionBERT: Improving generative masked language models with diffusion models. ACL 2023\n\n[2]  Diffusion language models can perform many tasks with scaling and instruction-finetuning. Arxiv 2023\n\n[3] Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation. AAAI 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnXWo7ByZ8", "reviewer": "Reviewer_ZRNS", "summary": "The paper introduces a methodology for adapting pretrained autoregressive (AR) language models to discrete diffusion (DD) models. It identifies two main points of conflict between them: 1) Causal attention (AR) vs. bidirectional attention (DD), 2) next-token-prediciton (AR) vs. denoising (DD). In order to resolve 1), the paper proposes attention mask annealing, where the causal attention mask is slowly turned into a full attention mask over training. For 2), the DD loss is adapted by shifting the logits by 1 to be in line with AR models.\nIn the experimental evaluation, they continue to finetune pretrained AR models of various sizes using the derived adaptation scheme. The models are evaluated on various knowledge, reasoning, coding, and math benchmarks.", "questions": "* From the paper, it is unclear to what extent the continuous-time discrete diffusion processes described in 3.1 are a novel contribution. If this is not novel, I would move it to section 2 for clarity about your contributions.\n\n* In diffusion processes, do we need to know the number of tokens ahead of time in order to know the number of MASK tokens? If not, how is decoding done?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "* Adapting AR models to diffusion models is a problem that is of interest to the ICLR community.\n* The proposed method is intuitive and simple.\n* The paper is well written and flows really well.\n* The results demonstrate that the adaptation is successful to some extent: Adapting larger LLMs leads to better performance.\n* The DD model shows superior performance on some tasks that mostly require infilling.", "weaknesses": "1. In lines 364-368, it states that \"by observing different scales of our adapted diffusion models, we can draw the conclusion that scaling diffusion language models results in improved performance\". This statement suggests that if a DD model was trained from scratch, performance would scale with the size of the model. However, this isn't supported by the experiments, which merely shows that AR models of increasing size can be adapted to DD models in such a way, that larger adapted models perform better than smaller adapted models. However, it is possible that the adaptation process is merely retaining some of the knowledge/skills that were learned by the pretrained AR model. It doesn't mean that the DD model could attain these capabilities itself through scaling. I think it is important to be precise here.\n\n2. In lines 367-369: \"This discrepancy [between DiffuGPT's performance compared to the base model vs. DiffuLLaMA's performance compared to the base model] is presumably attributed to the extensive amount of training tokens processed by the DiffuGPT series models, whereas the 7B model has not undergone equivalent training.\" This explanation may be true, but needs to be verified. Since DiffuLLaMA has been pretrained on more than twice as many tokens (65B vs 30B), it doesn't seem obvious. What is the reason not to finetune both models on the same data?\n\n3. In lines 373-375: \"In tasks that require more extensive global reasoning, such as complex mathematics and coding, DLMs consistently exhibit better performance compared to AR models that rely solely on left-to-right modeling capabilities.\" This claim can only be substantiated through a controlled experiment that continues to finetune the base model on the same data in AR mode vs DD mode. Table 3 conducts this experiment, but only on a single downstream dataset that is quite small. Having a fair, controlled comparison between AR and DD models is the biggest weakness of this paper.\n\n4. Table 2: Different models here correspond to different prompting methods. hit@k is not a prompting method, it is an evaluation metric whose value is at least as high as the standard evaluation metric with hit@1. Stylizing the respective row with hit@k incorrectly suggests that this is the best \"method\", which could mislead the reader into thinking that the difference to the LLaMA-2 baseline is smaller than it actually is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCxSqZsPJz", "reviewer": "Reviewer_rZH1", "summary": "This paper proposes to adapt pre-trained autoregressive models to build discrete diffusion models via continual pre-training. This way, they can convert AR to diffusion parameters by training on fewer than 200B tokens. Technically, they propose  (1) annealed attention masking, (2) inheriting the shift inductive bias from AR. The experiment section suggests that this diffusion model demonstrates strong performance in math, word lambada, and infilling, and ablation suggests that the shift inductive bias is crucial for the transfer.", "questions": "see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes to adapt AR pretraining parameters to Diffusion parameters, which solves a very important problem about improving the training efficiency of diffusion models. \n2. The result seems solid, with a fair amount of improvement over previous diffusion models. But I still think there are some unresolved problems as shown in the weakness section. \n3. The experiment seems pretty comprehensive, with a wide range of downstream task, generation quality, analysis, ablation, and inference efficiency.", "weaknesses": "Since the paper claimed to tackle the parameter transferability problem, there remain some unresolved scientific questions: \n(1) suppose we have more compute, is this transfer still optimal? would this transfer converge to a worse local optimum than if we train from scratch for longer? \n(2) The paper included ablation results on GSM8K performance, but more fundamentally, what's the loss in terms of PPL for the (without shift) and (without annealing) baselines? I think these are more important and more general-purpose to report than a specific downstream task. I saw the disclaimer that PPL may not directly reflect the correctness and it's only ELBO. I still think it's good to report ELBO on the test set in the paper.   \n(3) What's the number of flops for the training? 200B tokens is a bit abstract, especially 200B token for diffusion is different from 200B token for AR (due to lack of KV caching) and 200B token means different Flops across model sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "agi4TiVXkv", "reviewer": "Reviewer_87p8", "summary": "The paper addresses limitations in autoregressive (AR) models, particularly issues with global planning and intermediate token correction. In response, Diffusion Language Models (DLMs) were introduced but face scalability challenges due to the computational costs of training with only 400 billion tokens. This work proposes a novel training mechanism that redefines DLM objectives. The approach also addresses architectural limitations in AR models, using attention mask annealing and a shift operation to remove causal masking bias. Scaled models ranging from 127 million to 7 billion parameters, specifically DiffuGPT and DiffuLLaMA, demonstrate better or comparable performance to existing AR models. Additionally, the authors release open-source code for these models, providing valuable resources to the research community for faster inference speeds with minimal iterations.", "questions": "The current T is set to 32. It would help to see the quality performance across different T, since it was mentioned in section 4.5 that decreasing T leads to faster decoding but with loss of quality.\nPlease include this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- Well-written and easy to understand.\n- Comprehensive evaluation on multiple tasks, including word completion, reading comprehension, commonsense reasoning, math problems, and coding.\n- Ablation study conducted to emphasize the importance of shift operation and attention mask annealing.\n- Mathematically grounded and novel approach.\n- Demonstrates increased token diversity without compromising quality.", "weaknesses": "- Missing qualitative examples and human evaluation.\n- No statistical significance testing across tasks.\n- The current T is set to 32. It would help to see the quality performance across different T, since it was mentioned in section 4.5 that decreasing T leads to faster decoding but with loss of quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)