Unifying Autoregressive and Diffusion-Based Sequence Generation
Authors: Nima Fathi
Torsten Scholak
Pierre-Andre Noel
Venue: COLM 2025 (2025)
Keywords: discrete diffusion
generative diffusion models
language models
autoregressive language models
TL;DR: We build upon diffusion language models to 1. make them autoregressive and 2. use an hybrid of the "uniform" and "absorb" token noising processes.

Abstract: We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models.
We introduce *hyperschedules*, which assign distinct noise schedules to individual token positions, generalizing both autoregressive models (*e.g.*, GPT) and conventional diffusion models (*e.g.*, SEDD, MDLM) as special cases. 
Second, we propose two \emph{hybrid token-wise noising processes} that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a *novel inference algorithm* that leverages this new feature in a simplified context inspired from MDLM.
To support efficient training and inference, we design attention masks compatible with KV-caching.
Our methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting a promising path for autoregressive diffusion-based sequence generation.
See code and resources at https://hdlm-colm.github.io/ .

Published as a conference paper at COLM 2025
Unifying Autoregressive and Diffusion-Based Sequence Generation
Nima Torsten Scholak & Pierrre-Adré Noël
ServiceNow Research nima. fathi@mila quebec, (torsten.scholak,ppe perperrreandeeeeeeeeeeeerroooooo poeevvieervccn
noel
Abstract
We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models. We intro- duce hyperschedules,, which assign distinct noise schedules to individual token positions, generalizing both autoregressive models (e.g., GPT) and conventional diffusion models (e.g., SEDD, MDLM) as special cases. Sec- ond, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes, and we introduce a novel inference algorithm that leverages this new feature in simplified context inspired from MDLM. To support effi- cient training and inference, we design attention masks compatible with KV-caching. Our methods achieve saate-of-the-rt perplexity and generate diverse, high-quality sequences across standard benchmarks, suggesting promising path for autoregressive diffusion-based sequence generation. See code and resources at https: //ddss/cohmmmm mooolm..iit.... thub..o//.
1 Introduction
Generative diffusion models, primarily recognized for their impressive image generation performance in continuous domains (Yang et 2023), are rapidly gaining traction in lan- guage modeling, a discrete domain historically dominated by autoregressive (AR) models such the GPT family (Radford et al., 2019; Brown et al., 2020). Contrary to the apparent separation between AR and diffusion models, and despite their distinct historical develop- ment, this work reveals fundamental connection: AR models are a form of diffusion.
The core principle behind diffusion models involves prescribing a "noising" process that gradually destroys information in training data samples, subsequently learning neural network that progressively generates new samples from "pure noise" with a denoising process. The noising process acts form of data augmentation: together with the original training dataset, it specifies the curriculum on which the generator (denoiser) is trained. Part of the attraction for these models arises from their rich theoretical grounding, resulting in concrete practical techniques. In particular, a model's training and inference environments can be decoupled, allowing for compute-budee knob at inference time, and guidance techniques adapting a model's behavior to specific situations.
Despite the common use of Gaussian noise in continuous diffusion, the underlying princi- ples can be adapted to discrete state spaces (Austin et al., 2021; Zhou et al., 2023; Lou et al., 2024). Common practices for sequence generation have the noising process randomly and in- dependently substituting some original tokens by completely unrelated ones, i.e., uniformly sampled tokens or a special "mask" absorbing state. A noise schedule determines token replacement probabilities at different points in the curriculum. The resulting sequence at the schedule's highest noise level retains no mutual information with the original sequence. The generator is trained on this curriculum to enable the production of novel sequences.
This work unifies AR and diffusion sequence generation by introducing hyperschedules, allowing different positions in the sequence to be affected by different noise schedules. We
* Also Québec Artificial Intelligence Institute (Mila) and McGill University.

---

Published as a conference paper at COLM 2025
The training curriculum gradually destroys information from data sample (noising).
fT-1T
4r-21 21-211-1 1
f122
10
XT-1
data
noise
PT-1IT-2
P211
Pijo
At inference, new samples are generated in generation steps (denoising).
Figure 1: Generative diffusion models prescribe (through 9t/++1) a curriculum process (X+), then learn (through pf 11p) a reverse process (Xt) so that the marginal distributions match at each step & (vertical squiggly lines). XT is the training dataset and Xr is the generated output. This work focuses on discrete diffusion for sequences of discrete tokens. We show that standard autoregressive models (e.g., GPT) are an extreme case of this framework, unification enabling vast continuum of diffusion models, including autoregressive ones.
establish that autoregressive models, such as GPT, can be understood as diffusion models without data augmentation, utilizing a discrete noise schedule comprising only "full noise" and "no noise" levels. This unification expands model design space and enables variety of generalized AR-like approaches. With the hindsight, hyperschedule be identified in the "Mask" process of Bansal et al. (2023) well as in the "Swin-DPM" technique of Feng et al. (2024).
concepts can
Recent (Sahoo et al., 2024; Ou et al., 2024; Shi et al., 2024) and concurrent (Liu et al., 2024; Kim et al., 2025; Peng et al., 2025; Nie et al., 2025; Wang et al., 2025; Arriola et al., 2025) works have focused on mask diffusion models (MDMs). By specializing mechanism, these MDMs enable great simplifications over a general-purpose treatment: neural networks no longer need an explicit noise-level dependency, and more standard loss function can be used. However, our work shows that the same feats and simplifications can be accomplished in a general, non-MDM case. Motivated by the same rationale that led concurrent MDMs to conceive "remasking" strategies, we introduce hybrid noising pro- cesses, interpolating between the "absorb" and "uniform" processes combine the benefits of both and achieve state-of- oatt--of-hhe-at performances, with aspects further improved by our novel adaptive correction sampler (ACS) inference algorithm. Our hyperschedule-equipe approach supports specialized attention matrices enabling KV-caching and efficient training.
the "absorb" noising
In summary, our main contributions are:
we unify AR and diffusion sequence generation by introducing hyperschedules;
we consider hybrid noising processes, reaping benefits from both leading noising processes and achieving state-of-the-art performances, with and without our novel ACS inference algorithm; and
our hyperschedule- pooweee hybrid processes generalize multiple concurrent de- velopments to non-MDM setting, including efficient training and KV-caching.
Unification Through Abstraction
In this section, reconcile autoregressive and diffusion-based language models by abstracting-one their respective implementation details, instead emphasizing their shared essence.
Sequence Generator We define sequence generator as a stochastic procedure that repeat- edly calls a neural network to yield sequence of tokens, each taken from a finite vocabu- lary X. We write x, = (xf).... ... ,, - E d the state of the generator after calls: starting from a (trivial) initial state &o, the neural network specifies the probability pi 1j; 1&x) to get to state 81+1 from state &, The generated sequence &y obtained after T such calls. The bottom row of Figure illustrates this process (from right to left, writing X, the stochastic variable taking value &,).
2

---

Published as a conference paper at COLM 2025
Quenched (AR)
b. Flat Annealing (standard)
Block Annealing
d. Slide Annealing
1. Hypersschedul
1-5
[04 [+5 (A1)
1-0 1=1
1-3
1
1
1-5
(h.)
[eD (+]
[+2
(+5 (c4)
2. 2.Absorb Process
(22)
2=0
(k.)
(4.2)
3. Uniform Process
(Not useful)
(b.3)
(6.3)
4. Hybrid Process
1-U
1
(Not usefid)
1
[+U [+1
1=5 (8.4)
Figure 2: We introduce T-hyperschedules (top row)- - subjecting different token positions í with different noise levels (red high; blue low) at different generation step f and illus- trate how three different noising processes (bottom 3 rows) can be modulated by such hyperschedules. Hyperschedules. (a) Standard AR models (e.g,, GPT) determine tokens one by one, "quenching" each of them to full determination in a single step. They may thus be construed as an extreme case of a diffusion model. (b) Standard diffusion models (egg., SEDD) gradually anneal all tokens independently of their position. (c) Block-wise application of flat annealing, here for blocks of width w' = 3. (d) Annealing with a sliding window ("smoothed" AR), here using window width w 3. These last two examples share important features of both AR and diffusion models. While the 4 presented examples all generate p
token per step in the long-sequence limit (with the caveat that slide experiences an initial overhead of w - 1 steps), the last 3 patterns are all straightforwardly adapted to p ("quick draft") and p The absorb noising process a.k.a. Masked Diffusion Model (MDM) overwrites tokens with a special MASK token. These masks are "known unknowns": it is clear to a denoising model that it must put a non-mask token in its stead. Conversely, unmasked tokens are taken as "absolute truth" " during generation: once token has been unmasked, it remains unaltered until the end. (3) The uniform noising process overwrites with (non-mask) tokens selected uniformly at random from the vocabulary of possible tokens. The orange/ biitgg///lghttueeeee color coding is not part of the state x,, and is provided solely for the reader's 's convenience. The model thus has no direct way to know if a token has been altered before ("unknown unknowns"") and may thus revisit position's value many times during generation. (4) The lyyrrid noising process blends a little bit of uniform into the absorb process. When denoising, MASK tokens still represent clear "known unknowns", while unmasked ones have become "a priori good" candidates that may however need to be "fixed". We argue that it is desirable for models to learn to fix their own mistakes.
("think hard") regimes. Noising Processes. (2)
3

---

Published as a conference paper at COLM 2025
Discrete Diffusion Sequence Generators A sequence generator's 's neural network must learn the function Pa=ate (& 1|:)) such that the yielded Xy approximates the distribution of clean training samples xr (no hat!) from the training dataset. To this end, the "diffusion" paradigm (Yang al., 2023) is to prescribe distribution 91-11 (x:) from which we can sample x;-1 given x,. This noising process (top row in Figure 1) gradually destroys the information in the training data sample xT, until nothing remains of it in X we call curriculum the process {X:} obtained by recursively sampling the prescribed q; (x 1/xt) starting from training sample Xr. The training objective for pf-11 amounts to denoising such noisy samples, aiming to align the marginal distribution of X with its X, counterpart (noted X, X X in Figure
Token- Wise Processes The noising process is typically constructed out of many in- dependent processes, each affecting one of the original tokens, i.e., 9:/++1 (x (211))1)
Ih (x)1++1))))))))1 In practice, // (x)]x(4)) is given by the x-th column of a matrix Qtp 11 that is independent of the position For example, SEDD (Lou al., 2024) uses
Q exp((Or-: - Ør (11xk)) where the transition matrix Quok is either of
0 0
Quniform
Q Absorb
: 0
(1)
2 X
-
1 0
and 0o 5. ÜT are cumulative noise schedules such that dn A 0 and Øy Re co, Here X is the number of tokens in the set X, including special MASK token associated with the last dimension of Otok- In words, using QUniform gradually replaces tokens by random ones, while QAbsorb gradually replaces them by MASK. Generation starts from & sampled from the stationary distribution Po (i.e., random non-mask tokens for QUniform and all-masks for QAbsorb), and Pi-11 is learned in terms of diffusion weighted denoising score entropy (DWDSE) (Lou et al., 2024).
Although MDMs technically correspond to the above for the case QAbsorb, the literature has converged on a much simpler formulation in terms of z ag 2 41 2 ... 2 aT R 0 such that x; has probability ar-r to the original token y' and probability 1 ar to be MASK. The resulting transition matrix Q = + (1 "T-r ) Abssor may be further simplified, 47-1-1
allowing training pf 1++ using a weighted cross-entropy loss (Sahoo et al., 2024).
AR Sequence Generators Most modern language models predict tokens auforegressivelly: one token at time, each conditional on the tokens that precede it. This may be viewed as a sequence generator with T d: Xn = (MASK, MASK, .MASK) represents an empty sequences using a special MASK token, and each P1 (& predicts the token 8 to substitute the first MASK encountered in k, This can be viewed as an extreme case of diffusion model where the conditional probability 4t|++1 is actually deterministic function that masks-out the 1-th token, ultimately resulting in X9 composed solely of MASK tokens. However, notice that this (x: Th 9 1+1 (xf] x involves factors depending on the position i, which is forbidden in the discussion above; Section 3.1 palliates to this issue.
Transformers All sequence generators considered in this work are implemented as trans- formers (Vaswani et al., 2017), more specifically the backbone from Peebles & Xie (2023). One subtlety is that many diffusion and/or or masked language models use an ALIGNED configuration where each transformer cell predicts (output) the same token position as the one it receives (input). In contrast, AR models use the SHIFTED configuration: each cell predicts the next token in the sequence. These subtleties can be abstracted out at an high level (see details in Figure 4 in Appendix).
4

---

Published as a conference paper at COLM 2025
3 Autoregressive Sequence Diffusion
3.1 Hyperschedules: position-dependent schedules
We generalize standard diffusion curricula by subjecting different token positions i to different noise schedules, distinguishing the number of generation steps T from the number of noise levels T. Concretely, 4/p+1 (x'[x) is given by the x-th column of a 1/t+1 obtained by substituting each instance of U'Ti a7 in Qij(+1 by of or arj respectively, where the hyperschedule T/ E {0, 1, T satisfying T = T 2 - 2 ... 2 T/ 0 for all positions i E {0..... a - 1}. In effect, the noise schedule (o or a) unfolds differently at different positions; hyperschedules act as schedules for schedules. See examples on top row of Fig. 2.
We introduce two characterizations of an hyperschedule. First, we define the window width w as the largest (among all steps f) number of positions i for which (I), x is neither (0,0) nor (T, TT. All other things being equal, lower offers more opportunities to improve inference time (see Sec. 3.3 for examples). Standard AR models use T TQuench with value where i 2 and 0 elsewhere, and thus have 1 by construction (Fig. 2(a.1)). Standard diffusion models use Flat with value T - t (often called "noise level") for all i, using T T and w (Fig. 2(b.1)). Two w-parametrized examples are also provided: concurrent work on block diffusion (Arriola et al., 2025) may be understood in terms of Block (Fig. 2(c.1)), and we introduce novel Slide (Fig. 2(d.1))
Second, we define the token generation rate the long-sequence limit of the ratio d/ TT All hyperschedules explicitly presented in Fig. share the same p 1: in the long run, they require one model call to generate one token. However, all but Quench may be readily adapted to "quick draft" (i.e., > 1) or "think hard" (i.e., < 1) regimes.
3.2 Hybrid Processes: training denoisers that can fix their mistakes
In the absorb process, each step produces the curriculum by replacing some of xT's entries by the special MASK token. Conversely, at generation time, the only action available to the generator is to replace some MASK tokens by non-MASK ones. Notice that, unless the generator is "perfect", it may become apparent late in the generation process that some early token choices were, in retrospect, inherently incompatible. However, there are no action available for the model to "fix" these token choices: no backsies. Although such "hindsight" situations may occur in any domain, they are particularly relevant to computer code generation and other reasoning-intensee
tasks.
Conversely, the uniform process can replace any (non-MASK) token by any other one, both at curriculum specification and sequence generation. Thus, at no point in the generation process does the model have any indication whether it has already altered given token before. We hypothesize that this may cause a "lack commitment" on the model's part: how much should you "trust" the value of token? At least with absorb, MASK tokens capture known unknowns.
These observations motivate an hybrid forward process, of which we consider two vari- eties. The first one uses the SEDD framework with Qiok set to Qiyybrid (1 QAbsorr Y QUnnfform interpolating the QUniform and QAbsorb extremes according to an hyperparameter 0 < < 1. The evolution operator exp (0 Teft. )ffybria) can be solved analytically
(because QUniform and QAbsorb commute, see Appendix A), enabling use in practice with the standard SEDD loss.
Our second hybrid process variety is closer to the MDM framework: unless Ti = (in which case x = x)),, the probability distribution for x; is given by the xf-th column of (1 + e @Qunfform) ) (1 + (1 RAbsot)),,,, where 0 < < is a step-independent probability that the token is substituted by a uniform one, followed by standard MDM process
As an extreme example, consider the graph coloring of a particularly nasty instance.
5

---

Published as a conference paper at COLM 2025
henceforth. A weighted cross-entropy loss is used (see Appendix B.2) and, like MDMs, the neural network does not require an explicit noise-level dependency. Which the two variety is used can be inferred from which of y is specified.
3.3 Attention Mask and Efficiency
In SEDD, each call to the transformer predicts each entry of & 1 in view of all entries in &t. contrast, standard autoregressive models use a causal attention mask to ensure that 8/41 may only depend on 8j". Combined with the fact that 8/ = fif at all step f, this causal maskenables inference-time efficiency improvements such as KV-caching, MDMs such as Sahoo et al. (2024) can enable similar form of caching by relying the special role of MASK tokens.
However, new opportunities for optimization come up when the hyperschedule follows certain autoregressive-liee regular structure such as the ones seen in Figs. 2(c.1, d.1). Indeed, at each steps the hyperschedule T; and the state & both break in three components
T =
settled
rpetive
worthless
8 = qgettled gactive
soorthless
(2)
where: riettledd composed exclusively of zeros and
x sqettled matches the first entries of & 9; both raetive and sactive have at most w < d entries; and eyvrttheess is composed exclusively of repeated T while & sworthless bears no information about Thus, when using autoregressive attention mask on geettled, all the conditions on these tokens just as in a standard autoregressive model. We may completely ignore & worthhless which leaves a small number of positions that densely attend to & geettled gactive when generating x++1 See details in Appendix B.
met to KV-caching
3.4 Adaptive Correction Sampler
In addition to the theoretically-groundee inference schemes from SEDD and MDLM, we introduce adaptive correction sampler (ACS), novel variation on MDLM's sampler that allows the model to alter the value of already-unmasked tokens, and that has empirically shown to perform particularly well for our hybrid process of the e-variety. We write Piransfy the probability that MDLM's sampler (adapted to our hyperschedule) would unmask the i-th token if it is masked, and proceed as usual for the token that are so masked. However, where MDLM would leave already-unmasked tokens as they are, ACS has probability sample a replacement token from the model's prediction. ,/ serves as hyperparameter modulating the intensity of this correction. Pseudocode for the original sampler and ACS sampler can be found in Appendix D.
7 (1 (trsstr
4
Experiments
4.1 Experimental Setup
Our experiments investigate three primary design dimensions: (i) the choice of hyper- schedule (e.g., TFlatz USiide, and TBlock) illustrated in Fig. 2; (ii) transformer configurations, specifically ALIGNED and SHIFTED as depicted Fig. 4; and (iii) the hybrid process, par- ticularly our H-Hybrid with Qfyybri operator and e-Hybrid parametrized by e. Additional implementation details are provided in Appendix B.
4.2 Language Model Likelihood Evaluation
We evaluate model likelihood performance across two established datasets.
OPENWEBTEXT (OWT) (Gokaslan et al., 2019): Given that OWT lacks standard data split, we follow Ou et al. (2024), reserving the last 100K documents as a held-out set for test perplexity reporting. All OWT models utilize context length of 1024 tokens with the

---

Published as a conference paper at COLM 2025
Mark
Method
Test PPL +
7
(a)
Baseline SEDD-Absorb [14]
24.10
Flat
(b)
(a) Hybrid Process
22.30
0.01
(b) Weighted token-embedding(= (rainn===Hyyoon 7-Hybrid)
22.18
0.01
Flat
(d)
(c) Transformer time-conditioning
22.47
0.01
Fiatt
(e)
(c) Slide
riu=d
21.53
0.01
rw=d
Slide
Table 1: Test Perplexity for various design choices (lower is better), measured the heldout
100k sample from OWT dataset. All ablations use ALIGNED with d 1024.
Parameters
PPL (4)
OmniNet, (Tay al., 2021)
100M
21.5
Autoregressive
Transformer (65B tokens) (Sahoo et al., 2024)
110M
22.3
SEDD (65B tokens) (Lou et al., 2024)
110M
32.8
Diffusion
MDLM (65B tokens) (Sahoo et al., 2024)
110M
31.8
BD3-LMs L' 4 (65B tokens) (Arriola et al., 2025)
110M
28.2
y-Hybrid [y- 0.02, Tre A
(66 tokens)
110M
27.8
H-Hybrid [y= 0.02, SSnEEnn)
(56 tokens)
110
28.3
Diffusion (Ours)
y Hybbid [y= 0.02, 20 (taGNNn]
Bieck
(65B)
110M
27.1
7- H-Hbrid Lym 0.02, Blcck AAAGNND) (65B)
110M
27.0
7- H-Hbrid Lym 0,02, a sunrmn] (65B)
110M
27.5
y- y-Hybrid Ly= Dlick SHIFFE]]
(65B)
110M
26.6
Table 2: Test perplexities (PPL; 4) on LM1B. Perplexity values for diffusion models are upper-bound estimations. 'Reported in He et al. (2022). #Reported in Sahoo et al. (2024). Best diffusion value is bolded.
GPT-2 tokenizer (Radford et al., 2019).
LM1B (Chelba et al., 2014): We use the bert-base-uncased tokenizer, reporting test per- plexity on the predefined test set. All LM1B models trained with 128 tokens contexts.
4.2.1 Ablation Study
We conduct a systematic ablation study to assess the contributions of key design choices on OWT (Table 1). Our baseline model (a) follows the original SEDD-Absorb setup Lou et al. (2024). Introducing our hybrid transition operator Qityrrid significantly improves performance, noted in configuration (b). We further incorporate weighted token embed- dings inspired by Ou et al. (2024), yielding additional improvement (configuration (c); details in Appendix B.5). However, removing timestep conditioning (an operation that isn't theoretically justified for y-Hybrid, in contrast with MDLM and e-Hybrid) slightly degraded performances (configuration (d)), prompting us to retain this component. We also
explored the 10==1024 curriculum, which demonstrated modest improvements at the cost of slide increased computational overhead due to additional forward passes.
4.2.2 Language Modeling Performance
Following our ablation insights, we compare our best-performing configurations against state-of-the-art autoregressive and diffusion baselines on LMIB. Results presented in Ta- ble 2 illustrate significant improvement upon prior discrete diffusion baselines, reducing perplexity by approximately 19% relative to SEDD (Lou et al., 2024) and 3% compared to the strongest existing diffusion models.

---

Published as a conference paper at COLM 2025
Method
PTB
WikiText
LM1B
Lambada
AG News
Pubmed
Arxiv
Transformer
82.05
25.75
51.25
51.28
52.09
49.01
41.73
(Sahoo at. 2024)
SEDD Absorbt
96.33
35.98
68.14
48.93
67.82
45.39
40.03
(Lou et al., 2024)
M
90.96
33.22
64.94
48.29
62.78
43.13
37.89
(Shooo at.
BD3-LM L'
96.81
31.31
60.88
50.03
61.67
42.52
39.20
(Armala 2025)
y-Hybrid (444B)
89.94
30.02
61.01
45.38
67.51
46.57
40.62
ly= 0.01, 0.010 ALIGNED]
e-Hybrid (444B)
90.89
32.53
68.91
50.23
64.61
41.18
37.85
[== 0.01, Tra ALIGNED]
7-Hybrid
90.67
31.73
73.71
50.03
68.27
41.49
37.89
1 0.01, Side ALIGNED|
y-Hybrid
95.22
32.64
63.68
44.75
62.18
42.01
37.33
[y= 0.01,
Block SHIFTED]
Table 3: Zero-shot unconditional perplexity on seven benchmark datasets from Lou et al. (2024) and Sahoo et al. (2024) and Arriola et al. (2025). #Reported in Arriola et al. (2025). All models are trained for 524B tokens unless otherwise stated. All diffusion models are upper bounds; the best diffusion value is bolded. See Appendix F.1 for complete results.
4.3 Zero-Shot Generalization
We evaluate the generalization capabilities our models across seven standard datasets considered before by Lou et al. (2024) and Sahoo et al. (2024). As shown in Table 3, our Hy- brid diffusion language family of models, (y-Hybrid and e-Hybrid) consistently surpasses prior discrete diffusion approaches on 5 of the benchmarks. Moreover, we narrow the gap between diffusion-brl and AR models, outperforming AR baselines on two datasets.
4.4 Sampler Performance
We evaluate the performance gained from our proposed adaptive correction sampler (ACS). Table 4 empirically shows that ACS improves the e-Hybrid models' ability to correct ini- tial missteps, resulting in sequences that are both more coherent and of higher quality. Pseudocode for the original sampler and ACS can be found in Appendix D.
MAUVE (1)
Gen PPL. (L)
Entropy (1)
Model Family
Sampler
p 2
p p= 1
== 2
p p=1
== 2 p =
e-Hybrid
Original Sampler
0.848
0.779
121.90
129.52
5.49
5.50
[e
Trin ALIGNE]]
ACS
0.957
0.947
61.35
43.98
5.28
5.18
e-Hybrid
Original Sampler
0.778
0.847
139.64
142.13
5.43
5,46 5.25
-0-4/4 I= 0.01, BBakk ALIGNED]
ACS
0.813
0.916
71.77
59.15
5.38
Table Comparing the original sampler with ACS on e-Hybrid variants. For a more complete comparison please refer to Table 10.
4.5 Sequence Generation Quality-Diversity Trade-offs
We further investigate the trade-off between sequence generation quality and diversity by analyzing generative perplexity against token-level entropy and MAUVE? scores (Pillutla et al., 2021), as visualized in Fig. 3. Our hybrid configurations consistently achieve superior positions on both Pareto frontiers, indicating enhanced generation fluency, coherence, and diversity compared to baselines; new of-te--f-te-atttt in diffusion language modeling.
MAUVE strongly correlates with human judgment of text quality and diversity.
8

---

Published as a conference paper at COLM 2025
60
Model Family
erttane tor 305 Tox alpredt
Coneration Parplexity
ertaaaa you alymas
100
Takan rate [p)
120
140
140
5.2
5.5
0.6
0.7
Entropy
MAUVE
Generative perplexity as a function of token-level entropy. Right: Generative perplexity versus MAUVE score. Our models consistently outperform baselines, achieving lower perplexity at comparable levels of diversity and fluency.
Figure Left:
4.6 Additional Analyses and Results
Qualitative generation samples, both conditional and unconditional, are included in Ap- pendix G. Hybrid models exhibit robust performance, effectively handling extended context lengths beyond the training range, particularly for OWT. Comprehensive ablations detailed in Appendices 4.2.1, F, and F.4 confirm the effectiveness of our design choices. Specifically, we find optimal performance around small y values (0.01 0.1), critical for balancing token commitment and flexibility, as evidenced in Table and Fig. 10. Additionally, adjustments in generation rate p notably impact both quality and inference speed (Table 9), with our proposed Block schedules significantly benefiting from KV-caching (Table 11).
5 Conclusion
Diffusion-based language models offer some unique opportunities including theory- supported guidance strategies and the native ability to iteratively improve their answer but these benefits are no substitutes for language modeling performances. Staggering resources are continuously spent in scaling up AR models, engineering tools and techniques specialized the AR paradigm. How could diffusion models even dream of catching up?
This work takes significant step toward a bold strategy: language model, we wish convert it (e.g., fine-tuning) into an even better diffusion-based sequence generation model. This plan demands a SHIFTED configuration, an hyperschedules generalizing the AR concept (e.g., Tlide or Block),, and curriculum (such as our hybrids) teaching the model how to generate quality sequences without painting itself into corner.
starting from an already-great AR
Much of the design space opened by our innovations remain be explored by future work. We have merely glanced at the realm of possible hyperschedules, and our success with e- e-Hybrid illustrates that more involved curricula can pay off, without the need to provide explicit noise levels to eventual pre-trained AR model.
Our innovations also open the path for more fundamental work. Indeed, the limit < & d presents opportunities for tractable approximations of the joint distribution over w tokens. On a different front, while our current approach employs a uniform distribution for replacing tokens, further improvements in diversity and quality may be achieved with distributions that more accurately reflect plausible, "honest" and/on or "on policy" errors (rather than purely random token substitutions.

---

Published as a conference paper at COLM 2025
References
Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://penreeview http://wwwwwe/ffoun///e/em wet/fforum/id=ttyyyTT7777.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Infor- mation Processing Systems, 34.17981-17993, 2021.
Arpit Bansal, Eitan Borgnia, Hong-Min Chu, Jie Li, Hamid Kazemi, Furong Huang, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Cold diffusion: Inverting arbitrary image transforms without noise. Advances in Neural Information Processing Systems, 36:41259- 41282, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In Conference of the International Speech Communication Association (Interspeech), 2014.
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infinite-horizon world generation with real-time moving control. Xrv preprint ar arii:2411..0366. 2024.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019. URL http::t////kliinoo))) github. htt/OpenWeeTextCorpppeee Accessed: [DATEJ.
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffu- sionbert: Improving generative masked language models with diffusion models. arxiv preprint ar arXie:2211.15029 2022.
Ari Holtzman, Jacob Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Ce Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2742-2751, 2019.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arviv preprint ar Xiv::5522.06765 2025.
Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael Köómez-Bombaree Think while you generate: Discrete diffusion with planned denoising. Xrx preprint ar xrXiv:2410.06264, 2024.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference Machine Learning, 2024. URL https: http///ppeereeeeeeeeeeeeeee
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arvio preprint ar XxX::2500200999. 2025.
10

---

Published as a conference paper at COLM 2025
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. Xiv preprint ar (r:::22406.00733 033736, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Sherwood Yao, Jarrid Rector-Brooks, Alexander Tong, and Pranam Chatterjee. Path planning for masked diffusion model sampling. Xrv preprint ar Xix::25002.03540, 2025.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34: 4816-4828, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 18))9, 2019.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. The Thirfy-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https://openretee www.re...weeeooouuiitttt L4uaAR4ArM.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias, Simplified and generalized masked diffusion for discrete data. arviv preprint ar XrXio:2406004329 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063 2024.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference Machine Learning, pp. 10193-10202. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https::/proceediii iewrrreeeeeenneeenii.ses........ wtt.c//pppe..llee//eeeeeeeeeeeee file:/3f5ee243547deee Tbo003cc1c4aB455a-Pappe pdf.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint ar arXio:2503.00307, 2025.
Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. Xrv preprint arXiv:24499029000 2024.
Kun Zhou, Yifan Li, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion-nat: Self-prompting discrete diffusion for non-autoregressive text generation. Xiv, 2023.
11

---

Published as a conference paper at COLM 2025
A Evolution Operators
Although our SEDD-based curricula are a priori defined in terms of an arbitrary Qiok actually using a model in practice demands that we can analytically solve the evolution operator
(Ap(kQ for A IR+ This section re-derives the solutions for Unitfrmm and Abbsorb then extends the results to Hyybrid
Using the definitions in Eq. (1), we can verify
(Q (bbsort)) ) Q Absorb
(3a)
(QUniform) )2 - QUniform
(3b)
Uniform QAbsorb Quiiform
(3c)
Q Absorb Unifform QUniform
(3d)
Notice that, for any matrix Q- such that (Q+)2 = ^Q=, we have
($..)*
(
(Ag)
eqQ.
Σ
Q
1
k!
k!
k=
= =1 - A (1-
(4)
Together with Eq. (3), re-obtain the evolution operators used in SEDD
e QAbborr = 1 + (1 e Absorb
(5a)
QQnifoe
(1 -1-e - Qn
(5b)
Now notice that QAbsorb and QUniform commute
[Qbbsorb/ Uniiform QAbsorb QUniform QunitQUnnnuutttt Abssrt = 0
(6)
which enables the analytical solution for H
A Hyyna
(7a)
A((1 AAtbaan + Unifform)
(7b)
= (1 (1--7)QQbooor e AUQunffom
(7c)
= [1 + +(1++1111111 e - ( [1 (1 -1-e - Qmiffrrm
(7d)
1 + (1 e ((1-) RAbsorb (1 e - ) QUniform (1 e (1---()))) ~7A ) Absorb QUniform (7e)
-1+ (1 e (1 7) QAbsorb + (1 - 7 = - (1-c 7) e 74 Quiiform
(7f)
=1+ e1- (1- ) (e
(1-7)4 (e-(1-)A
e -)
(7g)
Equation (7g) is the desired analytical solution for the evolution operator.
Additional notes on ELBO and hyperschedules. Theorem 3.6 from Lou et al. (2024) provides an ELBO bound enabling training with the DWDSE training loss. This Theorem holds for arbitrary constant-coefficient diffusion matrix, but is derived for the special case of
schedules. One might thus wonder if the result still holds if we simply substitute in our Dosition-dependenn hyperschedules.
As mentioned in the beginning of Section 3.3 of Lou et al. (2024), such general diffusion matrix would be exponential in size, justifying decomposing the process into d independent subspaces in practical applications. The evolution operator for each of these subspaces has the form exp(AQuos and, because of their independence, the full evolution operator is simply the product of their action the independent subspaces.
When substituting in an hyperschedule instead of a schedule in SEDD's framework, the only difference is that the evolution operator for each subspace, Aap(AQuol gains dependency
12

---

Published as a conference paper at COLM 2025
X
x
x
x
xy-1
x
Sequence Generator
Sequence Generator
Posiprocsssing
Posiprocessing
Transformer
Transformer
Preprocessing
Preprocessing
8,0
x
X
X
X
&
x
g
(a) ALIGNED config.
(b) SHIFTED config.
Figure 4: Two transformer-based sequence generators for d 4. (a) The ALIGNED config- uration of standard diffusion models is reminiscent of masked language models. (b) The SHIFTED configuration is closer to autoregressive language models. Here f-1 represent token solely part of the conditioning (i.e., not generated), and may or may not be constant (egg,, BOS). Similarly, f represents that the output associated with the last token discarded. Our position-based indexing abstracts away these details.
on the position associated with this subspace. Crucially, the subspaces are still independent, and the overall evolution operator is still simple product of the independent solutions. All other derivations lending SEDD's ELBO bound thus directly generalize to hyperschedules, including the case of --Hybrid.
The case of e-Hybrid is more complicated. When O, we fall back on MDLM, and the ELBO derivations from Sahoo et al. (2024) thus apply. However, for ε > O, the evolution operator cannot be represented under the form exp( AQuok): we do not have an ELBO bound for c-Hybrid when e > 0. In any case, our empirical observations support its use in practice.
would be interesting to see if training with a proper ELBO-bounded loss could improve the performance of e-Hybrid models, and if something akin to our ACS algorithm could be derived from first principles. However, these concerns are outside the scope the present work.
B Implementation Nuances
This section discusses several implementation details that affect both our model training and evaluation procedures.
B.1 Training Setup
We now detail our experimental training procedure, structured into two distinct stages. In Stage 1, we initially train our base models in both ALIGNED and SHIFTED configurations using the hybrid noising process paired with the flat hyperschedule That- Specifically, we adopt different modeling strategies depending on the chosen variant. For 7-variant models, we extend the existing discrete diffusion framework from Lou al. (2024), integrating our proposed hybrid transition operator Qifybrid For e-variant models, we instead employ the newly proposed hybrid diffusion cross-entropy (HDCE) loss, detailed in Equation 8. Stage 1 WebText and LM1B. Each
models are trained separately on two standard datasets: WebnWe configuration undergoes training for approximately 850K gradient updates with batch
13

---

Published as a conference paper at COLM 2025
x
A/
Aligned:
Shifled:
Figure 5: Example of attention mask for ALIGNED and SHIFTED configurations. Although these naive masks are appropriate for inference, directly training them would be ineffi- cient; see Figures 6-9 for training-ready masks examples.
size of 512, processing roughly 444B tokens for OpenWeb and 56B tokens for LM1B. This pretraining took about 14 days on 8xA100-80 GB GPUs for Open and 7 days on 2xA100-80 GB GPUs for LM1B.
In Stage 2, we fine-tune the Stage 1 models under alternative Supperccheuuee--secfiiiie
Tolock and Tslide- These experiments involve custom sestomddesigned attention masks tailored to each hyperschedule. Due this customization, highly optimized attention kernels such as Flash-Attention are not applicable, necessitating reliance the standard Py attention mechanism, which incurs higher computational costs. Stage training continues for an additional 150K gradient steps, resulting in cumulative training volume of 524B tokens for OpenWebTex and 65B tokens for LM1B. This fine-tuning required approximately 3 days on 8xA100-80 GB GPUs for OpenWeb Text and about 36 hours on 2xA100-80 GB GPUs for LM1B.
B.2 Loss Function
Our training objective consists of two primary loss components: (i) a standard cross-entropy loss computed on the settled tokens, and (ii) a diffusion weiihtte loss calculated on the active tokens. We propose the Hybrid Diffusion Cross-Ents casss-Etropp (HDCE) as our diffusion-weighter loss, which blends per-token cross-entropy loss with a specialized weighting strategy contingent upon whether tokens are masked, shuffled, or unchanged.
Formally, the HDCE loss is defined as:
LHDCE(0) 1 d Nd
Σ Σ
Wi log (o xxx)].
(8)
where N denotes the batch size, d is the sequence length, and the per-token loss corresponds to the conventional cross-entropy formulation. The token-specific weights Wi are defined
14

---

Published as a conference paper at COLM 2025
if Xi is masked,
Pmskk (i,
1(
E)
Wi
Pmakk (1 Ae
if xi is unmasked and shuffled,
(9)
if xi is unmasked and not shuffled,
Pmask ti)
where Pmask fi, is the masking probability for token X and Ả and represent hyperpa- rameters controlling the relative importance of shuffled versus unshuffled tokens.
In practice distinguish two model variants based on the employed diffusion-weighhee loss. For 7-variant hybrid models, we adopt the diffusion-weighted denoising score entropy (DWDSE) loss proposed by Lou et al. (2024), denoted as LDWDE Conversely, for our e-variant hybrid models, we use the proposed HDCE loss as defined in Equation 8.
Letting LCE represent the cross-entropy loss computed over seettled, our overall loss function thus expressed as:
B2 LDWDSS (x.ctive), for 7-Hybrid,
C = B1 LCF (CCE(xeettinn (qsettled)
+ B2 LDDEE (xactive), for e-Hybrid,
(10)
where P1, B2 E R are hyperparameters balancing these two components.
Additionally, since early positions in the sequence tend to become settled sooner, we apply reweighting strategy to normalize the contribution of settled tokens at different positions.
Specifically, we partition the sequence of length d into blocks of width w, assigning each token at position a weight:
[i/w|
w(1) = [d/w] -1'
0
(11)
with the convention that if [d/w] 1, then (()) 1 for all i.
B.3 Efficient Training and Inference
As mentioned in Sec. 3.3, we take particular care in crafting our attention matrices to enable KV-caching at inference time. These scheme are particularly beneficent when w & d, but naively using attention matrix such as Fig. 5 would train the diffusion head on only w positions while demanding to process on average d/2 context tokens. Here we present how may train the diffusion head on about approximately half the positions, increasing the training time efficiency by factor d/w. w. Note that, under these efficient schemes, the reweighing of active tokens as given in Eq. (11) is no longer required.
Figures 6-9 provides examples of attention masks that are compatible with the KV-caching scheme presented in Sec. 3.3, while dedicating about half the positions the denoising task. Light red/blue squares represent positions that are settled, whereas dark red/blue represent positions that are active. In all cases, the top-left part of the matrix has an autoregressive structure, and the production of dark blue positions attends densely on the corresponding dark red inputs well as the light red inputs that precede them. All cases presume d 12 and w 4.
The ALIGNED cases are easier understand. For Tslider Fig. 6 presents a situation where was randomly determined that the denoising will be performed on the intervals < i < min + w, d) for E (2,5, 11). For TBlock, these starting points are always multiples of w, here / E [0, w, )w) The light green blocks indicate entries that are not actually involved in the denoising and could thus potentially eschewed.
Figures 8 and present the corresponding matrices for the SHIFTED configuration. Notice how settled tokens (light red or the gray & are repeated as the first input of an interval to denoise in the second half of the matrix, and how the last output of each such interval is discarded.
15

---

Published as a conference paper at COLM 2025
4
4,3
414
471
4
11
4,1
4.1
67.
47.7
42-
4/1
Figure 6: Example training attention mask for ALIGNED configuration for use with -0=4
SSide"
B.4 Inference and KV-Caching
At inference, both Euler and r-leaping analytical solutions are available; however, our empirical results suggest that r-leaping is the de facto superior choice. Because each step "settles" prefix of tokens, we can cache the corresponding key /value pairs and avoid recomputing them on subsequent passes. Concretely, let
L sequence length, p tokens generated per step, window width.
Then the number of transformer calls is
N (LL-wil [I--w)// +1,
and the total token-processing cost is
Without cache:
N x w,
With KV-cache: w + (N 1)
16

---

Published as a conference paper at COLM 2025
%
X
4.17
4.
63.
47.
4-1
100
Figure 7: Example training attention mask for ALIGNED configuration for use with -==1
Block'
B.5 Weighted token-embedding
Ou et al. (2024) demonstrated that when employing the absorbing transition matrix QAbsorb scaling the model's score by the analytic, time-dependent factor
exp(-7(1))
1 e-xx((00(()
causes the remaining score be independent of f, eliminating the need to explicitly condition on time within the network. However, when using the 7-Hybrid process (1 (bso) + Quniform/ this factor remains present but is insufficient for capturing all temporal dependencies. In particular, under the hybrid process, an unmasked token is perturbed with probability e (t0(() (whereas under QAbsorb alone, non-mask token remains unchanged).
In other words, when the model encounters an unmasked input token xf subject to cu- mulative noise (f), should treat that token as were unperturbed with probability and
e-)((t) and as it were masked with probability e (t()), One natural way to embed this inductive bias into the model is to interpolate the token's 's embedding accordingly. Denoting
17

---

Published as a conference paper at COLM 2025
R/
4
x,
x,
4/
A212
67.0
6..0
Figure 8: Example training attention mask for SHIFTED configuration for use with -0=4
Slide'
by f(xx) E moddll the standard embedding of token x/, we replace it with
( f(x+) + (1 (1t))))))) f(MMAK).
(12)
18

---

Published as a conference paper at COLM 2025
x-
x2.
s...
22.
211
Figure 9: Example training attention mask for SHIFTED configuration for use with T 4
Block
19

---

Published as a conference paper at COLM 2025
C Model Evaluation Metrics
C.1 Upper Bound Estimation of Perplexity
(To simplify the notation, the following uses y to represent non-noised samples.)
Evaluating perplexity for diffusion-based language models is challenging because the model's likelihood involves an integration over continuum of noise levels. In our work, we estimate the negative log-likelihood (NLL) via a Monte Carlo (MC) approximation over discrete set of diffusion timesteps. In particular, given trained model pe(y) and a diffusion process that perturbs sequence y into latent states & (with f E [0, T'), our goal is to estimate the per-token loss that, when exponentiated, yields an upper bound on the true perplexity.
Let
log pa (y) Ex-a(t) [og pe 2)] DKL (q((x) 1y) (()))
In practice we approximate the expectation with Monte Carlo samples:
1 M M
[yy)
log (y &,),, 4 ~ Uniform(0, 1).
Since our loss function returns the total NLL over a sequence of length (i.e., it produces tensor of per-token losses whose sum over tokens yields the total loss for sequence), define the average per-token loss as
e = L(y) d
The estimated perplexity is then given by
PPL exp(
Because the MC approximation truncates the integration to M samples, the resulting per- plexity is an upper bound on the true perplexity. our experiments, we typically set M = 1000, and we observe that increasing M further reduces the variance of the estimate.
For generators with semi-autoregressive or autoregressive configurations (with a fixed window width w), we calculate the NLL only over the raclive tokens (i.e., those tokens that actively being updated during generation). This ensures that the perplexity computatin iiooutttion fair and reflects the model's performance on the tokens whose values are uncertain, rather than being diluted by tokens that are already settled.
We summarize the estimation procedure in Algorithm 1.
our setting, the loss function y(0,y,t) returns a tensor of shape [B,d], If the generator is semt-auuoregresstooe with a fixed active window of width w, then the loss is computed only the rective tokens. The expectation over t is approximated by averaging over M independent samples. Since the integral is truncaled, the compuled perplexity serves as an upper bound on the true value. Finally, dividing the total loss by Niot gives the average per-token loss, that the perplexity is computed as exp (aug loss per token).
Remarks 1.
Remarks 2. All our "Test PPL" metrics are computed using the loss presented above, i.e., not the one presented in Appendix B.2. For this reason, the upper bound provided by Algorithm assesses the quality of the neural network at language modeling. In particular, it does not account for the full inference strategy. Therefore, although the Test PPL metric may indirectly be affected by the training objective (described in Appendix B.2), the choice of hyperschedull and the values of E or y through their combined impact on the training of the neural network's 's weights, these hyperparameters are not directly involved when running Algorithm 1.
20

---

Published as a conference paper at COLM 2025
Algorithm 1 Monte Carlo Upper Bound Estimation of Perplexity
Require: Model parameters 8, loss function C, evaluation dataset D, number of MC samples
M, sequence length d Initialize total loss Liotal f- 0 and token count Niota <-
2: for each batch y D do
3: Initialize batch loss batch f-
4: for = 1 to M do
5:
Sample 1; ~ Uniform (0, 1)
Compute per-token loss Lj + y,i, E RBxd Lbttch batch +
6:
7:
8:
end for
9: Dbttc - T Lbtch
> Average over MC samples
10:
Stotal <- Lotal + L Sbatch
11:
Niotal 4- Niotal + x
12:
end for
13:
Compute average per-token loss: l Lrotal /
14: Compute perplexity: PPL +- exp(() return PPL
C.2 Generative Perplexity Evaluation
In addition to the intrinsic perplexity estimation described in Section C.1, we also assess our generator via generative perplexity. In this approach, pretrained autoregressive language model - in our case, GPT2-Large Radford et al. (2019) - serves as an external judge of the generated sequences. This method has been used in prior work Keskar al. (2019); Holtzman et al. (2020) as a proxy for fluency and coherence when direct likelihood evaluation intractable.
Concretely, our procedure is as follows: We first sample sequences from our diffusion generator using the analytical solution Lou et al. (2024). Since our generator and GPIg-Large may employ different tokenization schemes, the generated samples are retokenized using the GPT2 tokenizer. The retokenized sequences are then fed into the GPT2-Large model, which computes the negative log-likelihood (NLL) for each token; this value quantifies the surprise of the judge regarding the generated text. Finally, by averaging the NLL over all tokens and exponentiating the result, we obtain the generative perplexity:
PPI gen = exp
log PGP12(xs)
N
where N the total number of tokens in the generated text and PGrT2(x;) denotes the probability assigned by GPT2-Large to token x;.
In the case of semi-autoregressivve or autoregressive models with a fixed active window of width w, compute the NLL only over the tokens corresponding to the active portion retive This ensures that the perplexity is estimated fairly by focusing those positions where the model is actually making nontrivial predictions.
The full procedure is summarized in Algorithm 2.
21

---

Published as a conference paper at COLM 2025
Algorithm 2 Generative Perplexity Estimation via External Judge
Require: Generator G with parameters θ, pretrained judge model / (GPT2-Large), number of samples 5
1: Generate set of samples [y(s)] using the analytical solution of G
2: Retokenize each generated sample using the GPT2 tokenizer:
(1)) < Tokenizegrr2
3: for each retokenized sample y(() do
4: Compute per-token negative log-likelihowe g(+) +
log P1 (g(1)
5: end for
6: Compute the overall average per-token loss:
i = Σ ((8)
1 N
7: Compute generative perplexity:
PPL gen exp(2)
return PPLgen
D Adaptive Correction Sampler (ACS) Pseudo Code
For completeness, we present pseudo code for both the original sampling procedure and our proposed Adaptive Correction Sampler (ACS).
Algorithm 3 Original Sampler
1: Input: model, context length L, total steps S, temperature T.
2: Initialize x t a tensor of shape [B, L] filled with mask tokens.
3: Compute timesteps {4:] linearly spaced between 1 and 0).
4: for =ari=0,.... 0 do
5: Set f; and < fi
6:
Compute transfer probability Ptransfer
7:
for each token in do
8:
if token masked and random draw is below Piransfer then
Add Gumbel noise to the token's logits and update it via arg max. 10: end if
9:
11: end for
12: Update x accordingly.
13:
end for
14: return x
22

---

Published as a conference paper at COLM 2025
Algorithm 4 Adaptive Correction Sampler (ACS)
1: Input: model, context length L, total steps S, temperature T, correction parameter 1/.
2: Initialize x <- a tensor of shape [B, I] filled with mask tokens.
3: Compute timesteps [t:] linearly spaced between 1 and 0.
4: for 5, do
5: Set f; and f+41
6: Compute transfer probability Ptransfer <-
7:
for each token in do
8:
if token is masked and random draw is below Piransfer then
9:
Update the token using the standard denoising update (with Gumbel noise).
10:
else if token is unmasked and a random draw is below (1 - Ptramsfer) then Update the token via a uniform correction mechanism.
12: end if
13:
end for
14: Update x accordingly.
15:
end for
16: return
E Experimental Setup
In our experiments, we adopt a training and evaluation protocol similar to that of Sahoo et al. Sahoo et al. (2024). We conduct experiments on two datasets: the One Billion Word Benchmark (LM1B; Chelba et al. (2014)) and OpenWebTie (OWT; Gokaslan et al. (2019)). For models trained on LMIB, we employ the bert-base-uncasee length of = 128 tokens, and report perplexities the test split of LM1B. In contrast, models trained on OWT use the GPT2 tokenizer Radford et al. (2019) with context length of d = 1024 tokens.
tokenizer with a context
Since the LM1B corpus predominantly consists of single-sentence examples, a straightfor- ward padding scheme for reaching a fixed context length may not be optimal. Accordingly, following Sahoo et al. Sahoo et al. (2024), we concatenate and wrap sequences to fit context window of 128 tokens. Similarly, for OWT we concatenate and wrap sequences 1024 tokens, rather than simply truncating or padding, thereby ensuring that our evaluation is performed on coherent text segments. In the case ÓWT, which lacks designated validation split, reserve the final 100K documents for validation purposes.
Our model architecture builds upon the diffusion transformer framework Peebles Xie (2023), augmented with rotary positional embeddings Su et al. (2024). We instantiate our autoregressive baselines - SEDD, MDLM - with transformer backbone as described in Sahoo et al. (2024): 12 layers, hidden dimension of 768, and 128 attention heads.
F Additional Results
This section presents additional results and ablation studies our family of models.
E1
Zeroshot Perplexity
In Table 5 we report the full list of results for the family of our models against all the reported models available online. This is an extensive and more complete version of Table 3.
F2 Generative Perplexity Using a Judge LLM
In this subsection, we report the generative perplexity (see Section C.2) of our model. Zheng et al. (2024) were the first to demonstrate that the generative perplexity evaluations base- line masked diffusion language models are flawed due to imprecise categorical sampling. They show that employing 32-bit floating point precision in categorical sampling via the Gumbel trick induces an artificial lemperaturellowering effect, which results in lower (i.e.,
23

---

Published as a conference paper at COLM 2025
Method
PTB
WikiText
LM1B
Lambada
AG News
Pubmed
Arxiv
GPT-2
138.43
41.60
75.20
45.04
(WebTeed)
Transformer
82.05
25.75
51.25
51.28
52.09
49.01
41.73
eta, 2222
D3PM
200.82
75.16
138.92
93.47
(Austin 2021)
Plaid*
142.60
50.86
91.12
57.28
(Calrajani
2124)
MD4
102.26
35.90
68.10
48.43
2012)
SEDD Absorb
96.33
35.98
68.14
48,93
67.82
45.39
40.03
[Lou et al, 2024)
MDLM
90.96
33.22
64.94
48.29
62.78
43.13
37.89
2024)
BD3-LM L
96.81
31.31
60.88
50.03
61.67
42.52
39.20
(Arrisaa et 2025)
RADD-A-DCE
107.85
37.98
72.99
51.70
al
H-Hybrid (444B)
89.94
30.02
61.01
45.38
67.51
46.57
40.62
1y 0.01, Tea ALIGNED]
e-Hybrid (444B)
90.89
32.53
68.91
50.23
64.61
41.18
37.85
[e= 0.01, Taa ALIGNED]
7-Hybrid (444B)
100.88
37.48
71.51
56.57
70.69
43.06
38.83
[y= 0.01, Tpu SHIFTED]
H-Hybrid
90.67
31.73
73.71
50.03
68.27
41.49
37.89
1y= 0.01, Side ALIGNED]
7-Hybrid
95.32
38.94
70.49
48.18
67.32
44.23
42.78
[y= 0.01, 1444 ALIGNED]
7-Hybrid
90.74
35.24
62.64
51.21
69.62
41.46
37.13
[y= 0.01, Blokk ALIGNED]
H-Hybrid
95.22
32.64
63.68
44.75
62.18
42.01
37.33
[y= 0.01, Block SHIFTED]
Table 5: Zero-shot unconditional perplexity on seven benchmark datasets from Lou et al. (2024) and Sahoo et al. (2024) and Arriola et al. (2025). 'Reported He et al. (2022). #Reported in Arriola et al. (2025). *The GPT-2 numbers are reported for the checkpoint pretrained on Web'Text and are not direct comparison. All models are trained for 524B tokens unless otherwise stated. All diffusion models are upper bounds; the best diffusion value is bolded, the second best values is underscored.
seemingly better) generative perplexity at the expense of reduced entropy- key indicator of generation diversity. Their proposed remedy is to cast the values to 64-bit floating point precision.
To ensure a fair comparison of generative perplexity across baselines, we report both the flawed (32-bit) and the corrected (64-bit) perplexity values in Table 6. All the entries were resampled using full precision. Our results indicate that, irrespective of the artificial temperature effect, models consistently outperform all diffusion-based counterparts.
24

---

Published as a conference paper at COLM 2025
Method
FP32-PPL 4
FP64-PPL
SEDD-Absorb [14]
43.41
105.91
MDLM [21]
43.88
108.88
7-Hybrid [y= 0.05, Flat ALIGNED] (444B tokens)
39.53
89.05
7-Hybrid [y= 0.01, Taltt SHIFTED] (444B tokens)
48.08
110.60
7-Hybrid [y= 0.01, 4014// ALIGNED]
Block
40.48
85.01
7-Hybrid [>= 0.01, Slide
ALIGNED]
61.05
131.45
H-Hybrdd [y= 0.01,
Block
w=//e4 ALIGNED]
76.12
121.99
7-Hybrid [>= 0.01,
Block
SHIFTED]
53.89
111.73
Table 6: Generative perplexities (PPL; lower is better) on OWT. All models were trained for 524B tokens unless otherwise indicated. "FP32" denotes the flawed 32-bit sampling, whereas "FP64" corresponds to the corrected 64-bit precision values. All available models were resampled using their published weights.
E3 Inference Pareto frontier Results
In Table. we report the results we utilized to report the Pareto frontier plots in the main paper.
Higher p values 8,4)
Method
MAUVE (1) 8
Gen PPL. (L)
Entropy (?) p 8
=
p 8
p
p
SEDD
0.410
0.491
139.2
130.1
5.72
5.63
MDLM
0.921
0.959
128.5
116.4
5.63
5.58
y-Hybrid [y- 0.05, Fra ALIGNE)]
0.809
0.817
85.9
89.5
5.37
5.38
y-Hybrid [y- 0.1, Fra SHIFFED]
0.666
0.700
99.2
93.9
5.46
5.45
7-Hybrid ly= 0.01
Tidde ALLGNED]
0.775
0.788
107.3
106.0
5.53
5.53
e-Hybrid 0.01, Tear ALIGNED]
0.848
0.928
84.2
69.8
5.36
5.33
e-Hybrid 0.0,, s0d444 ALIGNED]
0.964
0.811
104.2
76.9
5.42
5.25
Lower p values (p
2,1)
Method
MAUVE (1)
Gen PPL. (L)
Entropy (1)
p 2
p == 2
p
p
f
SEDD
0.512
0.457
127.2
126.8
5.60
5.58
MDLM
0.947
0.897
115.8
108.8
5.61
5.60
7-Hybrid ly 0.05, tra ALINEE]
0.877
0.895
97.9
96.8
5.40
5.41
H-Hbri fy- 0.01, Fra SHIFTED]
0.728
0.744
96.4
93.9
5.45
5.47
H-Hybrid ly- 0.01, ride ALSONED]
0.553
0.819
105.5
100.2
5.46
5.41
e-Hybrid [r 001, Teur ALIGNED]
0.957
0.947
61.3
43.9
5.28
5.18
e-Hybrid (r 001, -4 ALIGNED]
0.813
0.916
71.7
59.1
5.38
5.25
Table 7: Sample quality of absorbing state discrete diffusion models. Upper block: higher values (p 8,4); Lower block: lower p values (p 2, 1).
F4 Effect of Varying 7
In this section, we examine the influence of the hyperparameter which modulates the contribution of Quniform the hybrid process and thereby allows the model to reexamine its predictions after unmasking token. As shown in Table 8, while the corrective influence of Quniform is essential, the value of a must remain relatively small. If a set too high, the model tends to simply reshuffle the tokens and the MASK token, effectively undermining the intended unmasking process.
Another perspective is that increasing a reduces the penalty associated with errors in the unmasking operation, thereby devaluing its corrective impact. Moreover, during the denoising process, each token is influenced not only by its own prediction but also by
25

---

Published as a conference paper at COLM 2025
Test Perplexiity
360
y Value
Figure 10: Ablation study illustrating the effect of varying the y (for 7-Hybrid variants) parameter on perplexity evaluated on the OWT test split at the 26B-token observation point. Lower perplexity values reflect improved model performance. Consistent with our previous observations, Y between 0.01 and 0.1 yields optimal performance.
the context provided by neighboring tokens. Consequently, if a token (e.g., token A) is mispredicted, the resulting change in the overall structure may leave little opportunity for subsequent correction.
Remarks The analyses presented above are mainly intuitive. Further empirical investigation is necessary to confirm.
Configuration
FP64-PPL /
y= 0.01
84.32
== 0.05
82.27
y= 0.1
85.15
== 0.4
91.48
= 0.8
90.99
Table 8: Generative perplexities (PPL; lower is better) on OWT. All the models are trained under ALIGNED configuration with Block for 524B tokens. We use the double precision, denoted as "FP64-PPL".
To further examine the effect of a, evaluate trained models' test perplexity (on OWT held out set) after processing 26B tokens under various configurations. As shown in Fig. 10, small values- apprxxaappo 0.01 and 0.1-yield the best performance.
26

---

Published as a conference paper at COLM 2025
E5 Effect of Varying
In this section, we examine the impact of varying p in our 7-Hybrid models using the p 236 Block hyperschedule. Recall that the parameter p is modified during the generation process and directly only influences the quality of the generated sequence. Consequently, we adopt generative perplexity as our evaluation metric. For completeness completeness-ande and to facilitate comparison with prior baselines-
report the generative perplexity computed under both double precision (FP64) and full precision (FP32), as illustrated in Table 9. As p decreases, the generation process becomes slower, thereby entering the "think hard" regime; in this regime, the model tends to produce higher-quality outputs at the cost of increased computational time.
Remarks 4. This trade-off is a key characteristic of diffusion models, which inherently possess flexible inductive bias that allows for varying degrees of commitment in generation. In contrast, autoregressive models are restricled to generating one token at a time. Moreover, under the flawed 32-bit sampling scheme, increasing the number of sampling sleps effectively reduces the artificial lemperature, thereby reducing the tokenwise entropy. In contrast, the tokenwise entropy remains mostly unaffected when the Gumbel trick is executed in double precision.
Configuration
FP32-PPL 4
FP64-PPL
p 16 (T 64)
75.57
100.11
p 8 (T 128)
64.90
95.38
p = 4 (T 256)
53.02
81.37
p = 2 (T 512)
44.15
79.3
P = (T = 1024)
40.48
85.01
P = 1 (T 2048)
33.09
87.31
P 1 (T 4096)
25.39
88.75
p (T 8192)
24.05
83.21
Table 9: Generative perplexities (PPL; lower is better) OWT. The same very model -w=/// ALIGNED]) has been used under different generation regimes. (y-Hybrid [y= 0.01, Block P value as well as equivalent T "diffusion steps" are used in the table. "FP32" denotes the flawed 32-bit sampling, whereas "FP64" corresponds to the corrected 64-bit precision values.
E6 Effect of Varying l/ in Adaptive Correction Sampler
In this section, we investigate the effect of the hyperparameter y in our proposed Adaptive Correction Sampler. Table 10 illustrates results for our e-variety family of models. As we increase the number of sampling steps (corresponding to decrease in p), our model tends to overcorrect when y is too large, which ultimately harms generation diversity-marker as "extreme values" in the table.. To mitigate this issue, we find that using smaller I values is beneficial. We further suggest that the optimal choice of is related to the e value used during training: the more inherently corrective the model is, the smaller the optimal ! should be.
E7 Inference Speed Up with Caching
In Table 11, we report the wallclock time required to generate eight samples for various models on a single NVIDIA A100 80 GB GPU. Owing to our custom attention masks, we were unable to leverage fast transformer kernels such Flash-Attention, which in turn results in slower sampling speeds. We note that future work may design specialized kemelss that are compatible with attention masks to accelerate inference.
27

---

Published as a conference paper at COLM 2025
Model Family
Sampler
MAUVE (f)
Gen
PPL. (1)
Entropy (*)
p=8
p=
p=2
p-1
p=8
p=-
p-2
p-i
p-8
p-d
p=2
p=1
Original Sampler
0.779
ACS (g) 0255
0.955
0.821
0.839
0.928
79,64
65.06
55.05
04
5.35
5.28
5.24
5.19
ACS (g 045)
0.846
0.98
0.865
0.936
105.94
93.91
83.83
77.16
5.46
3.48
5.31
5.29
ACS (g 0.11
0.848
0.928
0.957
0.947
84
69.84
61.35
43.98
5.36
5.33
5.28
5.18
eHybrid
ACS ( 0.001)
0.871
0.949
0.919
0.998
80.37
64.42
55.48
45.80
5.35
5.31
5.25
5.15
The AlOONE))
values
ACS (/) = 0.99)
0.618
0.582
0.754
0.508
28.05
24.94
22.01
21.26
4.98
4.86
4.66
4.41
ACS (g) 0.75)
0.883
0.770
0.599
0,299
28.51
24.57
23.49
22.23
5.03
4.90
4.79
4.56
ACS (y) 0.5)
0.740
0.987
0.839
0.837
31.96
68.42
53.48
45.80
5.31
5.25
5.21
Original Sampler
0.916
0.976
0.778
0.847
148.45
130.16
139 64
142.13
5.39
5.35
5.43
5.46
e-Hbbrid
(y 0.25)
ACS (/ 0.05)
0.964
0.811
0.813
0.916
104.26
76.98
71.77
59.15
5.42
5.25
5.38
5.25
ACS (g 0.11)
0.568
0.746
0.767
0.974
144.71
107.06
101.30
75.91
5.53
5.30
5.38
5.35
ACS by 0.001)
0.979
0.847
0.977
0.906
150. 41
150.32
139.66 66
114.12
5.48
3.54
5.55
3.48
Table 10: Sample quality of e-variant models using different samplers.
Our Block models are intrinsically faster since, during sampling, only the tokens corre- sponding to the Tsettled and Tactive components are fed into the transformer (e.g,, w, 2w, up to d tokens). Moreover, incorporating KV-caching would further boost the sampling speed.
Time(4)
SEDD
68
MDLM
59
caching
36
H-Hybrid ITiIn ALIGNED]
131
H-Hybri (1400144 ALIGNED]
79
KV-caching
38
Table 11: Wall clock time reported in seconds to generate 8 samples single NVIDIA A100 80 GB GPU. The same number of diffusion steps were utilized for all the models.
F8 Training Throughput Evaluation
Table compares training throughput of our e-Hybrid [E = 0.01, Th'= d/4, ALIGNED] in both non-efficient and efficient settings, alongside MDLM (Sahoo et al., 2024), SEDD (Lou et al., 2024), and an AR baseline, on a single NVIDIA A100 80 GB GPU, We use the largest batch size divisible by two (64 for most methods; 32 for the efficient variant due to its "double-length context" strategy) and set the window size to w = d/4 (smaller w further amplifies the training efficiency gains).
Block
By leveraging our efficient attention masks and doubling the context length via concatena- tion of clean and noisy sequences, the efficient variant achieves nearly twice the active-token throughput of its non-efficient counterpart. This demonstrates that our hyperschedule variants incur substantially lower training cost while matching the effective throughput of other diffusion baselines.
28

---

Published as a conference paper at COLM 2025
Method
Throughput (token/ms)
Effective Throughpul (active token/ms)
e-Hybrid [e = 0.01, Tia //4 ALIGNED] (Non-Efficient)
100.31
25.07
e-Hybrid [e = 0.01, Tiaa //4, ALIGNED] (Eificient)
88.04
44.02
MDLM (Sahoo al., 2024)
97.64
54.42
SEDD (Lou et al., 2024)
99.29
56.01
AR
127.65
127.65
Table 12: Training throughput comparison single NVIDIA A100-80 GB GPU. Effective throughput counts only the active tokens processed by the transformer.
29

---

Published as a conference paper at COLM 2025
G
Samples
The following is a random selection of a few samples from our 3 diffusion language model families.
G.1
Undonditonal Sample
Tweetty-oomehhng host will soon have a new spot main commercial-run drugmakers' clinic in his home state of New Mexico.
Chris Brewster is convinced that could lead to the world market for Type II's genetically modified metallomics drugs from certain not-tou-fae fot-tou--amous contenders elsewhere in medical marijuana industry who have partnered up with the cannabis company Little Tryy Farms to roll out FDA approval for the widely used CRISPR-fre R-fce Jacoby-Glynda hybrid product.
At least three manufacturers have been selected for the enterprising position, which also includes Schluge Therapeutics, Porsale, Mosaic, Marin and IE Pharma. 'The total number of drug prices in America in 2013 had the highest in any single year''
Prasepalan, Dr. Dave Rouzean's 's permission to begin selling medicine has long been a crusader for medical marijuana and these days, Brewster and Dr. Rimelter are taking their criminal flair to the next level.
Rouxau and his fellow New Mexico drug regulatory experts will be making their cases in October, which would complete a 90-page interface presenting the medical world, checks and balances and system reviews for each company's patented drugs involved overall delivery by patient
providers and regulaturs.
"We have seen this is a breakthrough that patents have exploded," said Johnson, who represents Prasepalan's 's chiropractor, Adderall, and his partner, Rick Kelbyck, who specializes pharmaceutical treatment and drug treatment products Wellesley and Kroll Laboratories, among other giants, in Austin.
"We do have Symantec to make the prescription of Nathur Sherogood'' 's Marin-REX Therapeutics srppeuttis-Lunette a New Mexico-basee machine shipped a billion man on the verge of retirement must fear the official automated deportation system, fair trade prosecutor Rafaela Gonzalez thinks.
make," and an additional
25 miles away from Prasepalan, Hansen will charge direct fee
worth of medicine to every business the United States and Canada.j endoftext No
D.C.a remote northern island nation governed by rigid immigration rules with 90 million illegal immigrants, 85 million people without adequate food, and I million illegal, unauthorized immigrants that will soon surge around the United States?
Gonzalez got the idea from journalist at Caution Center paper to mark the anniversary of the United Nation's 's signing the massive Comprehensive Economic and Policy Agreement (COP21) 1995. Typically it's minor-level agreement that provides an objective, stable key equation wherein every country, every country can control both the economy and country's external affairs by changing its immigration and custorns policy. Democracy and separation the people law enforcement.
March this year, Gonzalez released a new chapter in career-spanning memoir Magic of Arrows: The Filibuster of Gang Targets and Promise of Perfection. Modeled after projection of long-term economic effects from government handouts to its leaflet, her book is dated May 21-24: Four days before America secedes from formally abolishing the six-nation customs union, America's ex-military leaders escalating pro-secularism.
force America break the military response
first chapter rated as one the lists of U.S. Category A cuuntries advantage," " with a stylized listing of 22 whose influence has been ascendant since the dawn of capitalism. As Gonzalez points out, the list displays the position bodies such as United States Samoa and Croatia, whose remotes are sprinkled with their own market share and have expressed renewed interest in reaching grouping that then can take tuugher measures against those cumpetiturs.
purposes, code "unfair
countries like the BRICS, 33-nation
"There tremendous opportunities" " Gonzalez writes. "From Washington perspective, it's easy to players- the world's 's poorest, country's 's most vulnerable-a aleeddy feeling the need for more calculated and conscientious policy in regard smuggling drugs and other illegal trade into midst."
The Pan American Conference of Scholars announced that more than 3,000 countries sent almost 1,000 requests for tickets, and more than 17,000 applications for tickets have been denied since March of this year. Last May, the Selena- Ranich Trade Union Confederation the country's 's only market, for which agribusiness corporations receive billions dollars grants receeved a "delusion team response" of 6,000 applicants in international training centres.
D.C. is not alone:
Robert Dellinger, deputy directur for the International Union Red Cross Office for Latin American Mission who awaits decision the file
Figure 11: Unconditional samples generated by y-Hybrid l 0.01, Flatt ALIGNED] trained on OWT dataset.
30

---

Published as a conference paper at COLM 2025
SALT LAKE CITY - The Utah Legislature committee chairwoman says problem persuade the state to legalize drug behavior.
Wroeff Bugler reports for Salt Lake Tribune Jan. 7, 2014 The bill's sponsor, Rep. Mike Tate, chairman of the Utah Business Improvement Association, says they would legalize the use of drugs "in variety of policy areas." A Republican lawmaker called them say ban on the use of marijuana is the best way to tackle the problems of drug rights. (Photo:
Cmdr.. Roger Sultanousi)
According a lawmakers 16-30 mujooiity the bill has the support of animal rights groups and equal repneentation. The bill would allow adult humans from nearly all biological categories to get arrested they haven't almady committed crimes and other vulnerable individuals who could be facing significant financial penalties. Utah's enactur percent has only 40 percent all parental consent for adult; that's not far as human children go, but nearly half teenagers are aborted at 18 months of age and illegal background checks don't exist for any other concern.
"Now focus on the serious crime aspect the bill," spokeswoman Heather Chien said at the time. "Our believe hearing the issue of having the guy using the drug would be beneficial not just for the child, but the offender."
comes as more and more farmers and livestock owners are scheduling change reviews. The Senate voted on Thursday to the pass or
either let best-information request the state's & enforgement motion, 54-12, to allow the vote by vote of more than two to one. Memo newsletter delivered to your inbox We're re sorry,
freedom of information request prioritized specifically by
something went wrong Get the pulse of Arizona Local news, in-depth state coverage and what all means for you Please try again soon, contact Customer Service 1-800-332-6733 Delivery: Mon-Fri Invalid email address Thank you! You'de re almost signed up for AZ Memo Keep eye out for email confirmn your newsletter registration. More newsletters
NEWSLETTERS
In answer to a question posed by friend, lobbyist, and political activist Fd Priderout, this hill ultimately failed. what will happen in the future, resolution Friday would call for the of fewer than percent of all adult diapers in a single year for the state's 's menial and surgical courler services.
Though it remains to
The measure was alsu tamper-resissant rabble fruit without trans fats.
"Despite some objections we need to make the safety mechanism work in future we don't believe is most effective, and that's just who they are as I've been trying to get our labor. We have to use combination of dog tags and earplugs, su that conditions," said Rep. David Cale-Dero.
improve
Copyright 2015 The Salt Lake Tribune; peeee--enndol -e--eenfoottteee-- Saattexxt(CAA.ARRRA CCA.GARY - It was the two-day drive north through the republic Ontario and the provinces
the past three months that signs giving freemen extraordinary tourism in the province appear to have slipped away, leaving many of the seasonal magnetes at bottom. Really?
Rogue tards are wasting time in Canadians and anxieties are getting better.
Conspiracists have become" "emerged," as experts characterize them, in new sense,, either as cungenial naivetés- quasi- religious adventurous sluts rather than numb, territorial isolates like northern Canadians who already born the same spot, or simply as less flexible and maneuvering players that can deliver on their hard conomic-mindedl defined tenets. Certain politicians are reading the Pearls Bible and happy that they lack cable, satellite TV otherwise space-tine time wurthy support in the province's 'an's often futile re-election campaigns. Yet many Canadians see the Alberta premier mure philosophical than conservative.
Some have joined the "unreliable" view that there another side to the tale. Amplification over the provincial election left Cayote Falls and Tire Centre, Conservative and Labor alike, shared the view month ago the periphery of the serenity of the Alberta electorate that the drums driving the Alberta govermmen's s provincial campaign. But the NDP MLA was also shaking that opinion with equal dismay when Ford's demand that union representation be halted sympathetic "no." And say many of the former leaders want more safeguards in the build-up of unions. One, minister Clifton Sanderson, acknowledged his former colleagues had brought with them new rules, regulations so on could be in their path. But, remains matter of interpretation.
the federal government for the next
years yielded somewhat
"We welcume that a promise transparency and clarity- that
Figure 12: Unconditional samples generated by y-Hybrid [y= 0.01, - ALIGNED]
- 40=4/4
trained on OWT dataset.
31

---

Published as a conference paper at COLM 2025
White House chief of staff Reince Priebus traveled to Washington, D.C., Wednesday to be briefed by Vice President Mike Pence and other top advisers about the nature of the crisis in Ukraine and its potential impact on the United States, The new official, who experts and other world leaders as lensions peak over President Donald Trump's 's handling of the refugee crisis and its potential for conflict with Russia. Jared Kushner's 's efforts pump up the Dubrovnik deal have raised concerns that it could backfire. Kushner Trump accused of colluding with the Kremlin, accurding to a New York Times report
authorized speak publicly by the White House, with senior American Russian policy
of trip to Moscow to meet with Ukrainian leader Mykola Azarov. The trip comes
White House officials said that press secretary Sean Spicer called Kushner "little stranger" when asked if he was familiar with the concerns swirling around the discussions, according to an account published by NBC News.
"The president-elect agreed be briefed by my team on Russian meddling our election process, Press Secretary Sean Spicer told NBC News in telephone call, adding that he didn't have an authority to disclose details of the discussions. Versions the meeting have drawn criticism from U.S. politicians, military officials and human rights advocates.
In separate report, former acting FBI Director Andrew McCabe said he not aware of any conversations with any Russian officials. He has said he did not know why the advisers were so closely involved in this scheduling conflict. "I very preliminary, telephone calls with colleagues both within conversations with any of them," McCabe said of the meetings, which included senior officials and a White House official. meetings involve joint efforts by the White House and individual individuals with the understanding that such meeting would not occur, McCabe said, adding that no other member of the senior staff was involved.
department and overseas, and have not had
"Please, contact the people you know with whom you have the knowledge
this room. Basically, I am asking you to do the tup officials associated with your Department of State," " McCabe said.
the wurk of reaching out each
"Since this is tense, I have trying to contact both the President and his staff and conduct the interview," McCabe said.
Guidance
White House advisers held series of meetings with top officials of the Department of State and its federal counterparts. However, the White House's ss meetings were themselves influenced by events tu help the new administration out greater involvement in an escalating crisis that has killed more than 200,000 lives since last August.
In speeches delivered to the nation, Trump touted Friday's ss meeting on June 20 as another reminder of the importance of sanctions against Russia. Trump said Russia's nuclea weapons "are insalting weak countries" and hoped "for normal" after a nosedive in the U.S. elections.
"We have a great relationship with Russia, and continue look forward strengthening our relationship with them," " said Trump.
"warm" Russia policy has deepened the already long-standing divisions over Obama administration, and Putin's recent ally Russia, who built a buffer zone along the Black Sea in 2013, ordered an invasion in 2014 of eastern Ukraine. White House spokesman Scun Spicer said he was "careful" with ties with R endoftext- mondoftet--On Tuesday night, LeBron often from the Hall Fame. Cleveland's most respected individual, who most known for his love of basketball, was getting a heart-to-heart with one of the most prestigious prospects in pro basketball history, sitting in the stands the Knicks' first guard of the year. The official photo the Knicks' current superstar, Geurge Conte, was taken by the lungtime reality television legend Wade Beggs
James received news he going
We demand James get a heart. Exactly what deserves hard guess. People to point a man taken
who
never weeks ago was the kind of person champion of truth would want to meet or make close friends with without his inner purpose. After all, people feel very similar they haven't I seen Wade Boggs' latest film, Clump, in which he takes matters his hands and seizes his wife the same wife that never had child up for
This event has the potential to be sour revenge for the Kevin Durant coronation that began a week ago a winner's 's assuredly standard season that had little impact on James' looming future. But that's the here, either; the only one he is saying is that the uninspiring
Figure 13: Unconditional samples generated by y-Hybrid ly= 0.01, w/=d/64 SHIFTED]
Block
trained on OWT dataset.
32

---

Published as a conference paper at COLM 2025
be violent might be protected for of the reasons he challenged.
Tsherberg is talking about former Republican Rep. Joe Burns who, in many ways tried to do better politically, canceled my appearance 2012. She said that when the Republican president and were playing with one other's decisions, it the Chechen deal.
"Every time think, there's nothing in issue," she said. "There comes time when I faced with reality in very difficult way. It's not something want to put the side of the my team past me."
During O'Rillly Factor in December 2011, when dealt with most of my viewers, watched the Obama Britain show again.
"I was watching very, very closely, knowing g that there
opportunity, I knew didn't want to be able to be ahle
help the Americans. I couldn't help that," I said.
Cack Obama also said several people without any authority, just due diligence, have commented his plan to return
Chechnistan and Russia. T. Obama said he's s not against Putin, and referred to himself as Putin- PM, who said he's taking
"America's s values forward."
"I think that's s turning point," " Lea said in 2008 just before the start of my comments, referring to Lea's parent party, Moscow's 's Communist Partty in 1997, according to The Associated Press.
Asked whether Iza believes the economy and its security should run well into the Chechen agreement, and the leader ISIS was this month, Iza said that Trump is far to go, more conservative agenda.
"He can't a politician
he's not fough guy who doesn't take person, she said. "My people have respect too. My children are not civil andRoll but they do not [acc believe me]."
After speaking at Trump's Palm Beach, a bus stop in Texas on Sunday, his attorney, Republican Frank Agoodz, said neither he nor Les accused me of arguing at the "Maxby Restaurant" in Texas.
"Williams, want them to help, I don't want them to be able to help me," Iza was recorded as saying.
"He said, YYu have opportunity in doing this and then you have do this, you can do this; it's starting, that's getting the right people to do it, and he said, They'r re going and did, yeah."" she tells The Associated Press on Friday ev evening. Sokherberg had not priur known lea's 's release- since his public heaaring, he husa't it been expecting anything since then.
But Iza said this is situation that will cause "Great pain" for all his family:
"It's important to my well-being, to care for others, about peace and just to be strong," she said. "As long as I don't. 't won't. I think won't, I will have apologies."
Their representatives before they gather on Saturday.
"It is man with the very strong side things, that has long way go," she said. "He's been singled out by the President.
He has turned even more extreme off the map and is being led by the Republican party and the economy that are making debt-laden economy out
dream.
"The public is mind-set, and his decision to speak to the public was a mind-set."
In a CNN videu excerpt that includes her closing remarks, the then-U.S. Senate wasift to safety to stand in front of Republican House-time House Speaker and former White House special counsel Paul Pro Romanoff,; -endofftt-eddttxtt---ooo &New Villa in the midst of not making cause in Sunday's clash at Villa, who started toestem to the title last season.
With Villa attacking you no certainly doubt it, if was the second title they had won the most recent five y years, over the likes of Mar. Canado, Rubbie, Fernandes and CFOi.
Villa
an answer that battle over Canado the help
Vy's and in speeding up the relationship with the
Portuguese forward in ways he hasn't shown in his past season.
West Villa turned their attention to defence in with Canado, who represented the England national team even last season after he became the first ever score four goals.
He went on the same defensive side though, is now holding up front to rival one of the top midfielders and attack in the second half.
West Villa's 's second-half form last season huwever, has been done again in favour to substitute Scott-Fidande Jr, who later moved to Paris City on Wednesday.
John Gomez and João Aguero have moved to
Figure 14: Unconditional samples generated by e-Hybrid [e= Trlat ALIGNED] trained on OWT dataset.
G.2 Conditional Samples
The cat sat on the mat and looked up, staring at the ceiling as if something interesting were up there. It made no sound, but occasionally flicked its tail back and forth. "Is he always like this?" asked.
She smiled and nodded. "Pretty much. Sometimes he stares out the window too."
glanced me, seemed to consider something deeply philosophical for moment, and then resumed staring upward.
"Do you think sees things we don't?" 1?" asked, half-joking.
"Probably," she replied, laughing softly "Cats always seem to have foot in another world, don't they?" I chuckled, taking sip from my coffee. "Mayhe we should take notes."
"Maybe should," she said, still smiling. "We might learn something." "
The cat yawned lazily, stretched, and settled even more comfortably onto the mat, clearly deciding that whatever secrets the universe held could wait little longer.
Figure 15: Conditional (conditioned on first 6 tokens) samples generated by e-Hybrid [e= 0.01, TFlat ALIGNED] trained on OWT dataset.
33


Reviews:
[{"review_id": "WR3wrEaOEa", "reviewer": "Reviewer_WiDd", "summary": "This paper proposes noise schedules (hyperschedules) that\n* interpolate AR and Masked Diffusion Models (MDMs). \n* interpolate Uniform State and Masked Diffusion Models.\nIn doing so, they achieve competitive performance on language modeling benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although papers like Block Diffusion [1] interpolate between autoregressive models (AR) and masked discrete diffusion models (MDMs), combining uniform state diffusion models (USDMs) with MDMs is a novel contribution. A key limitation of MDMs is that they cannot revise their predictions during the reverse sampling process. Integrating USDMs with MDMs addresses this limitation, enabling iterative correction and making it a significant advancement for the community.\n\n\n\n\n[1] Arriola et al., 2025 \"Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models\"", "reasons_to_reject": "The main reason I’m currently inclined to reject this paper is the lack of a principled ELBO formulation for their hyperschedules. It appears that the authors evaluate their models using the training objectives described in Appendix `B.2`. As a result, the perplexity (PPL) values reported in `Tables 1 and 2` may be incorrect or misleading.\n\n\nDeriving the ELBO for their hyperschedules should be straightforward. Specifically, the authors would need to use `Equation 10` of [2], and:\n* Reformulate their state transition matrices as rate matrices as per `Appendix C.1` in [1], and\n* Reformulate the score function using the mean parameterization approach, as outlined in `Appendix C.5`.\n\n\n**I would be happy to increase my score to 7** if the authors are able to adequately address this concern.\n\n[1] Sahoo et al., 2024 \"Simple and Effective Masked Diffusion Language Models\"\n\n[2] Lou et al., 2024 \"Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution\"", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCjx7Czuxx", "reviewer": "Reviewer_NmzT", "summary": "This paper unifies autoregressive and diffusion-based sequence generation by introducing hyperschedules, which assign different noise levels to each token position, encompassing both paradigms as special cases. It proposes hybrid noising processes that blend absorbing and uniform noise to allow correction of past mistakes during generation. There is an adaptive correction sampler, that further enhances output quality. The approach supports efficient inference via KV-caching and achieves good results on multiple benchmarks, narrowing the gap between diffusion and autoregressive models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposed a novel formulation that unifies autoregressive and diffusion-based sequence generation, which may contribute a fresh perspective.\n2. The proposed hybrid approach interpolates between absorbing and uniform noising. It allows the model to correct past token errors during generation.\n3. The empirical results are strong\n4. The experiments and analysis is well-structured.", "reasons_to_reject": "1. Lack of training cost analysis compared to AR models and diffusion models\n2. The experiments are mainly done on small models. I'm not sure when we scale up the mode, whether the advantage of this method will still be there. (I understand if the authors don't have resources to train models)\n3. For hybrid noising part, the theoretical justification is kind of limited.\n4. Lack of some related works, for example, The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control, which is related to this work.", "questions_to_authors": "1. You design attention masks to support KV-caching for efficient inference, but the training cost of your hybrid models is not discussed. Could you provide more details about this?\n2.  All experiments use relatively small transformer models (110M). Have you attempted scaling your method to larger models (e.g., 1B+)? If not, what are the expected challenges or limitations when applying at that scale?\n3. The ACS introduces a new hyperparameter η. How sensitive is model performance to the choice of η? Is ACS stable across different datasets and tasks, or does it require careful tuning for each setup?", "extra_scores": {}}, {"review_id": "uBLdB1VtpQ", "reviewer": "Reviewer_XPkA", "summary": "The paper introduces two key innovations: hyper-schedules and hybrid token-wise noising processes. Hyper-schedules assign distinct noise schedules to individual token positions, generalizing both AR models (e.g., GPT) and conventional diffusion models (e.g., SEDD, MDLM) as special cases. Hybrid noising processes interpolate between absorbing and uniform processes, enabling models to correct past errors. Additionally, the paper proposes a novel inference algorithm, Adaptive Correction Sampler (ACS), which leverages the features of hybrid processes. To support efficient training and inference, the authors design attention masks compatible with KV-caching. The methods achieve state-of-the-art perplexity and generate diverse, high-quality sequences across standard benchmarks, demonstrating a promising path for autoregressive diffusion-based sequence generation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think overall this paper is theoretically sound.  It reveals the fundamental connection between autoregressive and diffusion models and provides new insights for sequence generation model research and expands the design space for models.\nThe introduction of hyper-schedules and hybrid noising processes offers new technical approaches for sequence generation. Hyper-schedules enable fine-grained control over noise schedules for different token positions, while hybrid noising processes combine the advantages of absorbing and uniform processes, enhancing model performance and flexibility.", "reasons_to_reject": "I think the missing part is the analysis on the training and inference overhead. Although the paper proposes attention masks compatible with KV-caching to improve efficiency, the training and inference processes of the model remain relatively complex and resource-intensive. So I would like to see more results and explanations in this direction.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04VmPKg9Vc", "reviewer": "Reviewer_AaDn", "summary": "This paper shows how auto-regressive and diffusion-based methods for sequence generation can be unified conceptually through the notion of hyperschedules. In addition to the conceptual contribution, the paper introduces two hybrid noising processes, which interpolate between absorbing and uniform processes, enabling the model to fix past mistakes. Experiments show that the hybrid processes, together with a novel inference algorithm, can achieve lower perplexity than state-of-the-art diffusion-based models (while still not in par with auto-regressive models).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel conceptual framework unifying auto-regressive and diffusion-based sequence generation.\n- Technical innovations in noising processes and inference algorithms.\n- Promising experimental results.", "reasons_to_reject": "- I see no obvious reason to reject the paper, but I think the evaluation of quality-diversity trade-offs could be explained better.", "questions_to_authors": "I would like to know more about the evaluation of quality-diversity trade-offs, in particular whether you assume that perplexity is a measure of quality, since there are several studies showing that, while perplexity is a good measure of language modeling performance, it is a poor measure of text generation quality. See, for example, Carlsson et al. (2024) Branch-GAN: Improving Text Generation with (not so) Large Language Models. ICLR 2024.\n\nTypos and such:\n- Page 2: abstracting-out -> abstracting out\n- Page 3 Yang et al. (2023) -> (Yang et al., 2023)", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has universally strong reviews. Reviewer WiDd brought up concerns on fair test comparisons, and would revise the score 7 to if resolved. I believe their concerns were resolved by the author rebuttal and appendix C.1 of the original paper.

A couple comments that do not affect evaluation:
1. Nit: Position-dependent schedule would be a more precise term than hyperschedule. Hyperschedule does sound better.
2. Question: I am curious what a product process would look like. For example, taking the product of uniform diffusion as well as masking via the addition of a per-position settled bit. Perhaps this would be more useful with learned hyperschedules.
3. Nit: The set of variables is larger than necessary. For example, $y$ is not necessary as latent diffusion is not considered in this paper. In general, it's nice to minimize notation.
Decision: Accept