Anchored Diffusion Language Model
Authors: Litu Rout
Constantine Caramanis
Sanjay Shakkottai
Venue: NeurIPS 2025 poster ()
Keywords: discrete diffusion
diffusion language models
large-language models
generative modeling
planning
reasoning
chain-of-thought
Primary area: deep_learning
TL;DR: We propose the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that generates an important token mixture which guides the prediction of missing likelihoods, resulting in better likelihood modeling and generated text quality.

Abstract: Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both *likelihood modeling* and *generated text quality*. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the *Anchored Diffusion Language Model (ADLM)*, a novel two-stage framework that first predicts distributions over important tokens via an anchor network, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4\% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-art zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thought approaches. Please see our project page: [anchored-diffusion-llm.github.io](https://anchored-diffusion-llm.github.io/) for code and demo.

Anchored Diffusion Language Model
Litu Rout Constantine Caramanis
Sanjay Shakkottai
The University of Texas at Austin
(litu.rout, constantine, sanjay. shakkottaa)
Abstract
Diffusion Language Models (DLMs) promise parallel generation and bidirectional context, yet they underperform autoregressive (AR) models in both likelihood modeling and generated text quality. We identify that this performance gap arises when important tokens (e.g., key words or low-frequency words that anchor a sentence) are masked early in the forward process, limiting contextual information for accurate reconstruction. To address this, we introduce the Anchored Diffussion Language Model (ADLM), a novel two-stage framework that first predicts distribu- tions important tokens via an anchor network, and then predicts the likelihoods missing tokens conditioned on the anchored predictions. ADLM significantly improves test perplexity LM1B and WeeTeet achieving up 25.4% gains over prior DLMs, and narrows the gap with strong AR baselines. It also achieves state-of-the-ane performance in zero-shot generalization across seven benchmarks and surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model. Theoretically, we derive an Anchored Negative Evidence Lower Bound (ANELBO) objective and show that anchoring improves sample complexity and likelihood modeling. Beyond diffusion, anchoring boosts performance in AR models and enhances reasoning in math and logic tasks, outperforming existing chain-of-thhe of-thougght approaches. Please see our w.w.gluub...o//////mmmmoo for code and demo.
project page: https://anchored-dfffeee
Introduction
Large autoregressive language models (LLMs) have achieved remarkable success in next-token prediction, powering high quality text generation and emergent reasoning capabilities in Al systems. By generating tokens sequentially, autoregressive (AR) models like GPT (Brown et al., 2020), Gemini Prean-Gemini et al., 2023). LLaMA (Touvron al., 2023), and Claude (Anthropic, 2024) condition on growing prefix and excel at fitting the distribution of the next token. However, their sequential generation process makes it challenging solve complex reasoning tasks, since the model does not see the entire sequence all at once.
An alternative paradigm has recently emerged in the form of Diffusion Language Models (DLMs), which perform masked-token prediction via iterative refinement. Inspired by diffusion models for continuous data (Sohl-Dickstein et al., 2015), these approaches corrupt text (e.g., by masking (Lou et al., 2024; Shi et al., 2024; Sahoo et al., 2024; Ou et al., 2025) or random flipping (Austin et al., 2021; Lou al., 2024; Liu al., 2025)) and train model to denoise reconstruct the original sequence over multiple steps (Austin et al., 2021; Li et al., 2022a: Lou et al., 2024; Sahoo et al., 2024; Shi al., 2024; Ou et al., 2025). DLMs generate the entire sequence in parallel, allowing bidirectional attention for better context and potential gains in controllable generation. complex reasoning, and fast sampling. Despite their promise, masked diffusion models still lag behind AR models in modeling the likelihood of missing tokens and generated text quality. Even with modern training improvements, DLMs often achieve worse (higher) perplexity than AR transformers on standard benchmarks (Lou et al., 2024; Sahoo al., 2024; Shi et al., 2024; Arriola et al., 2025).
We identify key limitation in existing DLMs: when important tokens (e.g., low-frequency or semantically important words) are masked early in the forward process, the model lacks sufficient context to accurately reconstruct the original sequence. To address this, we propose the Anchored Diffusion Language Model (ADLM)(33) which modifies the noising process such that important
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Anchored Diffusion Language Model (ADLM), During inference, ADLM unmasks over two stages. (1) The Anchor Transformer outputs a sequence of anchor predictions; for any masked token, the corresponding anchor prediction is distri- bution with support skewed towards over important tokens. (2) The Denoising Transformer conditions on these anchor predictions to unmask the remaining tokens. Anchor predictions help unmask important tokens early by reducing mnchor-coniitonal entropy- guiding toward more accurate denoising.
cit
1s
playing
with
mask
dog
349
318
326
50257
Denoising Transformer (with bidirectional attention)
Anchor Transformer (with bidirectional attention)
64
50257
318
50257
326
502577
50257
mask
ist
mask
with
mask
mask
tokens remain unmasked longer in the forward cerress--quivalenttte they are revealed earlier in the reverse process. We theoretically show that ADLM reveals important tokens early during unmasking and improves sample complexity within directed graphical models ($4).
As illustrated in Figure 1, ADLM introduces a two-level anchored denoising process, A key idea is that of anchors- tokens whose inclusion as conditioning variables yields a substantial reduction in the conditional entropy of the remaining tokens (e.g., knowing "cat" and 'dog" reduces the entropy at position 4 and 6). the first stage, an Anchor Transformer outputs a sequence of anchor predictions, where each anchor prediction for masked tokens is a mixture distribution that is skewed toward anchor tokens. In the second stage, Denoising Transformer conditions on these soft anchor predictions to unmask the next set of tokens. We derive an Anchored Negative Evidence Lower Bound (A (ANELBO) objective to jointly train both networks within a single end and-to-end differentiable framework. For anchoring, the loss is imposed only on the important (aka anchor) tokens. As result, three things occur during inference (see Figure 1): (1) anchored predictions for already unmasked tokens peak around their true values (e.g.. 'a" in position 1, 'is' in position 3 and 'with' in position 5); (2) anchor predictions corresponding to important tokens have a skewed distribution focusing on important words (e.g., 'cat" and 'dog" in positions 2 and 7); (3) anchor predictions for all other tokens yield fairly flat, high entropy distributions. The denoiser's 's bidirectional attention across all anchor predictions ensures that these predictions influence the denoising of all unmasked tokens.
Notably, the denoiser's prediction for 'playing' now benefits from anchors like 'cat' and 'dog' (masked tokens in input Z, but whose likelihoods are high in the anchor prediction sequence); this is unlike standard masked DLMS (Sahoo et al., 2024; Shi et al., 2024) where 'playing' relies only on already unmasked tokens ("a', 'is', 'with'). At the output, positions corresponding to anchor tokens have higher likelihood for anchor words, increasing their chance of being decoded earlier. Thus, during inference, the sequence of anchor predictions creates latent reasoning space consisting of a sparse collection of anchors through which ADLM infers the full sequence more coherently and efficiently. Interestingly, these ideas extend to AR models where we demonstrate that anchoring enables a "look-ahead" planning behavior, which is in contrast to the commonly observed left-to-right decoding bias of standard chain-of -oftuuggttthhtt (CoT) reasoning (see Appendix C.2.5 for details).
(OOD) performance in generative modeling ($5.1), and also enhances the reasoning capabilities of AR models ($5.2). On the LM1B (Chelba et al., 2013) benchmark, ADLM achieves test perplexity improvement of 9.54% over MDLM (Sahoo et al., 2024) and 25.4% over SEDD (Lou et al., 2024). On OpenWebText (OWT) (Gokaslan & Cohen, 2019), ADLM reaches a perplexity of 20. 14 with 524B tokens, outper- forming MDLM by 12.3%, matching the hybrid (A++Diffusion) baseline BD3LM (L' 4) (Arriola et al., 2025), which achieves 20.73. In terms of generation quality, ADLM achieves a GPT-2 Large perplexity of 26.8 -surpassing MDLM by 39% and SEDD by 48%- is thus the first diffusion language model to exceed AR in MAUVE score (measures human- text quality) (Pillutla et al., 2021) using the remasking sampler. Furthermore, ADLM achieves state-of-the the-thh--att zero-shot perplexi- ties on out of 7 language modeling benchmarks and outperforms AR baselines on long-context and domain-specific datasets such as Lambada, PubMed, and ArXiv, demonstrating its strong language understanding and generalization capabilities. We also show that when anchoring is integrated into an AR model, it shows stronger logical consistency and planning capabilities in text generation and complex reasoning at GPT-2 scale.
Anchoring significantly improves both in-distribution and nut-of-distribuue
Contributions. This paper makes the following contributions:
We propose ADLM, a novel two-stage diffusion language model that improves the prediction of masked tokens through anchor-guided denoising ($3). We derive an anchored evidence lower

---

bound to train ADLM in an end-to-end fashion, proving improved sample complexity and better likelihood modeling in a DAG model ($4).
ADLM achieves lower test perplexities than prior DL.ss on LMIB and OWT, narrowing the gap with AR models ($5). Anchoring generalizes better zero-shot evaluation, improving perplexity on OOD tasks such as PubMed and ArXiv, outperforming both MDLM and AR baselines ($5.1).
We demonstrate the benefits of anchoring using two different samplers: (a) locked-in (Sahoo et al., 2024) and (b) remasking (Wang et al., 2025a) samplers. With remasking sampler, ADLM outperforms AR models in human-like text generation measured by MAUVE score ($5.1).
Beyond diffusion, we integrate our anchoring mechanism into AR models, which leads to novel reasoner that supplements conventional chain-of-thouugh Our results show improvements in next- token prediction and supervised fine-tuning Math (GSM8K (Cobbe al., 2021)) and logical reasoning (ProntoQA (Saparov & He, 2023) and ProsQA (Hao et al., 2024) ($5.2) tasks.
2
Background
Consider discrete state space S = VL, where == {I,... KK 1, K denotes the set of discrete
alphabets or tokens augmented with an extra (K) letter representing dummy token called 'mask'. Further, L is the dimension of each sequence x (x', ....... ,x²) E S for x' E V, E [L]. We represent each sequence as collection of one-hot encodings as: x (x xx...... ,,,,,, x"), where Σ = 1, أن 2 0 and x [x(x 1. In case of a mask, we denote the corresponding one-hot encoded vector as m = (0, 0, .... 0, 1)". Let X denote a random variable taking values in S. Given finite set of samples from unknown data distribution q(+) supported on S, the objective in generative modeling is to generate new samples from this distribution.
2.1 Auto-Regressive Models
Autoregressive models encompass widely used approaches in discrete generative modeling. These methods typically train a neural network to approximate the distribution of the next token conditioned on all previous tokens. This corresponds to the causal factorization of the joint data distribution g(-)
by modeling the causal relationships in X ~ q as follows (Jelinek, 1980; Bengio al., 2003): g(x)
g(x') ) (xx|[x]x* where x 1::f11 x² x²,
,x'=1, A neural network parameterized by pa is trained to approximate these factors. The training objective for the neural network is designed
x?.....
to maximize the likelihood of a given finite set of sequences, which is equivalent to minimizing
the negative ligeliike:hood:: (f) -xx~g (o =-Ex~g Σt log po (X'
This objective encourages the model to learn the conditional distributions, enabling autoregressive sampling from the learned distribution (-).).
2.2 Diffusion Language Models
Let T represent the finite number of time steps used in a diffusion model. We denote by t(1) = / and s(i) = اا where E {1, 2, .... T). For brevity, we drop the index from I and s. In D3PM (Austin al., 2021), the conditional of the forward process at time is given by
1.
<(zx)x) II [(221)x) 1=1
o(z)1x) Cat (z'; axx' + (1 a1-ay a.)m),
LE (1,2.....444 ,4, (I)
which has a transition probability g(z/(zz) = Cat(z);
an 2 + (1 m) (see Appendix A.2 for a.
details). The masking schedule a, E [0, 1] is predefined as a monotonically decreasing function of t with on- and an 0. The corresponding reverse posterior becomes:
(Cat)(// Cat uii co-axx + 1-20 m)
(Cat(zfiz))
z + m
((2112) x²)
z = m.
This reverse posterior is useful because as we see in (3), it helps parameterize the generative model to have a similar form. The reverse process of D3PM (Austin et al., 2021) defines 0 -8-paameterrzze joint distribution over sequences given by Po xx, 20.1). It follows Markovian structure with transition I1 po(z)iz.) Intuitively, given a noisy latent 2;, the model predicts clean token and then re-noises it forward according to the forward dynamics defined in (1).
probability pa pz(z,|z,)
Recall that x denotes sequence of K-dimensional one-hot encoded tokens, i.e., (xt) We slightly overload notation and use Xą (x))<z to represent a sequence of f-parameterized vectors
3

---

on the K- K-simplex. Each x defines distribution over the vocabulary, where one-hot vectors x' correspond to a token and lie at the corners of the simplex. Thus, we can interpret X as a mixture distribution over tokens, henceforth referred to as the predicted mixture token. With this notation, the probability of generating x' given 2; can be compactly expressed as po (,.) = (x+),x)))
The (general) discrete-diffusion setting in D3PM (Austin et al., 2021) has subsequently been special- ized to masking-based diffusion in MDLM (Sahoo et al., 2024) and MD4 (Shi et al., 2024). Their specialization has two properties: (i) zero-masking, where the predicted mixture has no support on the 'mask' letter, i.e., (xo(z))', m) = O, and (ii) carry-over unmasking, where for an already unmasked token (i.e., 2 A m), it continues to remain the same, meaning (2+), 2)) = 1. Thus, for each token I E [Z., this parameterization of the learned transition kernel leads to the following representation:
Cat (zz:: 2,),
z + m
Po (o(2))z)) = (2). 2/1 (2,))
(3)
CCat (z/: - x (2)) + 1- m
z = m.
The denoising network is trained using Negative ELBO (NELBO) (Sohl-Dickstein et al., 2015; Austin et al., 2021) (x; 0) :=
E 1zo lo xx 20)
Σ EZ 2
an(1)) aa(()
log (2()))()(()))xx))
(4)
--4(()
3 Anchored Diffusion Language Models
Our key idea is to anchor the denoising process using important tokens we call the anchor tokens (or [ANT] in short). These are tokens that, if revealed, make much easier to generate the remaining tokens. As an example, if the underlying data distribution could be represented as 2-depth tree (with each node on the tree being a token), knowledge of the value of the root node (anchor token) would lead to easier decoding of the leaves. As another example, in sentence, knowledge of the verb or noun (anchor token) is likely more useful than the articles (e.g.. 'a', 'an', 'the') or conjunction words.
Anchoring addresses the critical challenge posed by random masking in DLMs (Austin et al., 2021; Sahoo et al., 2024; Lou et al., 2024; Shi al., 2024; Wang et al., 2025a), where important tokens in sequence x may be masked in 2,, making it difficult to estimate the missing likelihoods. To overcome this challenge. we split the denoising process into two steps. First, we use an anchor network to predict the probability mixture over important tokens for each position I E [L]. Next, we employ a denoising network to aggregate these anchor predictions and compute likelihoods for the masked tokens. We call this approach Anchored Diffusion Language Model (ADLM).
ADLM Parameterization. The forward process in ADLM follows the standard absorbing discrete diffusion formulation (1), with the inference posterior given in (2). To improve denoising, we introduce new anchored parameterization of the reverse process. We propose break the one- step denoising process, widely used in practice (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Wang et al., 2025a; Ou al., 2025; Nie et al., 20, into two-stage anchored denoising framework. This allows latent reasoning over important tokens during pretraining.
Since the reverse process is Markovian, the joint probability distribution factorizes as: po(x,, 2o:1)
po(z)) )po (x(za) I po (Z) (x()). We represent each learned transition po (Z)(3) (e(ra) by the com- posite of two functions, and this learned function (that maps (Za()) 2(())) - [0, 1) is reparameterized through the pair (v, sp) as: po (Z) (((a)) == q(Za()) Zug): X (2())) Here, Yp denotes the anchor network, which predicts a mixture distribution over important tokens from the masked input Ze, and x.. (z,)) denotes the muchor-guided denoising network, which predicts likelihoods of missing tokens conditioned on the important token mixture. We analyze the benefits in $4 theoreti- cally, showing that anchoring reduces the training difficulty in DLMs by focusing optimization on important tokens. There are two key components in ADLM:
(1) Anchor Transition. Let A(-) be operator that takes sequence x
(x))= input and outputs an important token mixture y = (y') (/ A(x). We define the anchor transition as:
Cat (1 0z /z1 + a.m), r y (zx)) Cat(y!; (1- )a y (zr) + بلمرمم m). I-ae
z + m,
(5)
z m,
where y' A(x' = ).). other words, when a token 2, is already unmasked, the model preserves it as an important anchor token ( ([ANT]) with probability (1 ot). but can also re-mask it with

---

probability os (typically small). Conversely, when z, is masked, the anchoring network (-) predicts an important token mixture with probability a,-(1-oe)ar and keeps it masked with probability I-a;
1-0,- This aims to reconstruct the important tokens earlier during sampling.
(2) Inference Posterior. Anchoring introduces an implicit reasoning mechanism into DLM pre- training. Once the model is trained, modify the standard inference posterior to incorporate anchor-guided denoising. Since ADLM is trained to reason through anchor tokens internally, we do not explicitly decode the anchor tokens durine inference. Thus. inference nosterior is given bv:
Cat( z 0x)xx + o,m),
2 o m,
q(z I X (yx(z))))
(6)
Cat(zi) : - - (y + 1- m),
+ 1-a,
z = m,
where or controls the remasking probability at each timestep (Wang et al., 2025a). If token z is already unmasked, the denoiser network X (Nxx((z)) (2.)) carries it over to the next time step with probability (1 - ot). while still allowing small probability d, of masking for correction. When z masked, the inference posterior interpolates between predicting the missing token via the anchored logits and remasking it, with weights determined by or and the forward process parameters (at, a.).
Important tokens, once masked, lead to information loss that affects reconstruction by standard denoising network. By predicting these important tokens early via the anchor network, ADLM: (1) introduces intermediate latent reasoning, (2) maintains stronger context throughout the denoising trajectory, and (3) enables high quality sequence generation. In practice, even lightweight denoising network (e.g., using half the number of layers compared the anchor network) significantly improves overall likelihood modeling when guided by anchored predictions.
Training Objective. Given sequence x and important token mixture y. we optimize the parameters (v and (p) of ADLM using Anchored Negative Evidence Lower Bound (ANELBO) (see Theorem 4.1);
CANELBO (x. F; )² E Ez (x) |- log P (xly (Zo))] (Z)))
T
((()) (0(()) (*a(s)
Σ
EZuio~a(1 (xx)))
log(x (Z(()) x²) + log(y' (Z(())))
((((s)
where 7 controls anchor strength. For 0z(3) = 0 = 7, we recover the standard MDLM (4).
Anchor Token Selection. We study three different anchor token selection mechanisms: (i) Relative
frequency for LMIB and OWT in generative modeling tasks using DLMs (S5.1), where we identify important tokens using a trequency-based criterion inspired by TF-IDF For each token x' in
sequence x, we compute its relative frequency pxx') = + ビ. 1 (xx)=x)) Tokens with uxx') S are considered important and contribute to the anchoring loss. (ii) Digit extraction for GSM8K in math reasoning tasks using AR models, where we use the numerical digits as the anchors, and this enables out-of-order reasoning; and (iii) Extraction of verbs and nouns (excluding articles such as "a", 'an', 'the") for logical reasoning tasks in AR models with symbolic reasoning traces from ProntoQA and ProsQA. We defer further details anchored AR models to $5.2.
4
Theoretical Results
4.1 Anchored Negative Evidence Lower Bound
Recall that each latent variable Z, is corrupted version of the original sequence and y A(x) a mixture of important tokens. We define the anchored transition function as:
is
(Cat(y]; (1 (111-000 o, + o.m). Cat(y'; 1 (1-0(10 )a, y' 1-0,-0.0 m).
z + m
rly'lz
(8)
z m.
where 2 y and Ozt, oe time-dependent coefficients derived from the corruption schedule. This motivates our choice of anchored transition (5) in ADLM parameterization ($3). To align the model's anchored predictions with this target transition, we define the anchor loss:
LAnchor (x; :0) := (2 n (xc)
(rL(T(Y.(0) ( y) Il Tip (Y (1n1))
y A(x), (9)
where To is a learned parametric anchor transition function. We now derive the ANELBO objective, which integrates the anchor network within denoising model x. (.)(.)). The resulting bound regularizes the denoising process using structured guidance from the anchor predictions.
5

---

Theorem 4.1 (Anchored Negative Evidence Lower Bound), Suppose the inference posterior is parameterized as (6). Denote by 0 the collection of parameters of the anchor and denoiser networks, i.e., f = [b, pl. Given a sequence x = (x' )=2 let the important token mixture y y= =f A(x) be obtained through the operator A(-). Then, the anchored negative log-
likelihood is bounded by: log (xx) + Aucchr (x: S LANELBO v, (), where
CANELBO(X: v. () == (z)mm((1x)) -- Po (Zo))]
Σ B Beg
Auo (log(x4) (v)((Z()))) x²) + 1.ylogyy (20))(()
(1 (1-or())a0)) (o(1) and 7y > O.
with weight Ae(s)
1-dx(()
Remark 4.2. We choose a constant y to simplify the notation. Our derivation also applies to a time dependent "t. This only changes the contribution of the anchor loss in LANELBO(X; s), ip).
Implications. The ANELBO objective highlights two important aspects induced by anchoring:
The first term (x,, (yx(Zu())) x)) encourages the denoising network to model the likelihoods of missing tokens, conditioned on the output of the anchor network.
The second term lyg (()))))) y²) directly supervises the anchor network, encouraging it to predict important tokens early during sampling.
To summarize, anchoring improves likelihood because the denoiser does not waste capacity modeling high-entropy distributions over missing key words, having already resolved them via anchors.
4.2 Anchored Graphical Model Analysis
The core training objective in both AR and DLMs is maximum likelihood estimation (MLE). MLE has a rich foundation in graphical models (Koller & Friedman, 2009), providing principled way to understand expressiveness, tractability, and sample complexity. We reinterpret AR and DLM training as learning in directed graphical models (DAGs) and formally analyze our anchoring mechanism. While rooted in classical theory, we demonstrate that anchoring yields practical benefits in both large-scale pretraining ($5.1) and supervised fine-tuning ($5.2) tasks.
Assumption 4.3. Suppose the following properties hold: (i) Each conditional distribution p(x'l.) I-) is modeled as a categorical distribution. (ii) The model is parameterized by Conditional Probability Tables (CPTs); that is, distinct parameter is assigned to each configuration of the conditioning set. (iii) Anchor sets T. C [1,,..., L} {1} are fixed and of bounded size | S with d << L.
Proposition 4.4 (Reduced Sample Complexity via Anchoring). Suppose Assumption 4.3 holds. The sample complexity of MLE is given as follows: (i) Standard AR: Each token x' is conditioned on all previous lokens x1:1-1, The total number of parameters is O(K*), resulting in a sample complexity of OKK*). (ii) Standard DLM: Each masked token x' is predicted conditioning all other tokens x x'. The per-token parameter count is O(KL), leading to total sample complexity of ((LK*) (iii) Anchored AR: Each token x! is conditioned only on a fixed-size anchor set x*), The number of parameters per conditional is O O(K++1), giving a total sample complexity O OLLkd+1). (iv) Anchored DLM: Each masked token x is predicted using only anchor tokens x \ {xx). The per-token parameter count becomes O(Ka+1), ) resulting Implications. Assuming the existence of important tokens (anchors) in sequence, anchoring achieves exponential reductions in sample complexity: O ) to (LKá+1). We provide additional theoretical results and discussion in SA. We defer discussion S.3.2. We discuss an expectation
total sample complexity of O(LKd+1).
sample complexity to $A.3.1 and ixpectation-maximizzttion maximization (EM) (Dempster et al., 1977) interpretation of our anchored training procedure in SA.3.3. Finally, provide example in $A.3.4 that shows improved likelihood of unmasking anchor tokens early in the denoising process.
5 Experiments
Our experiments are designed to evaluate two main aspects of language modeling: (1) likelihood modeling and (2) generated text quality. Prior work has shown that improved perplexity during pretraining often correlates with better downstream performance (Austin et al., 2021; Lou et al., 2024; Shi et al., 2024; Sahoo et al., 2024; Wang et al., 2025a; Arriola et al., 2025). Since our focus is on pretraining, primarily evaluate models in terms of their test/validation perplexities, as well as

---

Table Test perplexities (PPL.) on LMIB and OWT. 'Reported in (Sahoo al., 2024). Bold: Best difflusin method. We retrain AR and MDLM to match performance reported in original papers. Our method outperforms previous diffusion language models using the same number of training tokens.
(a) LMIB
(b) OWT
Model
PPL. (L)
Tokens
Model
PPL. ())
Tokens
Antoregressive
Autoregressive
Transformer-X Base (Dai et al., 2019)
23,5
AR (setrained)
17.94
110B
OmmiNety (Tay et al., 2021)
21.5
ARl (Sahoo et al., 2024)
17.54
262B
Transformer' (Sahoo et ul., 2024)
22.32
33B
AR (retrained)
17.26
$24B
Transformer (retrained)
21.55
65B
Diffusion
MDLM (rctrained)
24.04
110B
Diffusion
ADLM (ours)
21.66
110B
BERT-Mouth (Wang & Cho, 2019)
142.89
SEDDI (Lou et al., 2024)
24.10
262B
D3PM (absorb) (Austin et al., 2021)
76.90
MDLM' (Sahoo et al., 2024)
23.21
2628
Diffusion-LM (Li et al., 2022h)
118.62
MDLM (sctrained)
23.17
262B
DiffusionBert (He et al., 2023)
63.78
GIDD Rütte al., 2025)
22.29
262B
SEDD (Lou
2024)
32.79
33B
UDLM (uniform) (Schiff et al., 2025)
27.40
262B
MDLM (Sahoo et al., 2024)
27.04
33B
DUO (uniform) (Sahoo et al., 2025)
25.20
262B
MDLM (actrained)
27.07
33B
ADLM (ours)
21.79
2628
UDLM (uniform) (Schiff' et al., 2025)
31.30
33B
ADLM (ours)
20.62
262B
DUO (uniform) (Sahoo et al., 2025)
29.90
33B
ADLM (ours)
26.40
33B
MDLM (Sahoo et al., 2024)
22.98
$24B
MD4 (Shi et al., 2024)
22.13
524B
GenMD4 (Shi et al., 2024)
21.80
524B
MDLM (retrained)
25.49
65B
BD3LM (7/ 4) (Arrioa et al., 2025)
20.73
5248
ADLM (ours)
24.46
65B
ADLM (ours)
20.14
524B
zero-shot generalization. Additionally, we measure generation quality using GPT-2 Large perplexity, entropy, and MAUVE (Pillutla et al., 2021) scores. While perplexity (PPL) captures the likelihood, MAUVE score measures divergence between neural text and human text.
5.1 Diffusion Language Models
Setup. We evaluate ADLM on two benchmarks: One Billion Words (LMIB) (Chelba al., 2013) and Open Web (OWT) (Gokaslan & Cohen, 2019). For LMIB, we use a context length of 128 with the BERR-baas-uucassd tokenizer and evaluate the standard test split. For OWT, we the GPT-2 tokenizer (Radford et al., 2019). Our anchor network adopts the transformer architecture from SEDD (Lou al., 2024), based on the Diffusion Transformer (DiT) (Peebles & Xie, 2023) with rotary positional embeddings (Su et al., 2024). The denoiser network uses the same base architecture but with half the number transformer layers.
We experiment with two diffusion samplers: (a) the locked-in sampler from MDLM (Sahoo et al., 2024), which fixes previously unmasked tokens by setting az O, and (b) the remasking sampler from ReMDM (Wang et al., 2025a), which allows re-masking with a small a, jf O. For fair comparison, we adopt the exact sampler configurations used in the respective baseline implementations.
Baselines. We compare against the following baselines: (1) the Autoregressive (AR) architecture from (Sahoo et al., 2024) trained with next-token prediction; (2) SEDD (Lou et al.. 2024): A score entropy
discrete DLM: (3) MDLM (Sahoo et al., 2024): A masked diffusion language model; (4) MD4 (Shi al., 2024): A masked diffusion model for discrete data; (5) BD3LM (Arriola al., 2025): A hybrid
approach combining AR and diffusion components; (6) ReMDM (Wang et al., 2025a): MDLM with masking sampler; (7) GIDD (von Rütte et al., 2025): A DLM that interpolates between masking and uniform noising; and flow matching methods, such as (8) DFM (Gat et al., 2024) and
(9) Forward- Borward-Backwart (FB) (Campbell al., 2022) samplers. We follow the implementation of these baselines from MDLM (Sahoo et al., 2024) and ReMDM (Wang et al., 2025a) repository. We describe each baseline and provide links the source code in SC.
5.1.1 Improved Likelihood Modeling and Generated Text Quality
We first evaluate ADLM on likelihood modeling and generated text quality using the locked-in sampler from MDLM (Sahoo et al., 2024). Based on our ablation study (deferred to SC), we adopt 5 our default configuration for anchoring ($3).
y 3e-3 and
Results on LMIB. Table (a) shows that ADLM outperforms previous diffusion models such as SEDD, MDLM, and DiffusionBERT. At 33B tokens, ADLM achieves a test PPI. of 26.40, improving over MDLM (27.04). Scaling to 65B tokens further reduces PPL. to 24.46, approaching AR models like our retrained Transformer (21.55).
7

---

Results on OWT. Table 1 (b) presents the test PPL of our proposed method, ADLM, across three training regimes: 1 10OB, 262B, and 524B tokens. At each scale, ADLM consistently outperforms iiffussionhbsed baselines such as MDLM and GIDD, as well as the hybrid (AR+Diffusion) BD3LM. Notably, at 262B tokens, ADLM achieves a PPI. of 20.62, narrowing the gap with the AR models, which reach PPL. of 17.54. ADLM" uses our multi-stage design (anchor and denoiser with y 0) to train MDLM that improves its PPL from 23.17 to 21.79. demonstrates that anchoring is not just adding extra capacity, but truly guiding efficient learning. At 524B tokens, ADLM further improves to 20. 14, approaching the AR performance (17.26). These results verify the effectiveness of ADLM in bridging the gap between DLMs and AR approaches, without relying AR components.
Generated Text Quality: While test evaluates the abil- ity to predict missing tokens, it does not necessarily reflect the quality of generated text. Following common practice, we assess generated text quality using GPT-2 Large. Table 2 shows GPT-2 Large PPL. scores on OWT for models trained on 524B tokens. Our method, ADLM, achieves the lowest PPL among DLMs, outperforming prior approaches such as MDLM and SEDD. ADLM surpasses the hybrid BD3LM at both L' = 8 and L' = 16 block lengths. Since smaller block lengths make BD3LM behave more like an AR model (L' = 1 is equivalent to pure AR), larger block lengths rep- resent the diffusion regime more accurately. Thus, higher I. provides a fair evaluation against other DLMs. We refer to Table in Appendix C.1.3 for a more extensive study.
Table 2:
GPT2-Large perplexities (PPL; }) on OWT (524B tokens). ADLM uses the remasking sampler with 1000 steps.
PPL (L)
AR (Sahoo al., 2024)
14.10
BD3LM (Arriola al., 2025) L' = 16
33.4
L' = 8
30.4
L' =4
25.7
SEDD (Lou et al., 2024)
52.0
MDLM (Sahoo al., 2024)
44.2
ADLM (ours)
26.8
Zero-shot Perplexity Evaluation. We train ADLM on OWT and evaluate its zero-shot general- ization across seven diverse benchmarks using validation perplexity. As shown in Table 4, ADLM consistently outperforms prior diffusion- based models such as SEDD and MDLM on 6 of tasks, and matches performance on the remaining WikiText benchmark. It also surpasses the hybrid BD3LM (with block length L' = 4) on five benchmarks. These results suggest that ADLM learns robust representations because the notion of importance captured by the anchor network generalizes even when the distribution shifts.
Notably, ADLM outperforms the AR baseline on three challenging datasets: (1) Lambada, which tests long-range contextual understanding, and (2) PubMed and (3) ArXiv, which evaluate scientific language modeling. These results indicate that ADLM not only narrows the performance gap with AR models but can exceed them on tasks requiring long-context reasoning and specialized knowledge. Importantly, ADLM achieves these gains with the same number of neural function evaluations (NFEs) as AR models, demonstrating both efficiency and strong -iitrr-diittiuttion generalization.
Remasking Sampler. Now, we evaluate ADLM using the remasking sampler (Wang et al., 2025a), with results shown in Table 3. This flexibility enables more expressive and diverse sampling. Our pre-training method, ADLM, when combined with remasking sampler, achieves ofte-of-the-ar performance across multiple metries. Importantly, becomes the first DLM to outperform the AR model in MAUVE score, particularly at T = 2048 and T = 4096. Anchoring provides up to 4x improvement in inference speed compared to ReMDM as evident from Table 3 (highlighted in green).
In addition to MAUVE, we report GPT-2 Large perplexity and entropy to assess generation quality and diversity with increasing number of sampling steps. While test PPL can be artificially lowered by repeating high-likelihood phrases, such models typically exhibit low entropy and fail to capture the richness of natural language. Our method maintains high entropy-elosely matching the data distribution while achieving low PPL and high MAUVE scores. This balance indicates that ADLM not only generates high quality human- like text but also preserves the diversity.
5.2 Auto- Regressiee Models
We observe that the benefits of anchoring extend beyond the diffusion setting. Specifically, anchoring can be incorporated into auto-regressive generative models as discussed in Appendix C.2, as well as for supervised finetuning (SFT) of pretrained auto-regressive models as discussed in Appendix C.2.5.
Below, we discuss the second setting of supervised finetuning of pretrained auto-regressive models. Our key observation is that inserting [ANT] after questions and before the start of (reason, answer) tokens enhances reasoning capabilities of AR models (see also Appendix C.2.5).
8

---

Table Evaluation sample quality using the ADLM with remasking sampler (Wang et al., 2025a) on OWT. ADLM outperforms state-of-the-an masked diffusion and flow How-matching methods. For 2048 and T 4096, ADLM surpasses AR in MAUVE score (measures human-like text).
Method
MAUVE (t)
Gen PPL. (1)
Entropy (t)
Data
1.00
14.8
5,44
AR (T=1024)1
0.760
12.1
5.22
T=1024
T=2048
7=4096
T=/024
T=2048
T=4096
T=1024
T=2048
T=4096
SEDD (absorb)
0.008
0.008
0.009
104.7
103.2
102.5
5.62
5.61
5.61
MDLM
0.042
0.037
0.035
51.3
51.3
50.9
5.46
5.46
5.45
MDLM+FB
0.133
0.197
0.243
33.8
28.6
22.8
5.35
5.28
5.18
MDLM+DFM
0.254
0.294
0.269
21.7
21.0
20.7
5.20
5.19
5.17
ReMDM
0.403
0.610
0.656
28.6
22.8
17.6
5,38
5,30
5.20
ADLM (ours)
0.699
0.788
0.791
25.4
20.3
15.9
5,35
5,28
5.19
T=128
T=256
T=512
T=128
T=256
T=512
T=
T=256
T=512
SEDD (absorb)
0.007
0.007
0.008
119.2
110.1
107.2
5.65
5.63
5.62
MDLM
0.015
0.023
0.031
61.5
55.8
53.0
5.52
5.49
5.48
MDLM+FB
0.064
0.084
0.100
42.8
39.6
37.1
5.44
5.41
5.38
DDLM+DFM
0.041
0.144
0.211
37.9
26.5
23.3
5.26
5.23
ReMDM
0.057
0.216
0.350
42.5
30.5
21.1
5,43
5.34
ADLM (ours)
0.140
0.349
0.573
52.5
39.85
31.6
5.52
5.46
5.40
Table 4: Zero-shot validation perplexities (L) of models trained on 524B tokens from OWT. ADLM
achieves ootteoff-hhe-rt among diffusion language models and outperforms autoregressive (AR) models on three benchmarks: Lambada, PubMed, and ArXiv. All models use 1024 NFEs.
Lambada
PTB
Wikitext
LMIB
AG News
PubMed
ArXiv
AR
51.28
82.05
25.75
51.25
52.09
49.01
41.73
BD3-LM (L' 4)
50.03
96.81
31.31
60.88
61.67
42.52
39.20
SEDD
49.86
100.09
34.28
68.20
62.09
44.53
38.38
MD4
48.43
98.16
32.42
MDLM
47.52
95.26
32.83
67.01
61.15
41.89
37.37
UDLM (uniform)
53.57
112.82
39.42
77.59
80.96
50.98
44.08
DUO (uniform)
49.78
89.35
33.57
73.86
67.81
44.48
40.39
ADLM (ours) (262B)
44.93
98.16
32.45
65.59
57.10
38.29
35.08
ADLM (ours) (524B)
44.32
95.37
31.94
64.43
55.72
37.56
33.69
Setup. We use a pretrained GPT-2 (Radford et al., 2019) model as the base architecture. We finetune the base model on math and logical reasoning tasks using standard SFT reasoning traces and answers. We evaluate on three benchmarks: (1) GSM8K (Cobbe et al., 2021) (02l1)-rade-schoee logical reasoning, and (3) ProsQA (Hao et al., 2024) planning with structured reasoning over graph-based inference traces. Our experimental setup follows the fine-tuning protocols outlined in prior work (Hao et al., 2024), enabling direct comparison with established baselines.
math problems with arithmetic reasoning, (2) ProntoQA (Saparov & He,
Baselines. We compare against range of latent reasoning and chain-of-thouge (CoT) methods. These include standard CoT finetuning (Wei et al., 2022), improved variants such as iCoT (Deng et al., 2024), and multi-stage fine-tuning approaches like COCONUT (Hao al., 2024). We compared with two additional basclines: No-CoT, which trains models directly on question-answe pairs without intermediate reasoning traces, and Pause Token (Goyal et al., 2024), which inserts special pause tokens between the question and answer to encourage thinking. These methods are finetuned using the same base model: GPT-2 (openai-communityygp())) with identical parameter count. We also include recent work BoLT (Ruan et al., 2025) that reasons to learn from latent thoughts.
5.2.1 Improved Reasoning using Anchored Chain-of-Thought
Inspired by recent work on chain-of-thouge prompting (Wei al., 2022: Hao al., 2024; Goyal et al., 2024), we investigate whether anchoring improves the reasoning ability of AR models in both math and logic domains. To operationalize our anchoring mechanism AR models, we insert [ANT] after question and before (reason, answer) tokens, and then use standard SFT with important tokens from reasoning traces as lables for these [ANT] tokens. We defer implementation details to Appendix C. We refer to this variant as Anchored Chain-of-Thoun Thhoughh (ACoT)), illustrated in Figure 2. Here, [BoA]/[EoA] denote the beginning and end of anchors, while [BoR]/ [EoR] mark the start and end of reasoning tokens. Quantitative results presented in Table 5.

---

Figure 2: Anchored Chain- of-Thought (ACoT), During training, anchor embeddings are passed through the lan- guage model head com-
[BoA]
[BoA] [BoR]
<314-72
<16-7-00
<y12-18>
[EoR]
Language Model Head
Embreddings
pute cross-entropy loss on an-
chor tokens. At inference, the
Anchor Transformer (with causal attention)
model reasons directly in the
latent anchor space without decoding these embeddings, enabling awture-aware, out-of- order reasoning since anchors need not correspond to imme- diate next tokens. Example trace from GSMSK dataset.
[BuA] 01100000000
Embeddings
[FoA] [BoR] -344-77 <I6-7-90 -992 18=
[[oRR]
Janet's ducks lay 16 eggs per day, She eats three breakfast every momning and hakes muftins for her
friends everyday with four. She wells remsinder ai the farmers' market daily for S2 frush duck egg: much
she make every day at
market!
Results on Math. On GSM8K, ACoT achieves accuracy of 45.2%, outperforming compared baselines, including standard CoT (42.9%) and multi-stage finetuning approaches like COCONUT (34.1%). Anchoring improves decoding by treating an ordered subset of reasoning trace tokens, after filtering out punctuation and arithmetic operators (+, -, +), as anchoring tokens. This guides the ACoT model through important tokens before generating reasoning traces and the final answer.
Results Logic. We anchor using valid nodes from the reasoning traces after pruning conjunc- tions, articles or adjectives, such 'every', is', and 'a'. As recommended in COCONUT, we pro- gressively increase the number of [ANT] tokens. For ProsQA, we gradually remove the reasoning steps while inserting [ANT] tokens, which helps enhance logical reasoning (Hao et al., 2024). On relatively easier benehmark ProntoQA, ACoT achieves 100% accuracy, matching or exceed- ing prior approaches. On the more challenging ProsQA benchmark, ACoT reaches 97.3%, im- proving over COCONUT (97.0%) and surpassing CoT variants except iCo'..
Table 5: Accuracy (% ) on Math and Logical Reasoning. ACoT improves the performance of prior (continuous) latent reasoning methods de- spite using the same multi-stage training setup as COCONUT. f reported in COCONUT.
Method
GSM8K
ProntoQA
ProsQA
No-CoTt
16.5
93.8
76.7
Pause Token
16.4
77.7
75.9
CoT
42.9
98.8
77.5
iCoT
30.0
99.8
98.2
COCONUT
34.1
97.0
Pause
24.1
100
96.6
BoLT
33.6
While standard multi-stage training such as in COCONUT leads to an improvement from 98.8% (CoT) 99.8%, our anchoring mechanism further enhances performance. By incorporating explicit supervision through [ANT], ACoT achieves perfect accuracy of 100% on this relatively easier benchmark, demonstrating the effectiveness of anchoring in logical reasoning. This study also opens up other intriguing questions and potential directions. Words corresponding to verbs and nouns relate to the concepts of terminals and syntuctic categories the Chomsky Hierarchy (formal language theory). We defer additional results and discussion to Appendix C.
ACoT (ours)
45.2
100
97.3
6 Conclusion
We introduce Anchored Diffusion Language Model (ADLM), two-stage generative framework that improves diffusion language modeling by leveraging anchor tokens (c.g., low-frequency or important key words). We provide theoretical justification along with strong empirical evidence supporting our results. Our method bridges the gap between diffusion and AR models likelihood modeling and generated text quality. ADLM significantly reduces test PPL on LMIB and OWT, outperforming previous DLMs in 6 out of 7 zero-shot benchmarks, and, for the first time, enables a diffusion model to outperform AR models in MAUVE score that measures human-like text generation quality. Beyond diffusion, we demonstrate that anchoring broadly applicable and improves reasoning in AR models. Our Anchored Chain-of- Thought (ACoT) method improves performance on math and logic benchmarks, outperforming existing approaches. These results highlight the impact of anchoring as a general-purpose general- purpose framework for language modeling and complex reasoning.
Limitation. While anchoring yields consistent gain, the definition of token importance task-specific. We use low frequency tokens key words proxies for importance, which may not generalize. Future work may explore adaptive or LLM- --gudddiedd anchoring for efficient planning and reasoning.
10

---

Acknowledgments
This research has been supported by NSF Grants 2019844 and 21 21124 the UT Austin Machine Learning Lab, and computing support on the Vista GPU Cluster through the Center for Generative Al (CGAI) and the Texas Advanced Computing Center (TACC) at UT Austin.
References
AI Anthropie. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024, URL https://wee cdn. anthropic www.co////////o Model_Card @aard_Caade______11_ .df..
Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:///ppeeeeeee http://ppppeeeeeeeeeeee/e seree/fooummidde :orym?
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
https: /tttps://openee http::/:///peeeeieee/ee/ee net/ forum?iddhi -
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb): 1(Feb):1137-1155 137-1155, 2003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020. URL https: http:////occceec..e/eees neurips wwcppaper_filie files faaper_ffileeepppeeeeee /pppperrileeessppp//////////// pdf.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httttts://opperevv/e weriiiiw.eeeeeeee nwt/ orrmm DdDD21262Y..
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint 131iv 2013.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? analysis of bert's attention. In Proceedings of the 2019 ACI. Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pp. 276. Association for Computational Linguistics, 2019. URL https: / https::/tacaaiii //clantthoola org/ V19-4828/.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Associa- fion for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615-621, 2018
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime G Carbonell, Quoc Le. and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019.
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodologicall) 39(1): 1-22, 1977.
11

---

Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:240S.14838, 2024.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P65/////11...// 1082. URL https: http///aaaatttthh
org/P18-1082
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024. URL /ttp:::://peervviii wtt.:eeette.oorrr.aaaaooooooooo
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skkylion007/ -ww.:0007.iitth github...... / OpeniebTextCorpue, 2019.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference on Learning Representations, 2024. URL https : /ttp///ppeeeeeeeeeeeeeeeeeeeeeee net/ forum? 2errun/id-ph0OCRRPP..
Ishaan Guirajani and Tatsumori Hashimoto. Likelihood-based diffusion language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openreview wtt.ooormmmdmem.mmmmee
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprins
arXiv:2412.06769 2024.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuan-Jing Huang, and Xipeng Qiu. Dif- fusionbert: Improving generative masked language models with diffusion models. In Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4521 4521-4534, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, Curran Associates, Inc., 2020. URL https://proceeding/ neurips.co cc/ pdf.
Frederick Jelinek. Interpolated estimation of markov source parameters from sparse data. In Proc. Workshop on Pattern Recognition in Practice, 1980, 1980.
Michael I Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence Saul. An introduction to variational methods for graphical models. Machine learning, 37:183-233, 1999.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https:/ /pw//eoeeeveeeeeeeeeeeeeeeeeeeee net www//eeruuuee wwwoorrrmmniiiiioeeeee CEKvH.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT Press, 2009.
Jeongyeol Kwon, Wei Qian, Yudong Chen, Constantine Caramanis, Damek Davis, and Nhat Ho. Global optimality of the em algorithm for mixtures of two-component linear regressions. IEEE Transactions on Information Theory, 70(9)66519 65456, 2024. doi: 10. 109/TTTT22024433355222
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nip. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681-691, 2016
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL. https:/ wwwp//veeneeeeeeeoo//e net/ htte///oommmmddnmnmmee
12

---

Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal. Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL betps://openree
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:
521-535, 2016. doi: 10. htte:/tacl__00111__ URL https: battps///aclae https////claaatoooroo.o oogg/16-1100000
Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stark, Yilun Xu, Tommi Jaakkola, and Rafael nomez-Bombarelli. Think while you generate: Discrete diffusion with planned denoising, In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview. hwt.:eee//fooooeee ftlfloormmmddddo MDNyyaaddD
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024. URL
https: http://opppeviieeeeeee
wwtpeeiwwwwweeeeeeeeeeeeeeeeeeee wwworrmmiiddiii CRIVIPA.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-333 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference on Learning Representations, 2017. URL https: //openreview www.iiew..eeeeeeeeee www//eeeurrrteee Bi
Radford M Neal and Geoffrey E Hinton. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pp. 355-368. Springer, 1998.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models text. In The Thirteenth International Conference Learning Representations, 2025a. URL https: / //pp//peeeeeeeeeeeeeeeeeeeeeeeee net/ forumm WWttktttott
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.0990 09992, 2025b.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL www.neeeeooommmddmmmaaa..a.m
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc- Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1525-1534, 2016.
William Peebles and Saining Xie. Scalable diffuusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Mcasuring the gap between neural text and human text using divergence frontiers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: : //tp/e/eeee/eeeeee/ee/ee net/ Korum?id=Tqx7hJppp
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yangjun Ruan, Neil Band, Chris J Maddison, and Tatsunori Hashimoto. Reasoning learn from latent thoughts. arXiv preprint arXiv:2503. 18866, 2025.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL https: / pettps://openeee //pp://veepeeeeeeeee wwwp://e//eeeeeeee wwt. met//oroum?dd-LAualee
13

---

Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffusion duality. Forty-second International Conference on Machine Learning, 2025. URL
wet//Eorum2Ad--PPYBF000
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thoure thought. In The Eleventh International Conference on Learning Representations, 2023. http://ormmmndggF/BZeZaa.. URL https: http:::://ppeee /ttp://vepe//eeeeeeeeeeeeee
Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bermardo P de Almeida, Alexander M Rush, Thomas PIERROT. and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL httph: httpp:://ppenreetaaae ovenreview
J6g5G1.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of- of-thought into continuous space via self-distillation. arXiv preprint arX1v:2502.21074, 2025.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In The Tiirth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-22655 Lille, France, 07-09 Jul 2015. PMLR. URL https://
proceedings. mlr. www.ppress/v///// https///v///ohl--eee wwwh-oiichiie dichl--ickttiin155 html.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da-Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pp. 10193-10202 PMLR, 2021.
Team- Geam-GGmmni,, Rohan Anil, Sebastian Borgeaud, Jean- Dean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 / 1805, 2023. URL https:// //rxiv. org/pdf/22 22122. 11805.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. for Computational Linguistics. doi: 18.18653/v 1/P19/ 1452. URL https: /clantoooooooy org/P19- 1442/1
Florence, Italy, July 2019. Association
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023. URL https:
//arxiv. org/ abs 2302. 13971.
Harshit Varma, Dheeraj Mysore Nagaraj, and Karthikeyan Shanmugam. Glauber generative model: Discrete diffusion models via binary classification. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //tp///eeee./eeeeee
Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.0448) 2025.
URL https:://eeeet /ttp:v/aaavvvvavaa/aaa org/ pdtp//f///// /200 04482.
Alex Wang and Kyunghyun Cho. Bert has mouth, and must speak: Bert as markov random field language model. arXiv preprint 1rXiv:1902.04000 2019.
14

---

Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025a. URL https: https://aaxxvv.ooo wwarxiv.org/abb/25005.. 00307.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.003000 20255.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /tRptt/////ooeeeeeeeee www.:ewwweee.fooeoeeeeeeee fet/ffrrmmidd _VjQLMeSB_J.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the paper's contributions and scope as dicussed in $1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations have been included in $6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofss
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions and proofs have been provided in $4 and $A. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: All the information have been disclosed in $5 and $C.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spencsource dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The paper provides open access to the data and code (https:/aancoree -ifffusiion Ilm. uin.gihuubioo/ with sufficient instructions to faithfully reproduce the main experimental results in $5 and $C.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips cc/
suuides//ooeSuumissse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//ni //nips. cccccc//pooo wppppi/gguuieeeeeeeeeppppppppppe
CoodeSummis somiissionnl assionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and details have been included in $5 and $C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes]
Justification: The experiments have been conducted by fixing the seed for different sources of randomness, which ensures that multiple runs produce nearly the same result. We have included results with 2 independent runs for the base generative models, and matched the numbers up to the first decimal point, which indicates robustness of the conducted experiments. We provide additional information about the statistical significance of the experiments in $C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The compute resources have been discussed in SC. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pentps:/nneuus http///eerrpp/eeeee ceabliic/tthisshideeee
Answer:
[Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes]
Justification: The paper broader impacts have been discussed in &C. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
18

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system lears from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes]
Question:
Justification: The safeguards have been described in $C.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators original owners of assets are properly credited in $C. Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should provided. For popular datasets, thodee thcode. wtt.cooo////a//e datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the dataset/codee/model has been discussed in &C. Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There is crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [ÑA]
Justification: There is no human subject involved in the experiments. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution. For initial submissions, applicable), such as the institution conducting the review.
not include any information that would break anonymity (if
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required. Answer: [NA]
Justification: The paper does not use LLMs in this research. Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.M policy (https: //wp/r/nss.////// cttuurppscc//ootte /ew.eeee/eeeeee//ee// for what should or should not be described.
20

---

A Additional Theoretical Results and Proofs
This appendix provides complete theoretical results that were either stated without proof omitted from the main text due to space constraints. For completeness, we restate key theorems and provide detailed proofs, along with additional theoretical insights. In $A.1, Theorem 4.1. In $A.2, we derive the transition kernel for masked DLM for completeness. Finally, in $A.3, provide detailed discussion on the statistical benefits of anchoring in both diffusion and autoregressive models, with an emphasis on sample complexity and likelihood modeling.
present the full proof of
A.1 Proof of Theorem 4.1
We follow the standard discrete diffusion analysis for the NELBO objective (Sohl-Dickstein et al., 2015), but incorporate our anchored parameterization into the divergence computation.
Theorem A.1 (Anchored Negative Evidence Lower Bound). Suppose the forward process follows (I), and the inference posterior is paramelerized by anchored denoising in (6). Denote by 0 the collection of parameters of the anchor and denoiser networks, i.e., 0 = [1, Let A(-) be
Tiz as input and returns an important token mixture y = (y') =(x) as output. Then, the anchored negative log-likelli likelihood bounded by:
an operator that takes a sequence x
- log pxx) + +-yC (x; vp) S LANELBO (x; v ()), where
C CANELBO bp = 1z) |- --oog P:: (x (Zn))
T
E
Zuc (1(x
Ae(s) Σ
(x x²) + loglyy (2(0))
1=1
1=1
with weight )e(e)
(1-011)) Jaic)
and 7 > 0.
1 1-0a(0)
Proof. We first derive the bound for sequence length T, = 1; the extension to J. > is straightforward following the standard analysis (Sohl- Sohl-Dickstein al., 2015).
We start from the standard negative log- log-likelihood:
(x)) = loo / po(x, Zo....,
71) (i)......... 71)
-ooog /
po Zo:1)
((Z0.1 (x) di (o.)..).
((Za.1xx)
Applying Jensen's inequality yields:
log (x + log o(Zg x)
T
((Zo(1) Ze(a): x)
- log (x < Ey(Zon (x))
log
-NELBO 0)
po (21)
pa (z(() Z((z))
Thus, the NELBO decomposes into:
T log Po + DKL (g(Z(x)) Po (Z)) + Σ
[XELBO 0) Eyiio..1 (x)
DEL (9(Za(e) Ze() x)) x (2(00)))
i=1
Combining the NELBO decomposition with our anchored loss (9) gives:
(ANELBO th :p) LNELBO (x; v/, (0) + ALuchor sp)
T
=Ry(Zoale xx)
log Dogpo(x Zo) (x (Z1)
Σ
DRL (oz Ze(0) x))lpo( (Z Z(())
+ +7
DKL. (ra,(r(V. Vly
= =Eq(Zon (x)
- -10gpe(x|Zo)
+ Egizon (x)
+ (Zo. (x)
Σ DRL(g(Z.(o)) ( + DDDa
21

---

The three terms in the above expression have natural interpretations:
The first captures the reconstruction loss the final step of the reverse process. The second term measures the error due to mismatch between the stationary distribution of the forward process and the initial distribution of the reverse process. This vanishes when the reverse process is initialized with a sequence of all masks.
The third term aggregates the KL divergences across diffusion steps, and encodes the difficulty of denoising masked tokens. Our anchor network aims to reduce this difficulty by enabling early decoding of important tokens.
We now focus on analyzing the third term defined as:
Laitfusion (x; (p) DRL (rk,)a((2s)))zip x0)p (2100))))))))))))))2))))) (10ka
Since = A(x), each KL divergence can be computed by splitting into two cases, depending on whether the token Zu(s) is already unmasked.
Case 1: Zu0 # (unmasked). In this case, the diffusion loss for the 7th KI.-Divergence term:
((Zx(1) Z((0), x
log ((Ya(s) Z(()) rip(' (2(3))
dirrusion (x; 1/, 1p) E2yc (2()):
log
+ +7 +
Po (2(()()) (e(s))
((Zx(e) = ((((()) * m. x) Po m Zi(i) F m)
<Z(((()) = m|Z((() * m. m,x) log
(((s(1) x Zu() * m. x)
+ +((0000 = x|Z0(1) * m. log
Po (o((2()) = x x|Z(tt) # m
r(Va()) m Z1(00) # m,
+gr(Ys() m ((() * y)
(o(((())) m m/Z(()) * m)
r(Ya() = y Z1()) * m.
+gr(Es(1) = ZiZ((0) * m. log
TY((s()) =y z # m)
((Zx2)) m Z+(i) # m. x)
= <Zz(() = m/ m * m, x), log
((z(1) = m ml # m, Xx (y 2(0)))
a(Z.(0) 21(s) + m
+ +(((a(0 = x x * m, m,x)log
a ((a(()) = x # m. Xx (y ((())
r(Y,() ml Zu(1) # m
+rr(Ys() = -m # yog
r(Ys(1) = ml\(()) * m, y (Z)()))
r(Y,(1) y Zz(z) m
+rr(Ys(0 = y|Z((1) # m. log
r(Ys(s) = y y|Zu(( # m, y (((())
=11 +1++70000) log oe(e) O(()
(1 )2()
(1- (1-0((0) 0c(i)
(1 +1+7)(1-0(0) log
Implications. This result demonstrates that when the generative model's reverse transition aligns with the inference posterior for unmasked tokens, the diffusion loss becomes zero. This validates the effectiveness of our two-stage parameterization. The key implications are:
Unbiased Learning: Anchoring introduces additional bias when its distribution matches
the inference posterior (6).
Tight Variational Bound: The ANELBO objective (7) remains tight bound on the data
likelihood, ensuring the theoretical soundness of our formulation.
Case 2: Z, (masked), Following the anchored denoising formulation, obtain:
22

---

((Zx(a) ((0)), x)
r(Va(s) Z(())) rx((x(() (((1))
Cdimussion (x: (p) Bocl (x(())
log
+ (Y) (eco)
log
pe Po(Zx()) Ze(1))
((Zx(1) = m ((()) = m. x)
<(((s()) m|Z((() m, x, log
po
21(1) m)
((Za(1) = X Z = mm. .m
+ (Z(o0))) = x|Z((1) = m, log
po(()()) = x x|Z1(0) m)
r(Ya()) = m ((()()) = =m, y)
+yr(Ya()) = m/ mlZ((1) y)
(p(()()) m (z(1) = m
r(Xs(1) = y 21Z)() = =m,y/
+gr(Xa()) = y(Z((0) = y) log
(x(()()) y|Z((1) m
((a(11) ml 700 = m. x)
= -(z(()) = m/ mlZ((0) = m, x) log
(Z.(1) = ml mlZ((0 m, x Xx (v/ Z1())
(Z Z(( m.
+ =((2(()) = x x|Z((1) = m. m, log
(Zs(()) x|Z(() = m. X (y (Z)()))
r(Y,()) Z(())()) m
+Tr(Ya() = m/ m|Z((0 = m, m, log
r(Yo()) = m ml
y,p (2)(1))
r(Ys(1) - -y m)
+Tr(Va(0 = 2 = m, y) log
r(Ya(s) = y y/Z(()) = m. y (2(1))
-a, a;aa:
G. aa-a0
1 0g 0ea0
1-0c
+ 0. - + a:00
1-0t
log
log
1 0tt
-0 1-a1
Otf
Os 00-a0++a00
(X (Z))) (Ze))).
1-ar
1-0, 1:0g
0g3 Jza+t)
1-0t
0g 04 + 0400
1-02
log
log
Ot
1-a, 0.00
- 04
a,-0,
(2p())))),
1-cce
1-
O((s)) ((())))(()))) - (lo(3)
+ log(y)((2())))-
-((())
Combining Case and Case 2, we conclude the proof.
Summary. The complete derivation of the ANELBO (7) formally establishes the theoretical sound- ness of our two-stage ADLM parameterization. It confirms that anchoring introduces no additional bias and remains tight bound the data manifold. When the anchor and denoising networks properly aligned with the inference posterior, the KL terms decompose nicely under our parameteri- zation. This leads variational bound that reflects both token-level reconstruction and anchor-level guidance, enabling effective learning in large-scale diffusion language models.
A.2 Derivation of Absorbing Transition Kernel
This derivation is special case of the D3PM framework (Austin al., 2021) applied to masked diffusion language modeling. We include a simplified proof here for completeness.
Recall from $2.2 that the forward noising process is defined as:
2 (2,1x) = Cat (Ze; 0zx + (1 a.)m), te
T T TT..,, 11.
(10)
23

---

where m denotes the mask token distribution and E [0, 1] controls the corruption level at time t. We aim derive the transition kernel:
where 0t|s := a for g(zr|zz) Cat (2 + (1 arls atls)m)
(11)
Law of Total Probability. We begin by marginalizing over Zy:
(Zz,/x)= Σ g(z, Iz, x) g(z,(x)
2,
( Us = m) g(z = m|x) + g(zilza = x) g(2. = x|x).
Simplifying the Components. From the forward process Eq. (10), we know:
qz, x(x) Clat
q(z, = m|x) = 1 Cly.
Let == q(z, x/z. =x), and note that since m an absorbing state, we have:
m) 0, q(zt = m 2, m) = 1.
(z:
Then, the marginal probability of z = m given x becomes:
q(z, m x) q((2 m 1g m) q(z, = m|x) + q(2, mlzg =x) g(zz -x/x)
(1 a.) + (1 - a) a,
200z
From Eq. (10), we also know:
(l2 = m/x) =1 04.
Equating the two expressions:
at
1 1-a0, Ott
a
OAje
Oss
Thus, we have shown:
(Zzlzz, 2.) = Cat (Zr:ar ag + (1 a
which completes the derivation of the absorbing transition kernel.
A.3 Anchored Graphical Model Analysis
The foundational principle behind both AR and DLM pre-training is Maximum Likelihood Estimation (MLE), which optimizes their respective log- likelihoood objectives. MLE has been extensively studied in the context of graphical models (Koller & Friedman, 2009), offering principled framework to an- this section, we recast AR and diffusion
alyze expressiveness, tractability, and sample complexity. training as instances of learning in Directed Graphical Models (DAGss and use this perspective to formally analyze our anchoring mechanism. We show that anchoring- conditioning only on small, to significant reduction in parameters and sample complexity.
important subset of tokens-lee While related ideas are well-established probabilistic modeling, we demonstrate their effectiveness in large-scale language model pre- pre-training ($5.1.1) and fine-tuning ($5.2.1) tasks.
(x)...., x)).. we consider DAG denoted by EL, where token position in the sequence), and E denotes the set of directed edges representing conditional dependencies. Each node x' takes discrete value from a vocabulary of size K, so
Setup. Given sample
T= 21. I.) denotes the set of nodes (each corresponding
x' E V. Let क. S (1...., .,, L} \ (1} denote the set of mı } to be the corresponding parent tokens.
parent indices of node and define T {x
In the following, we analyze the sample complexity of learning graphical models for language, comparing standard approaches with our proposed anchored models. Our focus is to demonstrate that anchoring-by conditioning on small, important subset of tokens-can dramatically reduce the number of parameters and training samples required.

---

Assumption A.2. Suppose the following properties hold.
Each conditional distribution p-x='.)) is modeled as categorical distribution.
The model is parameterized by Conditional Probability Tables (CPTs); that is, a distinct
parameter is assigned each possible configuration of the conditioning set.
Anchor sets тi C. [l,..., L} {1} are fixed and of bounded size |Ti] S d, with d & L.
Proposition A.3 (Reduced Sample Complexity via Anchoring). Letx= = (x.,..,,x") be a sequence of discrete random variables, each taking values in a finite vocabulary V of size K, i.e., x² E EV with [V] = K. Suppose we are given N i..dd samples { i ~ 9. and Assumption A.2 holds. Then the sample complexity of MLE under different modeling paradigms is as follows:
1. Standard Autoregressive Modeling: Since each loken x is conditioned on all previous tokens x <1:1-1, the total number of parameters is O(K2), resulling in a sample complexity
of (KL).
2. Standard Diffiusion Modeling: Each masked token x' is conditioned on all other tokens
x xxx x². The per-token parameter count is O(K)), leading total sample complexity of OLLK").
3. Anchored Autoregressive Modeling (A2R); Since each wken x² is conditioned only on a fixed-size anchor set x"', the number of parameters per conditional is O(K&)) giving total sample complexity (IKK++)).
4. Anchored Diffusion Language Modeling (ADLM); Each masked token x is predicted using only anchor tokens x {xx).. The per-token parameter count becomes O(K++1), resulting in total sample complexity of O(LK4+1).
Implications. Assuming the existence of important tokens in an anchor set of fixed cardinality d, anchored modeling achieves exponential reductions in sample complexity without sacrificing model expressiveness or decoding fidelity. A2R improves upon standard AR models by reducing the sample complexity from O(L)) (LLK++1). ADLM offers an analogous benefit, reducing the sample complexity from O(LK4) 4) to (LL These results highlight the theoretical advantage of anchoring in nigh-dimensional structured prediction settings, particularly for language modeling.
Remark A.4. Consider a sequence of length L = 1024 and vocabulary size K 50257 (as used in MDLM training) for a line network x² + x² - ... + xt. Under standard autoregressive modeling, the total number of parameters in CPTs required to model the full joint distribution is O(K*) 0(502571022) which is computationally intractable to estimate.
In contrast, under the anchored autoregressive model (A2R) with a small anchor set size, e-g., d = 1 for the line network, the total number of parameters reduces to OLL?2) = O(1024 x 502572), which is within the scale of modern large language models.
Similarly, for diffussoon models, anchoring reduces the per-token parameter complexity from K 502571624 to kd+1 502572, leading to sample complexity of O(LK O 50257*)
These exponential savings illustrate that anchored modeling makes otherwise intractable parameter estimation feasible in large-scale (diffusion/AR) language modeling.
We provide detailed derivations and discussion of these results in the subsequent sections A.3.1 and A.3.2. Our anchored training procedure also has an interpretation of expectation-maxinn maximization (EM) (Dempster et al., 1977), which we discuss in A.3.3. In A.3.4, we provide an example to show how anchoring helps improve the likelihood of decoding important tokens.
A.3.1 Sample Complexity in Standard Training
Assume we are given N i..dd samples (x; [ drawn from an unknown distribution q over sequences x; 24, where V is vocabulary of size K. Our goal is to estimate the parameters of conditional probability model p(x(0) (), where 0 = [1, using MLE. The standard MLE objective is defined as:
LMLE (p)
Σ log p(x,/((p))))
Since the structure of the underlying graphical model G is typically unknown, common modeling assumption is to use fully autoregressive factorization of the joint distribution. This leads to the
25

---

following objective:
N
(R)(V.)) = Σ log p(xpp(x)bb p)
Σ
log pxx//x///1 U: sp)
1=2
which models the joint probability by conditioning each token on all previous tokens.
Alternatively, in masked diffusion language modeling, the likelihood of the missing token is computed by conditioning on all tokens except the masked token:
Diifussion (19, (2)
Σ logp(x/1xx \ X/, 12) (x/=m)
where denotes masked token and x; x is the set of all other tokens in the sequence.
In both formulations, each conditional probability is modeled as a categorical distribution over V and is thus associated with a CPT. Learning such model amounts to estimating these CPTs. The number of parameters required for each CPT depends exponentially on the size of its conditioning set. For example:
To model g(x+), we need K parameters define 10)+
For g(x²|x+)) we require K2 parameters, one categorical distribution for each value of x
In general, for q(x' xx1t-1), the CPT size is K', as we require a distribution over K values for each of the K configurations of the conditioning context x1:1-1
Summing over all positions yields the total number of parameters in the model:
K(K4-1)
Σ K'
K K-1
In this tabular setting, is well known that the sample complexity of MILE grows at least linearly with the number of parameters in order to guarantee accurate estimation. Thus, the sample complexity of learning such a model is O(K 4) which becomes infcasible even for modest values of L and K. For example, with L 1024 and K 50257 (as in GPT-2's vocabulary size), the number of parameters is on the order of 502571024, which is computationally intractable.
This exponential order highlights the need for structure- aware modeling techniques, such as anchoring. which constrain the dependency structure and significantly reduce the number of learnable parameters. In the following sections, we show how anchoring enables more sample-efficient learning by focusing on a subset of important tokens that govern the generative structure of the data.
A.3.2 Reduced Sample Complexity via Anchored Modeling
key motivation for our approach is the observation that sentence can often be accurately decoded given small set of important tokens. We propose to leverage this property by computing the likelihood of missing tokens while conditioning only on a carefully selected subset of important tokens, referred to anchor tokens, rather than the full context. This design leads to substantial reduction in the sample complexity required for maximum likelihood estimation.
Identifying the most informative tokens is an exciting problem that has been studied extensively in the NI.P literature (Linzen et al., 2016; Li et al., 2016; Sundararajan et al., 2017; Clark al., 2019; Tenney et al., 2019; Khandelwal et al., 2020). In this work, adopt simple yet effective information -hhorrttcccc strategy: tokens with low marginal frequency in the given sample tend to carry more information ($3). Hence, we treat such low- wwwffreuenccy tokens candidates for anchoring in language modeling tasks. Our empirical results support this strategy across two commonly used generative modeling benchmarks: (1) LMIB (Chelba et al., 2013) and (2) OWT (Gokaslan & Cohen, 2019), and seven downstream evaluation benchmarks ($5). We further demonstrate the benefits of anchoring in AR models by exploring alternate anchoring strategies:
In logical reasoning benchmarks such as ProntoQA (Saparov He, 2023) and ProsQA (Hao et al., 2024), root nodes of reasoning traces reliably serve as effective anchors.
26

---

In mathematical reasoning benchmarks like GSM8K, we find that early steps in reasoning traces-cee excluding arithmetic operators (such as - x, and +) contain the important information needed to derive the correct answer.
Anchored Autoregressive Modeling. Incorporating anchored tokens into our graphical model framework yields a more compact parameterization of the conditional likelihood. Specifically, we introduce Anchored Autoregressive (A2R) modeling, which modifies the conditioning structure in the likelihood. The training objective becomes:
LA2R (v)
Σ
log (xsp(x)/))
Σ log log/(x//>>1
where x denotes the set of parent nodes treated as anchor tokens.
This formulation significantly reduces the number of parameters and thus the sample complexity. Consider a line graph where each token depends only on its immediate predecessor: x + x²
+ x². In standard AR modeling, x is conditioned on all preceding tokens, requiring K parameters and yielding sample complexity of O(KK).. In contrast, A2R limits dependencies to single-token token anchors (i.e., |l 1), so each conditional requires only O(K²) parameters. The total sample complexity becomes (LK3), a drastic reduction from O(K²). For example, with sequence length L 1024 and vocabulary size K 50257 (as in GPT-2), standard modeling scales as (5525771021) whereas A2R scales as ( x 502572)
Anchored Diffusion Language Modeling. The anchoring mechanism generalizes to the diffusion setting via our ADLM parameterization ($3). The corresponding training objective is:
CADIM(U) =
log logp(x/[x x), 1 (x/=m]
where m is the 'mask' token, and 1 (x]=m) ensures that only masked tokens contribute to the overall loss. As in A2R, anchoring reduces the size of the conditioning context, leading to significantly lower sample complexity in estimating the parameters of the denoising model.
Learning Anchors via Kl--Divergence. While anchored modeling reduces sample complexity for the decoder, learning the anchor network could be challenging because the number of possible token subsets grows exponentially with sequence length. To address this, we use simple strategy- such as selecting frequenyyy tokens-Bhe for anchoring, which are easy to compute from the input sequence.
We introduce regularization term based on KL-divergence that encourages the learned anchor distri- bution r (y x) to align with the chosen anchor distribution r ryy|x). Since the model doesn't 't have ac- cess to the actual parents during inference, replace x," with predicted anchor tokens sivenns-yp2a (x (xx)1-1) for AR and y. (x; x!) for diffusion- to the following regularized objective:
arg min dipp
[AARR/ADLM(()))) (p) >DKL (r(ylx) || [+(((x))
where 7 > 0 controls the anchor strength. This objective encourages the model to first decode the important tokens before reconstructing the rest of the sequence.
Summary. By leveraging inductive biases about the distribution of important tokens within sequences, our anchoring approach achieves substantial improvements in sample complexity. Crucially, avoids the combinatorial explosion of full-context modeling while delivering strong performance generative modeling and complex reasoning tasks. Importantly, the proposed method is theoretically grounded, computationally tractable, and readily scalable to modern language models.
We note that exact inference in graphical models NP-hard in the worst case (Koller Friedman, 2009). However, many real-world scenarios do not exhibit worst-case behavior. As result, such problems can often effectively tackled using approximate inference techniques that operate over functional representations, rather than tabular. While the tabular form remains sufficient to highlight the importance anchoring, more expressive functional representations can further complement and enhance its effectiveness.
27

---

A.3.3 Interpretation Through Expectation-Maxiiizttiie
Our anchored training procedure can be naturally interpreted through the lens of the EM algo- rithm (Dempster et al., 1977), a classical framework for MLE in models with latent variables. In setting, the anchor tokens y act as latent variables. The anchor network (parameterized by (p) estimates a soft distribution over important tokens from the observed sequence x (E-step), and the denoising model (parameterized by 1)) uses this distribution to reconstruct the full sequence (M-step). While classical EM alternates between these steps, this is computationally expensive for large language models due the doubled training time. Instead, we unify both steps into a single ANELBO training objective, which allows efficient end-to-end training (approximately two months to train ADLM). Although our implementation does not explicitly perform separate E and M steps during training, the EM interpretation offers valuable theoretical insight for future research.
We now formalize this interpretation. Consider parameterized model p(x(0) with parameters = [e), v). where &b governs the denoising (generative) model and yp governs the anchor network. Recall from anchor transition function in $3 that r(y|x, X, (p) r(y(x) when z, + Therefore, minimizing the ANELBO objective is equivalent to maximizing the log likelihood in the fully- observed DAG Thus, the task simplifies to maximizing the marginal likelihood of observed data:
arg min [xx; 30 (p) = arg max log p(x(p),p) thpp
(12)
In practice, this objective is generally non-convex and does not have closed-fore solution. However, meaningful convergence analysis can carried out under simplifying assumptions (Neal & Hinton, 1998; Koller & Friedman, 2009; Kwon et al., 2024). In the following, we assume specific update rule analogous to EM and show that leads to monotonic improvement of the ANELBO objective.
Assumption A.S. Suppose the parameter updates follow the anchored EM update rule:
Wi+1: Pi-1 arg min V.p
Σ r(ylx, pog yEV
where r(y) Xx, (Pi) is the anchor distribution predicted by the anchor network at the current iterate i, and p(x, y Wb, sp) is the joint likelihood of observed variables and anchor tokens.
Theorem A.6 (Monotonic Improvement of Anchored Likelihood). Suppose Assumption A.5 holds. Let the anchor distribution be parameterized as ply|x, vh sp) = r(y|x, sp). Then the ANELBO objective undergoes monotonic improvement:
[(x; U141, (1+1 S ((x:
Implications. Theorem A.6 guarantees that the anchored EM procedure produces non-increasing negative log- ligelikll at each iteration, thereby ensuring stability and convergence to a first-order stationary point. Notably, this convergence behavior emerges even though the anchor tokens are unobserved (Kwon et al., 2024); they are estimated in the E-step and subsequently used in a regularized M-step to update both the denoiser and anchor network parameters. In practice, we approximate these two steps using single gradient update for efficient scaling.
Proof. Optimal distribution over anchors. We begin by conditioning the standard negative log- likelihood on the anchor tokens V. By introducing arbitrary distribution r(y) over the anchors,
We note that our analysis can be easily extended to the case where some tokens have been masked. This follows from conditioning on the latent variables similar to diffusion models also refer to our derivation in Appendix A.l for extending this analysis to latent variables.
et al., 2015). We
28

---

we apply Jensen's 's inequality to obtain an upper bound:
((x; 4p) -log -1ogp(x((5)
llg Σ
p(x, x11,
yEV
r(y)
= log Σ
p(x, x11.
r(y)
YEV
r(y) log
(Jensen's inequality)
r(y)
YEV
Σ
r(y) log plg Σ r(y) log rly)
yEV
yEV
Expected Eieergy
Entropy
= F(; (1b,1)).
The upper bound F(r, (fr.[b)p)) referred to as the free energy. The free energy is minimized two steps. First, minimize with respect for fixed vh, jo, and then the optimal r* to optimize v.P. s.. We now simplify this expression to identify the optimal choice of r:
F(r, = --(ry) log plo y/bh, pl +
Er(y) logr(y) yEV
r(y)
yEV
Σ yEV
r(y) logply v. Σ
r(y) logg(x/)))) +
Σ r(y) logr(y)
yEV
yEV
r(y)
Σ yEV
24 p0) +
Σ r(y) log yEV
p(y|x, v.
Σ yEv
log)) w/ (0) DKL (r(v)lp(y)xx
= log p( v DKL (r(y))l((y)x v))
(since plyl,, v/ v) = r(y]x, ip)).
The inequality becomes an equality when DKL (r(y) x)) O. Therefore, the minimum is attained when r [y) = r(yl,, .p) for a fixed (&, v). motivating our choice anchor transitions in $3.
Next, we show that the anchored log likelihood improves monotonically under the anchored EM procedure using the optimal distribution r" (y).
Decomposition of the log-likelihood. Using the identity
p(x.ylb..p) p(y th) p(x)bbp)
we can write:
F(r", [v. vp) (ogp(x(0.p)) log p(x, logp(v/x, v.
Expectation over r" (y). Multiplying both sides by r(y|x, sp:) and summing over y E V:
Σ pz) - Σ (o) log y wlv.p) sp) + YEV
Σ YEV
r(y|x, vi logplyxx,,,,.
YEV
Since log p(x p/x|1// b/, vp) is constant with respect to y, we simplify:
- -logp(x(11, (1) = Σ
+ Σ
lo) v).
yev
yEV
29

---

Difference of log-likelihoods. We now compute the difference between two consecutive iterations:
- log p(x Vi+1 Pi+1) + log log
E (y)(x y/141, + Σ r(ylx, logp(y 1441, (P++1)
r(y x, 9e) log ylub, pp)) Σ r(y log 16
Σ logg
+ Σ p.) y/ss,
ply|x, 2 pp+1)
Σ r(yix, log
x t/i pp))
Applying Jensen's Inequality. Using Jensen's 's inequality on the last term:
p(y |x, 14411 sp(+1)
(ply|x, Ui+i (00-1) ply|x, the 1p:)
Σ Crly|x, x, pi) log
< log Σ r(y|x, spi)
p(y|x, 1/. (9))
log Σ ply|x, (141,
log 1 = 0,
where we use the fact that r(y|x, px) ply|x thx, pi) and that ((-)x, is valid probability distribution.
Monotonicity. Thus, we conclude:
- -log (( + log logp(x((1,
Σ r(y|x (pi) log log y/ (1+11
+ Σ
y
S 0,
where the final inequality holds because (#/+1, Pi+1) is chosen to minimize the expected negative log-joint likelihood under r p.).
Therefore,
[(x; Vi+1, (i+1) S L(x; s/a, sp;),
which proves monotonic improvement of the ANELBO objective.
A.3.4 Improved Likelihood Modeling During Inference
Example. To illustrate how anchoring improves likelihood modeling during inference, we consider a DAG model with L 3 nodes:
(x', x² x²). Each token takes values from discrete vocabulary = 3, where m denotes the 'mask' token. We discretize time into T steps, with t {0, 3+ 30 1}. Each token x is represented as a one-hot vector, i.e., corner point of the 3-dimensional probability simplex.
== {0, 1, m3 of size
2
The full state space is S y and contains 33 27 possible states. Let X be random variable taking values in S (such as X = = (1, m)) E S), which is represented by a mixture distribution:
0 0
x
1
We define DAG structure where x" is the parent node, and x² and x3 are its children:
q(x) g(x²) g(x 1xx) q(x*lx²)
30

---

Suppose the data distribution is supported on two states:
(1,0,0) with probability 0.9, and
(0, 1, with probability 0.1.
This structure can be interpreted as a logical circuit: xx2 = 1, it activates the bulb on the right activates the bulb the left (a* 1). In this circuit, x²² determines the entire configuration of the sequence and thus acts as an important node.
(ar3 1); otherwise, if x²
It is easy to verify that the token z² is the most informative variable for modeling the joint likelihood. When z²2 = which occurs with high probability q, the entire sequence is determined. Thus, the anchor token is x²² and it is represented
column vector: x² [1. O,
This example designed to illustrate how identifying and conditioning on such informative tokens (anchors) can improve the estimation of masked token likelihoods. Anchoring thus provides, as we discuss next, principled mechanism for improving inference quality discrete diffusion models.
Forward Process. In our discrete diffusion framework, the forward process gradually corrupts an input sequence x (x+xx², x²) by replacing tokens with special mask token m, according to fixed noise schedule. We choose the forward noise schedule to be:
o((() (i) -1
This defines the amount of information retained from the original input time step t. The forward transition at each step is defined such that the token x' is preserved with probability (n(()) and replaced with the mask token m with probability (1 at())) For brevity, we drop from the noise schedule and denote by az. The conditional distribution of the forward process is given by:
q x II [l))))x),
q(2)) = Cat (zt; axx' + + )1-) E (1,2,3) (13)
Here, z denotes the corrupted version of token x' at time t, and Cat(-) denotes the categorical distribution over the vocabulary V {0, 1, m.). The vector x" E R is one-hot column vector corresponding to the original token x', and m = [0, O, 1]T is the one-hot vector for the mask token.
To make this conerete, consider the example input 21 (1,0,0) from earlier setup. This corresponds to the one-hot matrix:
0
17
X =
At time t == (i.e., i 1), we have az
Substituting into the forward conditional gives:
((11/a(x1)
0
3
3
Each column of this matrix represents the categorical distribution over V {0, 1, for the corresponding position in the sequence. For example:
The first column corresponds to ziya Cat(O, 31 $
The second column corresponds to z ~ Cat( 0.
The third column corresponds to / Cat(O, 0,
Similarly, compute the following conditionals:
0
0
07
0
((22/a|X1)
q(z)
0
0
1
1
1
31

---

This formulation captures the probabilistic corruption of each token independently according to the noise schedule, blending its original identity with increasing uncertainty (i.e., masking) over time. provides a concrete foundation for analyzing how anchoring improves denoising, especially when key tokens (like x2 in our setup) are retained or inferred with higher confidence.
To illustrate the benefit of anchoring during inference, we analyze a single reverse step in the diffusion process: transitioning from Z2/3 to 21/3- Suppose we observe 22/3 (m, m, m) q(-(x()) i.e., the sequence fully masked this step. Ideally, we would prefer to unmask 2 = [0. 0, 1] to
z [1, 0, O]", since this corresponds to the parent node x² 0 [1, 0, O]T, which is the most informative token in the sequence. Decoding this parent token early makes it significantly easier to infer the full sequence, and conditioning on it reduces the sample complexity of estimating the model's conditional probability tables from exponential to polynomial, as discussed in $A.3.2.
We now compute the likelihood of correctly recovering the important token under both the standard reverse process and our anchored reverse process. Finally, show that anchored reverse process yields a higher probability decoding the important token.
Standard Reverse Process. As described in $2, the reverse process is parameterized as:
po (2 /2/22/))) = q l////////// x'g (22/aa)
Cat Cat(z) ( 2//3), Cat (2//3² x
2/3 # m
(14)
m
2/33 m.
In this case, all tokens are masked at 22/3. To proceed, we estimate X1. (Z2/3) using samples from the data distribution q. Recall that q is supported on:
(1, 0, 0) with probability 0.9, .2 (0, 1, with probability 0.1.
Thus, we compute the predicted token mixture as:
(2/23) = 0011 0.9, ToT, x²(22/3) (0.9, 0,,,
xx(22/3) = (0.9, 0.1,
This concurs with the zero zero-masken masking parameterizatton discussed in $2. Substituting into the reverse conditional (14), we obtain:
70.05
0.45
0.45
Po po(21/3/22/a) =
0.45
0.05
0.05
0..0
0.50
0.50
Each column represents the categorical distribution over the vocabulary {0, m. for zi/a: 2 2 respectively. We now compute the likelihood of decoding the target partial sequence (m, 0, m). i.e. correctly unmasking the important token:
po(21/3 = (m, O, mm|22/3 = (m, m, m)) (0.5) (0.45) (0.5) = 0.1125.
This probability reflects the chance of correctly decoding only the important token using the standard reverse process. In the following section, we contrast this with the likelihood achieved under the anchored reverse process used in ADLM.
ADLM Reverse Process. As discussed in $3, anchored reverse process is parameterized as:
(2) q(z),3 /2//33 (y
Cat(zi /3i 22/3))
if z # m
(15)
Cat (1/8a xx/ (22/3)) }mmm if 2//3
Given an input sequence let the operator A(-) construct the anchored sequence y = A(x)
(x', O, x²). This operator overwrites the second position with the important anchor token [1, 0, 0, while copying the first and third tokens from x. Since the anchor loss (see Eq. (9) is applied only to important tokens, the anchor network is trained to predict y (22/s) = [1, 0, 0 In contrast, the first and third tokens are jointly parameterized and optimized with the denoiser network to maximize the overall sequence likelihood. The denoiser network x(,() behaves as follows:
32

---

If y'p (Z2/a) is unmasked, then x (Zz(z))) simply copies it to the output due to the carry-over unmasking parameterization ($2).
Otherwise, it defaults standard MLE estimate as the vanilla reverse process.
Therefore, the predicted token mixture becomes:
{)((((2))))) = [0.1,0.9,0]* (2((((2))))) = [,,,,0", x3((((22)3)) = 0.,,0.
Substituting these predictions into Eq. (15), the conditional distribution under the ADLM reverse process becomes:
T..05
0.50
0.457
P(o.el (21//) /2/3)) =
0.45
0.0
0.05
0.50
0.50
0.50
We now compute the likelihood of decoding the desired partial sequence (m, 0, m):
Pie.el 11/3 (m, 0, |22/3 (m, m)) (0.5) (0.5) (0.5) 0.125
This is higher than the standard reverse process likelihood computed earlier:
po /3 (m, 0, m)|z2/a = (m, m, m)) 0.1125
This example illustrates how anchoring improves the recovery of important tokens during inference. Similarly, the likelihood of 21/3 = (1, O, m) given 22/3 (m, m, m) higher for ADLM com- pared standard DLM. By prioritizing important tokens like x², ADLM improves masked likelihood modeling and reduces sample complexity of the denoiser.
Practical Considerations. While this example assumes sampling from the anchor distribution followed by denoising, such a two-step pipeline is not end-to-end differentiable due to the intermediate practice, we implement anchoring by projecting the anchor logits through a linear layer into the embedding space of the denoiser. This allows gradients to flow from the denoiser output back to the anchor network though the linear projection, enabling efficient joint training of both the modules; refer to implementation details in SC.1.3.
sampling operation.
Summary. Our anchored graphical model analysis demonstrates the dual advantage of anchoring in language models: (1) reduced sample complexity during training ($A.3.2) and (2) improved likelihood modeling during inference ($...)4). While this analysis was performed on small DAG (L = 3) and coarse time discretizations (T' = 3), the insights generalize to broader classes of DAGs and finer time discretizations. We believe this theoretical understanding complements our strong empirical results across generative modeling ($5.1) and complex reasoning benchmarks ($5.2) in the main draft ($5) and also the Appendix C. We hope it offers compelling justification for the use of anchoring general framework for language modeling.
B Additional Background and Related Works
In this section, we provide extended background and related work relevant to our proposed approach. We focus two model families: diffusion language models ($B.1) and autoregressive models ($B.2), covering their recent developments.
B.1 Diffusion Language Models
Diffusion models are built on two stochastic processes: forward (noising) process that gradually corrupts a clean input x into noisy latent representation z, for E [0, 1]. and reverse (denoising) process that reconstructs x from 21. The effectiveness of diffusion models is primarily attributed to two factors: iterative refinement through multiple steps, and a simple regression- based training objective (Sohl-Dickstein et al., 2015; Ho et al., 2020).
For continuous domains (e.g., image generation), the forward process is typically modeled as an (OU) process that adds Gaussian noise with increasing variance -Soht-Dickstein al., 2015; Ho et al., 2020). the discrete setting, the forward process defined by either: (a) uniform noising, where each token is replaced with another random token from the vocabulary (Austin
33

---

et al., 2021; Lou et al., 2024; Liu et al., 2025; Varma et al., 2025), or (b) random masking, where each token independently replaced by a special mask token m (Austin al., 2021; Lou et al., 2024; Sahoo et al., 2024; Arriola et al., 2025; Wang et al., 2025a; Ou et al., 2025; Nie et al., 2025a,b).
Recent studies (Austin al., 2021; Sahoo et al., 2024; Lou et al., 2024; Shi et. 2024; Ou et al., 2025)
demonstrate that random masking provides improved training stability and sample quality compared uniform noising. In these masked DLMs, the reverse process is trained to progressively reconstruct the sequence from a fully masked input ž (m, m....... m) by unmasking tokens step-by-step. Training is done via negative evidence lower bound objective (Austin et al., 2021), which admits score- based interpretation (Lou et al., 2024) and supports time- uie-inndeeendet parameterization (Ou al., 2025), enabling simplified training and efficient scaling (Nie et al., 2025a.b).
The indepeeeeedd formulation is based on the insight that in masked MLMs, the number of masked tokens implicitly encodes the timestep. Therefore, the denoising network does not require explicit time embedding as input. Our proposed ADLM follows this time-independent parameterization, simplifying the model training and improving scalability without sacrificing performance.
B.2 Auto-Regressive Models
Explicit CoT Fine- Fine-Tuning (GPT-2 baseline). In this baseline, GPT-2 model is fine-tuned to generate chain-of-thought (CoT) reasoning traces followed by the final answer (Wei al., 2022). It serves strong supervised benchmark for comparison, and include several variants in our experimental evaluation. We demonstrate how anchoring can be integrated on top of standard CoT to complement reasoning.
COCONUT (Hao et al., 2024) extends CoT by replac- ing discrete reasoning tokens with continuous latent representations. A GPT-2 model is fine-tuned using a multi-stage procedure that gradually introduces continuous latent "thoughts" between the question and answer. When integrated with our anchoring mechanism, this approach also yields strong performance on symbolic reasoning benchmarks like ProsQA (see &C.2).
COCONUT Soaiin-f-CCntinuuuu--iiiiee
CODI (Continuous Chain-of- Chain-of-Thought via Self-Distillation). CODI (Shen et al.. 2025) distills reasoning steps into continuous latent space, achieving final accuracy of 43.7% on GSM8K (Cobbe et al., 2021). 1). This approach outperforms the previous best GPT-2 finetuned model by roughly 9.6% (from 34. 1..1% % 43.7%) as shown in Table CODI is orthogonal to our method: while it compresses reasoning via distillation, our model explicitly guides reasoning via anchor tokens without relying on distillation. Exploring combinations of both ideas is an interesting direction for future work.
Since CoT and COCONUT are closely related to our work, we provide more detailed discussion of these baselines and their variants in $C.2.
B.3 Token Unmasking Strategies in Diffusion and AR Language Models
A large body of work has explored strategies for adaptive unmasking tokens during the generative process in diffusion and AR language models. These strategies include:
Greedy decoding, where the most confident (i.e., nigh-probability) tokens are unmasked first (Nie al., 2025a.b).
Locked-in sampling, where once token is unmasked, it is fixed for the remainder of the generation (Sahoo al., 2024).
Remasking sampling, which allows previously unmasked tokens to be re-masked and resampled future steps (Wang al., 2025a).
Top-p (nucleus) sampling, where the sampling distribution is restricted to the smallest subset of tokens whose cumulative probability mass exceeds a threshold (Radford et al., 2019: Wang & Cho, 2019; Varma et al., 2025).
Top-k sampling, where only the top & tokens with the highest logits are retained for sampling (Fan al., 2018).
Beam search, where a set of candidate sequences (beam) is expanded and pruned at each step until termination condition is met.
While these approaches focus on selecting tokens based on confidence or likelihood, our work introduces new approach: token importance. Rather than decoding the most likely tokens, which
34

---

often correspond to frequent but semantically shallow tokens such articles or conjunctions, our anchoring mechanism prioritizes decoding informative tokens likens-ttppccllly content- bearrng nouns, verbs, or entities that anchor sentence.
By identifying (using a jointly trained anchor network) and decoding these anchor tokens first, our model improves contextual understanding and enables the denoiser to more accurately recover the remaining tokens. We demonstrate that this stralegy effective across two representative sampling strategies in DLMs: the locked-in sampler (Sahoo et al., 2024), and the remasking sampler (Wang al., 2025a), which internally integrates top-p (nucleus) sampling.
This shift from decoding based on likelihood to decoding based on semantic utility offers a new perspective generative planning and opens the door to further improvements in interpretability. reasoning, and controllable generation. This also opens up an interesting research direction-look ahead planning and reasoning in AR models-as we demonstrate in $C.2.
C Additional Experiments
This section provides additional experimental details on our proposed anchoring mechanism. applied to both masked diffusion language models in &C.1 and autoregressive models in $C.2. We present benchmark datasets, training procedure, ablation studies, and additional quantitative and qualitative results to support the findings discussed in the main paper.
Broader Impact. This work introduces an anchoring framework that improves likelihood modeling and generation quality in DLMs, while also enhancing complex reasoning in AR models. On the positive side, ADLM has the potential to increase the accuracy, interpretability, and efficiency of language models applied to critical domains such as education, healthcare, and scientific research. Its ability to prioritize semantically important tokens may contribute to the development of more transparent and explainable AI systems.
However, these capabilities also carry risks. Enhanced reasoning and generation fidelity may increase the potential for misuse, such as generating persuasive misinformation, reinforcing biases, enabling manipulation. As with other generative models, there is possibility of producing deceptive or harmful content if deployed irresponsibly. We emphasize the importance of safeguards and responsible deployment mitigate these risks.
Reproducibility. To support reproducibility, we provide complete pseudo-code and all hyperparam- eter settings used in experiments. Our training and evaluation protocols are aligned with prior work to ensure fair and transparent comparisons.
Safeguards. To mitigate risks of misuse, we recommend applying standard safeguards such as dataset filtering, usage auditing, and model alignment techniques, including Reinforcement Learning from Human Feedback (RLHF). We also advocate for releasing models and code under responsible use guidelines with access controls and documentation to promote ethical deployment.
C.1 Diffusion Language Models
This subsection provides extended experimental details for diffusion language models. We describe the compared baselines in $C.1.1, outline the training and evaluation benchmarks in $C.1.2, and present implementation details along with additional results in SC.1.3. Finally, we include qualitative examples and analysis of samples generated by ADLM in 1C.1.4
C.1.1 Compared Baselines
We compare ADLM against broad range of autoregressive, diffusion, and hybrid (autregres- sive+diffusion) language models. Each baseline is evaluated under the same training and evaluation protocol as ADLM, using identical tokenizers, data splits, and sampling steps. Hyperparameters are adopted from official implementations tuned for faimness when not explicitly provided. Below, we summarize each method and provide links to source code where available:
Autoregressive Transformer (AR): A standard GPT-style transformer trained using next- token prediction. We use the architecture from the MDLM repository."
Phttps://gite www/kulesooo..reeeeooooo
35

---

SEDD (Lou et al., 2024); A discrete diffusion language model that denoises using a neural network trained to approximate score entropy. Source: https:// //gtt//b/..//./////// com/ /ouaaron/
Score- Saare--ntrooy-Dieee
MDLM (Sahoo et al., 2024): A masked diffusion language model trained with the NELBO
objective (4). MDLM uses locked-in sampler where tokens, once unmasked, remain fixed.
Source: https: : //gtthub.. com/kut monnkulesoov-eee goruggrouuuuggu /mmpmmmmmmmmmmmm
MD4 (Shi et al., 2024): A masked diffusion model for discrete data trained with the NELBO
objective (4). Source: https: : //gbttu......//gg//gg//..g....
thub. com/ google e deelee-deepndddd /d/m44
BD3LM (Arriola et al., 2025): A hybrid model that combines autoregressive generation
with block-wise diffusion- based refinement. This enables parallel sampling in each block and sequential across blocks. Source: https: : /tggtg/gbbb/////////// com/ /ww//elllleoolleeeee geovvgrropp///o group/bd3lmm ReMDM (Wang et al., 2025a): Extends MDLM by incorporating a re-masking sam- pler, which allows the model to re-mask and re-predict tokens during inference. This re-masking sampler improves generated text quality. Source: https:/ htpp::///ithbbb com/ kuleshov- proup/remdm
GIDD (von Rütte et al., 2025): GIDD introduces a general interpolation between masking and uniform noising in discrete diffusion models. As a concurrent work ReMDM, addresses a key limitation of MDLM (i.e., the inability to revise tokens once unmasked) by allowing previously unmasked tokens to updated during inference. This flexibility enables the model to iteratively correct its own mistakes and refine its outputs more effectively. https: //g thub. com /ttp///teeeetdidddeee
Source:
DFM (Discrete Flow Matching) (Gat et al., 2024): DFM is flow-based approach for discrete data that replaces the diffusion loss with flow matching objective (Gat al., 2024). At the time of writing, the official implementation was unavailable. We instead reference a simplified public version capturing the core ideas For evaluation, we adopt the DFM sampler built on top of the MDLM base model, available in the ReMDM repository:" Forward- -ackaddd (FB) (Campbell et al., 2022): FB is corrective sampler that combines forward and backward transitions to improve reconstruction accuracy (Campbell et al., 2022). It is applied as a post-hoc sampling method on top of pretrained models such as MDLM. We use the publicly available implementation from the ReMDM repository:".
All models are evaluated using the same number of neural function evaluations (NFEs), training tokens, and sampling steps where applicable. This ensures a consistent and fair comparison across all baselines, allowing us to isolate the effect of anchoring on model performance.
C.1.2 Training and Evaluation Benchmarks
We evaluate diffusion language models on two fronts: (1) generative modeling quality and (2) zero-shot generalization to downstream tasks.
For generative modeling, Billion Words (LMIB) (Chelba et al., 2013) and Open Web (OWT) (Gokaslan & Cohen, 2019).
two widely adopted masked language modeling benchmarks: One
These datasets provide large-scale and diverse natural language corpora for evaluating the ability of models to understand and generate natural language.
For downstream evaluation, assess zero-shot likelihoods seven standard benchmarks spanning commonsense reasoning, scientific language, and formal language domains. These include Lambada, PTB, WikiText, LM1B, AG News, PubMed, and ArXiv. Below, we briefly describe these benchmarks.
One Billion Words (LMIB). We use the LMIB dataset (Chelba al., 2013) which consists of news crawl data collected by Chelba et al. (2013). The dataset is released under the Apache 2.0 license. Following prior work (Sahoo et al., 2024; Wang et al., 2025a), we train ADLM for IM steps using batch size of 512 and context length of 128. which corresponds to approximately 33B tokens. larger variant trained for 2M steps sees approximately 65B tokens. The autoregressive baseline is trained for 0.5M and 1M steps respectively to match the total number of tokens processed. For evaluation, we use the standard LMIB test split.
http:///pt.tttooo///////////oaaa benchmark/
36

---

OpenWebi Text (OWT), We use the Open WebText dataset (Gokaslan & Cohen, 2019)', which is public reproduction of the WebText dataset originally used in GPT-2 (Radford et al., 2019). It consists of web content extracted from high-quality Reddit URLs. The dataset is licensed under Creative Commons CCO license ("no rights reserved").
Zero-Shot Evaluation Benchmarks. To assess generalization, we perform zero-shot evaluation on seven diverse benchmarks covering language understanding, scientific articles, and long-context reasoning. We measure the perplexity on the validation sets of the following benchmarks:
Lambada (Paperno al., 2016): This benchmark evaluates the ability of language models to predict a target word based on a broad context. Each example consists of a short narrative (context) followed by a target sentence with its final word omitted. Unlike typical language modeling tasks, solving Lambada requires understanding of longer-range dependencies beyond just the last sentence.
Penn Treebank (PTB) (Marcus et al., 1993): A classical benchmark for language modeling, which is used evaluate syntactic fluency and local coherence in generated outputs.
WikiText (Merity et al., 2017): A long-form language modeling benchmark based on Wikipedia articles. This evaluation emphasizes factual correctness and world knowledge. AG News (Zhang et al., 2015): A text classification dataset with four categories, which is designed to predict the topic label given a news headline or excerpt.
PubMed (Cohan et al., 2018): A benchmark based on biomedical research articles, cu- rated for studying text summarization. Each example consists of document body and a corresponding summary, typically derived from the abstract or conclusion section.
ArXiv (Cohan et al., 2018): A benchmark similar to PubMed, but based on research articles from the ArXiv repository across diverse scientific domains. Summaries are typically derived from the abstract or conclusion sections of the papers.
Licenses and Usage. All datasets used in this work are publicly available and licensed for research use. OWT is distributed under the Creative Commons CCO license, and LMIB is under the Apache 2.0 license. All baseline models (e.g., SEDD, MDLM, BD3LM, ReMDM, GPT-2) and implementations are sourced from repositories released under MIT or Apache 2.0 licenses.
All reported perplexity values for diffusion language models in this paper are upper bounds, rather than exact likelihoods. This is because diffusion models do not compute normalized likelihoods in closed form due to the intractability of the reverse process. Instead, we follow prior work (Lou et al., 2024; Sahoo et al., 2024: Arriola et al., 2025; Wang et al., 2025a) and evaluate diffusion models by estimating a negative log-likelihood upper bound via importance sampling or ELBO-based objectives. In contrast, perplexities for autoregressive models are computed exactly via standard token-wise cross-entropy. As such, direct PPL comparisons should be interpreted with this distinction in mind.
C.1.3 Implementation Details & Additional Results
Architecture Details. We apply learnable linear projection within the denoiser network (.) that maps the anchor logits y (2:) directly into the embedding space of the vi-transfomee. This modular design avoids explicit decoding and re-embedding of anchor predictions, enabling fully end-to-end differentiability between the anchor and denoising networks.
Although the ANELBO loss (7) is defined over all tokens (xf in the sequence x, only the masked tokens contribute to the denoiser loss via (y (2,)). and only the important tokens contribute to the anchor loss via (z,). In practice, this is implemented using carry-over unmasking (Sahoo et al., 2024) or by multiplying indicator functions in (7). Specifically, we use 1 [z/==)) to mask the denoising loss, and 1 (Ax))()())))) to mask the anchor loss.
Both networks follow the DiT-Base architecture (Peebles & Xie, 2023), which is consistent with SEDD (Lou et al., 2024) and MDLM (Sahoo et al., 2024). The anchor transformer network uses 12 DIT blocks. The denoising network uses 6 DIT blocks. Each block has hidden dimension 768 and 12 attention heads. The input sequence length is 1024 for OWT and 128 for LMIB. We use the same AdamW optimizer for both anchor and denoising transformers with learning rate = 3e-4 and and no weight decay.
hattp://Skyyiooo007..o gith.u. http://gtt.abbboo/eeeeeeeetttrrr
37

---

Algorithm 1: Anchored Diffusion Language Model (ADLM)
Input: Anchor network (+).), denoising network x²(),, number of steps T, noise schedule Ott remasking schedule dt
Output: Generated sequence Zg
Initialize ZT &- (m, m. ...,, m)
P Fully masked sequence
2 for
Too to do
(- i/T, 8 =
Compute noise schedule: deg, ax
Compute remasking schedule: dr [0, ofTi Compute anchor transition using noisy sequence:
b Follows Eq. (9) (Wang et al., 2025a) Eq. (5) in $3
Cat(z (1 - o,ly' + o,m),
z + m
r(y.lll =
Cat(z':: 0u-(1-or)o, los y' (z)) 1-a, 1-a,-a0 m),
z
1-a,
1-ac
Compute inference posterior using anchored prediction y&p (zr):
Eq. (6) in $3
Cat (1 ox)x² o.m). Cat(z'; (1-01-a)) x (y 1-ac
z +
q(2) x (yr((z)))
1. m)
z
1-0c
Sample z! ~ q(zz)2z) x (z)()))) for all E {}
Update 2, f- 2s
10 end
return ZO
Table 6: Evaluation metrics on OWT (78B tokens) for our ADLM algorithm across different (a)
anchoring loss coefficients (7) and (b) anchoring thresholds (T).
(a) y
3e-2
3e-3
3e-5
(b)
1
5
10
20
NLL (4)
3.1084
3.1055
3.1663
NLL (4)
3.1429
3.1055
3.1358
3.1770
PPL (4)
22.386
22.321
23.719
PPL. (1)
23.171
22.321
23.008
23.976
BPD (4)
4.4878
4.4803
4.5680
BPD (4)
4.5342
4.4803
4.5241
4.5835
Sampling Implementation. We provide the pseudo-code for ADLM sampling in Algorithm 1, which employs the standard locked-in sampler (Sahoo et al., 2024) with remasking schedule ar (Wang et al., 2025a). The locked-in sampler is special case obtained by setting a, = 0.
Implementation Details for OWT. As OWT dataset does not provide official train/test split, use the splits used in prior work (Sahoo et al., 2024) and train ADLM for 1M and 2M steps with a GPT-2 tokenizer, batch size of 512, sequence length of 1024, and linerr diffusion schedule. This results in approximately 262B (1M steps) and 524B (2M steps) tokens. The AR baseline is trained for half many steps under the same configuration ensure comparable number of tokens seen. We use the last 100K documents (held out during training) for evaluation.
Ablation Study for OWT. In Table ((b) (262B tokens) of the main paper, we show that simply using our two-stage architecture without the anchor loss already improves test perplexity from 23.17 to 21.79, validating our architectural choice. Adding the anchor loss further reduces perplexity to 20.62, highlighting the effectiveness of anchoring in likelihood modeling.
To better understand the role of anchoring, we conduct an ablation study on OWT using 78B tokens over 300K training iterations. We evaluate the impact of two hyperparameters: the anchoring loss coefficient (7) and the anchor threshold (r), discussed in $3. Table 6 reports results across different values. In Table 6(a), we observe that y = 3e-3 yields the best trade-off across negative log likelihood (NLL), perplexity (PPL), and bits per-dimension (BPD), outperforming both higher (3e-2) and lower (3e-5) values. Similarly, Table 6(b) shows that т 5 achieves the best results, outperforming thresholds such as 1, 10, and 20. Based on this study, we use 3e-3 and 7 5 as our default configuration across all experiments. With this default configuration, the training loss and validation PPL per iteration is shown in Figure
38

---

5.5
Training Loss
MDLM
Validation PPL
50
MDLM
4.5
ADLM (ours)
40
ADLM (ours)
4
30
3.5
trainerjglobal
0
500K
IM
1.5M
500K
IM
1.5M
Number of iterations
Number iterations
Figure 3: Training loss and validation PPL versus number of iterations on OWT. We train both MDLM (Sahoo et al., 2024) and our ADLM model for 2M iterations (524B tokens). As discussed in $4, anchoring improves the sample complexity during training, resulting in faster convergence and lower validation perplexity. While the anchor loss is part of the training objective, we only visualize the NELBO here for a direct comparison with MDLM.
Implementation Details for LMIB. The experimental setup follows prior works (Sahoo et al., 2024; Wang et al., 2025a). For LMIB, we follow the same setup as OWT, except a shorter sequence length of 128 tokens and the BERRT-base-uneased tokenizer. Since LMIB includes an official test split, use for evaluation. We use the same anchoring configuration (y = 3e-3, T = 5) as in OWT.
Remasking Evaluation. For the results reported in Table 3, generate 5,000 samples using sampling steps ranging from 128 to 4096. We evaluate each set of samples using MAUVE score, GPT- 2 perplexity, and entropy. All hyperparameters follow the settings recommended in the remasking sampler developed in ReMDM (Wang et al., 2025b). For clarity, we report the exact values used in experiments:
sampling steps: Number of sampling steps used per generation; values range over {128, 256, 512, 1024, 2048, 4096).
p: 0.9 Top-p value used nucleus sampling.
1y: 0.02 Parameter used in ReMDD strategies.
fon 0.55 Activation time for remasking in the ReMDM loop.
loft: 0.05 Deactivation time for remasking in the ReMDM loop.
O'on: 0.9 Fixed masking schedule (xfonn used in the ReMDM loop.
Discussion on Anchoring Attention. Anchoring is completely different from the traditional attention mechanism. In the example shown in Figure 1, attention layers operate only over the tokens available in Ze A key limitation arises when important tokens mechanisms are unable to access or reason about these missing tokens. In contrast, our anchoring mechanism explicitly predicts important tokens its output, which can then be attended to by the downstream attention layers of the denoiser network. As such, our method is orthogonal to attention mechanisms. fact, anchoring provides an efficient way reduce the sample complexity of these layers and improve training as discussed in $A.3.
maakedd-ttanda
attention
Discussion on Size Generative Perplexity. In Table 7, compare the generative perplexities of ADLM against a range of autoregressive, masked language, and diffusion-based models, all evaluated using GPT2-Large over 1024 unconditional generations. A key advantage of ADLM lies in its ability to produce high-quality generations with significantly fewer parameters and fewer sampling steps than prior diffusion language models.
Specifically, ADLM achieves perplexity of 15.7 using 4096 sampling steps, outperforming Plaid (Gulrajani & Hashimoto, 2023), which reaches perplexity of 19.7 despite using nearly five times more parameters (1.3B vs. 293M). Even at lower sampling budgets (e.g.. T-2048), ADLM maintains strong performance (20.1), reducing the performance gap with GPT2-medium (12.4) and outperforming SERT--arge+Gibbs while using at least 40M fewer parameters. This indicates that anchoring enables more efficient denoising compared prior diffusion language models.
Moreover, ADLM demonstrates a consistent reduction in perplexity as the number of sampling steps increases, validating the role of iterative refinement in enhancing sample quality. These results
39

---

Table 7: Generative perplexities evaluated over 1024 unconditional generations using GPT2- Large (774M params) the evaluation model. ADLM significantly outperforms prior masked and diffusion language models under comparable sampling configurations. Notably, it surpasses Plaid (Gulrajani & Hashimoto, 2023) while using only ~20% of the parameters (5x smaller size), and nearly matches the performance of GPT2-medium despite having ~00 fewer parameters.
Model Type
Evaluated Model
Params
Gen. PPL (L)
Autoregressive
GPT2-medium (Radford et al., 2019)
345M
12.4
BERT- LaRge + Gibbs (Wang & Cho, 2019) (7'=2048)
334M
487.0
Masked Language
BERT-Larg + Gibbs (Wang & Cho, 2019) (1'=65336)
334M
28.7
Plaid (Gulrajami Hashimoto, 2023) 109)
1.3B
19.7
SEDD-medium (Lou et al., 2024) (7'))088)
424M
27.3
Diffusion
SEDD-medium (Lou et al., 2024) (1'=1000)
424M
31,95
MD4-medium (Shi et al., 2024) (0'=1000)
416M
27.51
SFDDDsmall (Lou et al., 2024) (7=1000)
165M
42.94
MD4-small (Shi ct al., 2024) (7=1100)
165M
33.16
MDLM (Sahoo et al., 2024) (locked-in, 7=1000) 7=10000
170M
44.2
ReMDM (Wang al., 2025a) (remasking, 1004)
170M
28.6
GGM (Varma et al., 2025) (T=4096)
387M
19.5
ADLM (ours) (locked-in, T²=1000)
293M
32.9
ADLM (ours) (remasking, 7=1000)
293M
26.8
ADLM (ours) (remasking, 7=1024)
293M
25.1
ADLM (ours) (remasking. T-20488
293M
20.1
ADLM (ours) (remasking, T= -0066)
293M
15.7
highlight the sample efficiency and scalability of our approach, making it a practical alternative to large autoregressive parameter-heavy diffusion models for high-quality text generation.
Discussion on MAUVE Evaluation with BERT Embeddings. We recompute MAUVE scores using bert-base-a uncaedd embeddings (maximum sequence length = 512, T 4096) to verify the robustmess of our evaluation. Compared the best-performing baseline ReMDM (Table 3), our method achieves 7% higher MAUVE score, confirming that ADLM generates more coherent and human-like text under both GPT-2 and BERT embedding evaluations.
Table 8: Evaluation OWT. MAUVE scores computed using bert sest-baae-uncaseee embeddings with maximum sequence length = 512 and T 4096.
Method
MAUVE (t)
Gen PPL (4)
Entropy (f)
RcMDM
0.702
17.6
5.20
ADLM (ours)
0.754
15.9
5.19
C.1.4 Samples Generated using ADLM
In this section, we present qualitative samples generated by our method ADLM and compare them with MDLM (Sahoo al., 2024). We illustrate outputs under two sampling strategies: the locked-in sampler (Examples and IV) and the remasking sampler (Examples II and III). As shown, increasing number of sampling steps and enabling remasking leads to more coherent, paragraph-structun generations. These qualitative observations align with the quantitative results reported in Table 3, where ADLM achieves significantly higher MAUVE scores and lower generative perplexities than the existing diffusion language models.
Example-I. This example shows sample generated using ADLM with 1000 sampling steps and the locked-in sampler from MDLM. While the generated content is largely coherent, it lacks structured paragraph breaks and clear discourse delimiters. The result reads continuous stream of thoughts rather than segmented essay. Notably, even this partially structured output is both qualitatively and quantitatively (Table 3) superior to MDLM (Example-IV), which fails to organize ideas into meaningful paragraphs without breaking logical continuity. In subsequent examples (Example-Il and Example-III), we show that increasing the number of sampling steps and incorporating remasking in ADLM significantly improves human-like text generation quality.
40

---

her life Example-1: Output from Anchored Diffusion Language Model using locked-in sampler
<|endoftext(>> knows they didn't have the answers to answer this. It was solid piece of early Survivor and people glossed over it. Sadly guess that's communityat, and
where the Tribe go. It wasn't family. It was based that was everyone else." " Probst Steve Wataon.
Never as good as our pundit, we we'll repair the damage ve've done!
Tell Random Story
A large facet of the new season will be to tell a random story, not sure if it's how the players are portrayed. But it's fairly obvious that this specificity will be replaced by community. The players will be given history, the people we've seen seemed less and more surprising as the episodes went.
The host will highlight Tukuku Tshivu respect the senior group and ditches the Amazonians in their barbaric ways. After that we will see Toatia; some've described Tasha as someone who listens to the players despite scheduling theirs, and she's a pretty Open Tribal so far.
captain that's always tried to
"And there's Andrea the Polarnesian," Probst continues, remembering the wood thinner swimmer named Andrea whose entire life she's been challenging both Rossheimerk and the predecessor to make IHOPE more racially inclusive. Paris, so much for standing up to her at the beginning, was willing to take no down. This shoved that the guild wouldn't just be her. Andrea had been waiting for free play, few months and months, and she stayed. But boy did she know that players with auch wide did require a fierce lane for attention. There's a reason she was ripped from her alliance for a reason we'll likely see players end up in separate struggles with. Paris.
to free play the beginning
For those players who can't wait to the transition into the beginning, you waited for three
the big reward before the game will be in reality. weeks for example, you'll still need fully footed shoes to get your reward, you von't get old shoes. The urgency of you signing long enough to get this reward makes big of social sense because that's the way the society goes, as the rewards are physical. The maximum rewards have yet to determined but we'll all keep tabs on the Breck King for progress.
Go Jaeger, the Trienge! It's some fun! This a system will work well but not for an individual tribe in particular. With two challenges means lot of people only
full one day of fun and rewards. And if you haven't yet seen a single challenge, sure not the one played by the KaZinaks. Julie, Caleb, and Shirishio Polyhrine have chosen creative challenge themes for them, albeit stoic winter difficult.
Winter's captains might get less access have be shallow session where everyone goes over to hash out rewards conflicts. It won't take place mid-season like winter, and it won't go organically. Book dramas are inevitable, but reboots have less incentive to fail if the show does, and network commitments can to put everyone peoples hustle in sorech don't. I'm not sure this makes up for these two things.
gameplay, but it doesn't
From What Built
Yes for the record, Jason and Miss were from a school group on the controversial side of history. Now Jay and the photographer will have unique way of representing the tribe while keeping the story relevant. Though the cast will have something to do with it (gohan, skinnyfoots), there will be sone mismatched players who generally won't fill the ample, talented roster. Which they clearly will. Anyone stuck trying to play too heavy the other sides of the story? Time out is Rachel Probst, who only worked one Survivor Bannett final with EMA, '97: Sandra's power was barely that of Bailey Devevaorda, and Candice Burke was Pats to name a fev. Like Raj wound Katie's name, her tribe will have an extraordinary impact on all of the stories and alliances.
Thai's Olympic tribe another portfolio for the series. Last season, Zinke Group managed to represent 13 Pacific nations on the world stage by visiting the indigenous peoples of them in various languages (Lahonese
41

---

for example) and including First Peoples Tribal Councils; half of Taiwan's "transformations were so successful that they typically received their return for their contributions. It was creative action, and even if mn<deenoott
The output generated by ADLM exhibits coherent structure and articulates thoughts in paragraphs like human-wrine written text. this example, we use our pre-trained ADLM base model and apply the remasking sampler (Wang al., 2025a) with 2048 sampling steps during inference. The model is able to maintain logical progression across long sequences, keeping together facts, statistics, and historical context into coherent narrative.
Example-II.
This qualitative output complements the quantitative results reported in Table 3, where ADLM achieves the highest MAUVE score among all diffusion-basee language models. MAUVE is known to correlate well with human evaluation of fluency and coherence, and this example provides evidence supporting ADLM's superior generation quality.
Example-II: Output from Anchored Diffusion Language Model using remasking sampler
</ndoftext] >> health and demographics compared to the EU edit
Historical European birth rate map
for EU and European men and immigrants(4 [4]
[5]
B.R.E.O, [6] Birth rate by country Birth Rate co- Interval Interval 1910
50.4% (3. 17) 4. 4.8% (12.6) 1910 61.5% (4%) 1920 51. 1% (3.0% 0%) 4. 8% (12.6) 1920
63.5% (2.5%) 3.4% (1925) 59.4% (2.2%) 1930 62.7% (2)1%) 3.4% (1930) 60.2%
(2.0%) Other caregiving* 1910 49. 1% (3. (3.0%) 4.8% (12.6) 1910 63. 1% (2.8%) 3.4%
(other) 1920 62.7% (2.5%) 1930 60.5% 1%) (b) First
(c) Birth: average is for life and
(for girls, for boys, both to sexes. d) Sex only and for people can't be combined, have to be under 15 years of age to have a means to measure motherhood) Oldest age. In the mid-19th century, at least half the population had abortions before sterilization had taken. Abortion increased more rapidly the twentieth century (nearly half that time, 1.6 million births, infant mortality over 2 million during the period up to 1960). In a study of Austrian prisons, 50% of the prisoners under the age of about 20 had been conscripted maltreated for purposes before 1882. Even in cases where there was thorough investigation, the participant had to remain in cruel conditions, including medical experimentation, electric lashings, etc., while pregnancy rates were very low[ []. The sterilized prisoners were not only much more likely to have babies than the prisoners with pregnancy; they less likely have abortions. Researchers found that the number of births to sterilized prisoners between 1882 and 1930 was historically highest in the Czech Republic.
In Egypt, the penal code threatened local women and men with death if they didn't marry foreigners. Jihadista were alaughtered and people to flee out of Egypt's borders. Likewise, foreigners women were forced to marry men from Egyptians as well. The Roma continued to rise, creating an informal climate and assimilated life in the country. Soon Egypt teemed with lesbian
forced
abandoned by migrants shortly after departing for the country. This ethnic seeds played role widely agreed that this prejudice towards foreigners, and other groups, arose among Orientals and had little influence at the time, probably because of the security of limited exchanges between the two peoples.
decline of the Ottoman Empire.
England, it was a family tradition to ferment wine up until the middle the 19th century. After defeating the crown in 1791, the Shelburne brothers flattened the cladding of the Osgoode Hill to form pot for setting in their wine cellar; in the process, they extracted quintense from wine. Thus in the first half of the nineteenth century, the of preglycerine only lasted three thousand years, when any other taxes were levied. Poland, torus
42

---

helped prevent tuberculosis before it became widespread, as the pilgrim resisted wine, showing evidence of the disease. The use of torus accounted for the extinction of European flu.
Prevention meaaures edit
Pumpa and band vein pumping apparatus
Because of increased RME and MMB in the central 20s researchers about examining how physical physiological factors might affect pregnancy and had been done the epidemics, there were two types of PNEA delivery devices used: cardiac pumps and ECG band pumps. The former was placed in the blood vessel, through which 20 mL/mm flow of gold tubing 70,744 744 feet (21,131 m) rated for blood pressure and had continuous output of
its prevention.
(100 yps) was produced and blood pressure measured. Approximately 7, 7,000 km (3,200 mi) of gold tubing was used for each of these two devices. The hg pump that was placed in the patent vein could go from 5 (1.8 in) to band pump had the same potential but the added disadvantage of releasing gas too hard. More than 100 million births were prevented by these pumps every year. [8]
(1.5 in) of tubing to 3,000 feet/1,
(2,100 m/720 ft).
Figure Heart signs fetus during conception of the mother.
The gauges measure blood vessel pressure and blood pressure for 24 h during the first month (320137 weeks) after conception, then sample cylinder was placed endoftettt
Example-IIL In this example, we use 4096 sampling steps for ADLM with remasking sampler. The generated text demonstrates strong discourse-leve coherence, well- well-structured paragraphs, and natural transitions between topics. For instance, the model begins with a detailed sports commentary and transitions smoothly into socio-political news report. These shifts are logically segmented with paragraph-level breaks and consistent timelines (e.g.. 18: 00 PST, 30:00 PST), giving the impression of human-authored journals. This qualitative improvement aligns with the quantitative results reported in Table 3, where ADLM achieves the highest MAUVE score among existing DLMs.
Example-III: Output from Anchored Diffusion Language Model using remasking sam- pler
</endoftext/> were pretty good in that regard. Dwight Howard chipped in 15 points, and wh to argue that defense doesn't deserve stat of the award?
Stephen Curry, who was on the floor soon got there, finished with 27 points. Wesley Matthews put in a great effort as well.
Washington (Curry 30-31, Wall 80-78, Wall 82-7)
I listened more this game than originally planned. The Wizards took control of the game when Otto Porter scored 22 points.
Takeaways:
The Wizards took control the game during the first quarter, mostly courtesy of Dennis Schroder and Shaun Livingston. In the second minute, Stephen Curry hit a perfect layup that bounced to the hoop for dunk, Durant scored two quick points make 15-7 with about minute Jameer Nelson hit a layup in the corner from range to make it 18-9.
minutes left, and in the third
18:00 PST
Wall and Marcus Thornton each dribbled their way through traffic and made two threes, the first by Andre Drummond with three minutes left to record his
43

---

the second by Kevin Durant with five minutes left. However, both shots were blocked at the basket. Kevin Durant missed most of the rest of it with an ankle injury, and Austin Rivers, getting last-second shot off him pretty badly, tried knock the ball down, but couldn't the ref just waved it back. Then Rivers shot, then Thornton drove his way in front Durant, and Wall had to to knock Curry's first shot free. This was probably the highlight of the game; the defense wasn't matched with the offense very well against the Thunder from here out.
10th career triple-double,
reverse dribble move
Halftime:
Oklahoma City gave up 10 points for coming up in the final minute, and Zach LaVine came off the bench. In the fourth minute Russell Westbrook in a low drive to post up screen while the ball was on the floor, but with 10 seconds left in the game, Zach LaVine stepped onto the floor and slammed the ball in the basket for the bucket.
30:00
Tony Allen was brought in for Monta Ellis, and he just could not start. He tried his way to the corner for 3-pointer, but he was shot. Nerlens Noel quickly got up and knocked it back down low, and Allen back the stands the arena, knocking the ball
high and out bounds.
Oklahoma City's bench never stopped suiting the game went on. This is team that is making eepp in the NBA, with DeMarcus Cousins and Chauncey Billups leading the way, and it's just the right thing do such crazy situation. <lendoftext (CN) Hundreds of Seattle residents Tuesday marched to Columbia City to protest the construction of Tesla's new headquarters in one of the largest nationwide demonstrations in years, Seattle police clashed Washington State University.
with student proteaters at park
Thousands of the student marchers gathered at around 2:30 p.m. in Seattle, capital of Western Washington as they confronted riot police carrying rocks, rocks bottles, and gas masks. Large demonstrations of this kind are nearly every weekly the sprawling city. Teala CEO Elon Musk said he didn't know how much the new headquarters could cost him but that it would be enough for the car company to complete its plans. However, the cost estimate is unknown. In the debating point, the city of Seattle, state infrastructure agency, the Netropolitan Transportation Authority and the thousand-plus protesters wait for judge's approval to assess possible damages.
Tesla's headquarters is the second-tallest office building proposed in Greater Snohomish County. It is planned on the Seattle Pacific University campus. Tesla also has 14-story supertall building on Mercer Street that would also include mixed-use office space. There also construction a 50,000 square foot garage that is expected to open next year. There is also parking garage on the 8.7 acre property. No residences are listed in Tesla's property plan approved by federal agencies.
Tesla and company officials are expected to have the newest office building at 240,000 000 square feet made for public access public, once it's slated for completion. The company is already in court for waiver that allows it to maintain the building because it is in park. A lawyer for the company said on April that a judge in Seattle expected rule on the waiver, which would allow the building to be enabled. This is one of several lawsuits that have emerged in recent weeks since Tesla's new headquarters was announced in March. Across the endofexttt
Example-IV. For completeness, we also include samples generated using MDLM (Sahoo et al., 2024). In contrast to ADLM,, the generated lacks coherent paragraph structure and clear discourse delimiters. The output tends to drift semantically and fails to maintain logical continuity across
44

---

segments. This qualitative degradation is consistent with the lower MAUVE scores and high Gen PPL reported for MDLM in Table 3, reflecting lower human-like text generation quality.
Example-IV: Output from Masked Difftusion Language Model using locked-in sampler

> told local reporters. "Any interlocutor "UCU
useful for formal
consulting. "CCU basketball coach, classmate of three, David Irving, is teacher at Memorial High and is among the same players who recently won his credentials in PE Illustrated for the category. Four longtime players make such notable contacts: two career elons (not Columbus native champ Bobby Pulfres; Erie it's Duane Wright than Vin Diesel) and he's the current fourth student the entire campus, as far as entrance per pupil is well. According to Wright, Shaw was about to thirsty with traders celebrating with an invented flavoral beer and the extra muggers when he approached the team with shot Irving, something he was doing while across on Thursday acquired Colt rifle.
spoke he tried to reach with his fists and pose to look good and enhance his visual presentation. During a photo shirt Shaw took on the stool, his head fell like the head of fungus straight for hanging liquid. He could not believe the team were able to get Strough."Thhrr
afternoon
around six seven guys and somebody takes shots," Aggaeed says, "and once they got ready they sat beside them laughing for a bit. And everyone was super friendly. They usually get been late and then it would be 104 o'clock while we went... .It surprising people a lot for the guys here. UCU quarterback Jeffrey Zifits calls him group of hives stirring spirits. "This group meant for the senior players and one of the best young performers,"thh 'the team or the officials or coaches or shatever, but it's not meant for Gasta-Grader," Zifit says. "You could see it first hand each time we went fishing with fans... It wasn't just people throwing "It's the biggest thing they did after third windows and they put it out there," Zifits, "we didn't let them but brought more energy here. When we came showled them it was like a CFL game. The fun played with energy and it's great. "It is definitely in the vein of the spirit of cabo, kind of bar that lets people in two buckets and for Wright, if you can't squeeze your 1ip while golfing, go bouling and play "You expect to be at the conference in this room that vacillums with everybody else," Wright says. "So, just remember, and our motto in a group is 'Reason, when it comes to material Value, we're a group and all our resources are all being used by the future promote the future. It works out. They do it by laughing. "Pulkiie" "Pulkies" students really do love coming back, says Shaw, then a member of the PE group. "Jefferson will still come back and brilliant and all that. 'We can't let them shut them down. If it goes their way, they are not letting us down.' Working title: Mentors, I need to get out here we're all together with students to make it through this. "PS<| / |>By Colin Collins Piper for PRINT CLOSE British university governors unveiled the cancellation of student-helle protest in defi of the declaration it government has gone into pitch to the Liberal Democrats sack it for several hours for its register of aupport for atheists.
"Yys."You
The university's slaughter of Scottish-ent eerrriihenncmmmseingg religious figures and included tacit approval by both houses of parliament before the Niche government was in administration.
It is understood that the load followed the closure of the university's reserve vice chancellor subsequently who died Parkinson's syndrome as his wife's tombstone was on fire.
Catherine Baird of Parliamentary Coalition for Government (Scrabble) leader Lassie Mann joined further the condemnation of the stalls. Ma Baird said: "Good thing the studenta approached the univeraity and they were confident that you found much solace.
"It's a sad day in British life and I think it would be tragic if we ever had this again but this is a very simple decision, to be clear." Scalric-chancellle Jack Major who attended the protest told press conference: "It has been very challenging but would like to later spologise personally in the language of protesters of what happened.
45

---

"The Department Students and Service at the University California, Berkeley was placed following people's request and it is no surprise that the university is now in administration but those schools needed to be confirmed afterwards that they would face disciplinary action.
Directors Major aaid: "I'm minority Instructors were very disappointed about the protest within the university and the <iveeraity</endooteett))))
C.2 Auto-Regressive Models
This section provides an extended discussion of our anchoring mechanism as applied to autoregressive language models. We begin by contrasting standard autoregressive training with our proposed anchored autoregressive training framework. In $C.2.1, we outline the AR baselines used for comparison. In $C.2.2, describe the training and evaluation benchmarks. $C.2.3 provides implementation details necessary for reproducibility. In $C.2.4, we demonstrate the benefits of anchoring for likelihood modeling on the OWT benchmark. Finally, $C.2.5 presents results math and logical reasoning tasks, showing improved reasoning capabilities due to Anchored Chain-of- Thought (ACoT) fine-tuning.
Discussion on standard AR and anchored AR models. Figure 4 illustrates the standard training setup for autoregressive large language models (LLMs), where each token is predicted based on its left context. This sequential decoding lacks structural guidance to prioritize important tokens.
In contrast, as shown in Figure 5, we decompose LLM training into a two-stage process. In the first stage, an anchor network used predict the likelihoods of important tokens (e.g., 'cat' and 'dog' higlighted in blue))-effrrre to anchor tokens or [ANT] condi-conit on the preceding context. An anchor token is not necessarily the next token in a sequence. Its prediction is supervised using a cross-entropy loss applied only at the anchor position.
In the second stage, a lightweight autoregressive LLM (half the number of transformer layers used in Figure 4) is trained using the standard next-token prediction objective, but conditioned on the output logits from the anchor network. Rather than sampling tokens and re-embedding them, we project the anchor logits through linear layer and feed the resulting representations directly into the LLM transformer, similar to the ADLM setup discussed in $C.1.3. This allows gradients from the L.LM's output to backpropagate through the projection layer to the anchor network, enabling joint training. Importantly, this anchoring mechanism allows the model to "look ahead" by leveraging important tokens in sequence, which improves reasoning without significantly altering the decoding process.
Formally, our anchored autoregressive training loss becomes:
L
L
LAZR (Mox(V/,5 :p) = -Ex-ng
Σ
log Pe (x"1x)"--(xxxx--))) - -xExmn
Σ
log & (V'(
LI=2
11=2
where Y = A(X) is the sequence of anchors obtained through the operator as in ADLM ($3).
C.2.1 Compared Baselines
We compare our method against diverse range of latent reasoning and chain-or of offthhuuht (CoT) approaches. We also include baselines that do not use reasoning traces during training. To ensure a fair comparison, we use the same base implementation from prior work (Hao et al., 2024) and integrate our anchoring mechanism into the identical multi-stage training pipeline".
CoT (Wei et al., 2022): The base model is fine-tuned using the question as context and the concatenated (reasoning, answer) as tokens contributing training loss. During inference, the model first generates a reasoning trace and then the final answer.
No-CoT: A standard supervised fine-tuning baseline that does not use reasoning traces. The model is trained to predict the answer directly given the question context.
Pause Token (Goyal et al., 2024): A pause token is inserted between the question and answer without using reasoning traces. The number of pause tokens is set to match the training stages of COCONUT, to ensure fair comparison.
46

---

playing
fighting
lion
playing
fighting
the lion
Large Language Model
Large Language Model
Anchor Network
349
3188
760
326
6-1
329
64
349
318
1760
326
64
320
Cal
18
playing
with
dog
is
plaving
with
dog
Figure 4: Training of standard autoregres-
Figure 5: Training of anchored autoregressive (A2R) models. An anchor network first identifies important tokens (e.g.. "cat', 'dog' shown blue), which are supervised via an auxiliary anchor loss. A lightweight LLM is then trained to predict the next token based anchored predictions.
sive (AR) models. A neural network is trained to predict the next token using causal attention (left- to-right context). All tokens contribute equally to the training loss, and the model treats the se- quence uniformly without structural guidance.
Pause-as-Thought in COCONUT: This variant replaces the continuous latent thoughts in COCONUT with pause tokens while following the same multi-stage training schedule.
iCoT (Deng 2024): Internalizes the reasoning trace into intermediate transformer lay- ers, allowing the model to reason implicitly without generating explicit steps for reasoning. COCONUT (Hao al., 2024): This method uses continuous latent representations (referred to continuous thoughts) instead of discrete reasoning tokens, and inserts these continuous thoughts directly in the embedding layers before processing through the transformer block. This is motivated by the idea that reasoning can often be more intuitively encoded in continuous latent space, especially when explanation using words difficult.
BoLT (Ruan et al., 2025): Bootstraps its "reasoning to learn" ability using latent thoughts and self-distillation. This useful in data-constrained environments, such as GSM8K.
One key distinction between methods that introduce new tokens between the question and (reasoning, answer)-such Pause, SIOONUT-PauSe----Tiie explicitly applies an anchor loss on the inserted [ANT] tokens. This external supervision provides semantic guidance on which tokens are important, resulting in better performance on both language modeling and complex reasoning tasks.
and our method (ACoT>-is that ACoT
C.2.2 Training and Evaluation Benchmarks
Text (OWT). We conduct next-token prediction experiments on the OWT (Gokaslan Cohen, 2019) to evaluate likelihood modeling in the AR setting. In this setting, the model does not have access to structured reasoning traces as the Math and Logic benchmarks described below. The training and validation splits follow the same setup used in the diffusion experiments ($C.1.2).
Math (GSM8K). The GSM8K dataset (Cobbe al., 2021) contains grade-school math word problems requiring multi-step arithmetic reasoning. Each problem is presented as a natural language prompt followed by intermediate reasoning steps and final answer. Successful modeling GSM8K requires accurate understanding of the question through natural language and execute multi-step arithmetic reasoning to derive the final answer.
Logic (ProntoQA, ProsQA). For logical reasoning, we evaluate on ProntoQA (Saparov & He, 2023) and ProsQA (Hao et al., 2024). ProntoQA consists of deductive reasoning tasks, where the model must verify or falsify a hypothesis using given set of symbolic rules. We use the 5-hop variant, which serves controlled benchmark for logical reasoning.
ProsQA (Hao et al., 2024) is a more challenging planning task that requires navigating through complex reasoning graphs to arrive at the correct answer. The model must identify and follow valid
47

---

Chain-of-Thought Data
q1
42
q3
F
72
F3
a1
Stage-O: standard pre-training
41
42
43
[BOA]
[ROA]
71
72
1'3
Q1
Stage-1: insert anchor token
41
42
q3
[BOA]
[ANT]
[EOA]
71
12 73 a;
Stage-2: insert anchor token
42
43
[BOA]
[ANT]
[ANT]
[EOA]
72 73 Qg
Figure 6: Multi-stage training pipeline for Anchored Chain-of-Thought (ACoTT, Here, [BOA] and [EDA] denote the beginning and end of anchors, respectively. Many reasoning traces contain redundant information, increasing entropy and making the reasoning process harder to learn. By supervising the model through small of important tokens extracted from the reasoning trace, ACoT encourages more structured intermediate computations, guiding the model to reason in more targeted and interpretable way. To reduce the number of additional tokens produced, we drop reasoning tokens for every [ANT] insertion. For example, we drop in Stage-1 and 12 in Stage-2 demonstrate this phenomenon.
inference paths among distractors. We follow the multi-stage fine line-tuning procedure described in (Hao et al., 2024) to enable direct comparison with previous baselines.
Evaluation Metric. For all reasoning benchmarks, measuring whether the final answer produced by the model matches the ground truth.
accuracy the primary evaluation metric,
C.2.3 Implementation Details
GSM8K. To ensure fair comparison with prior baselines, closely follow the training protocol used in COCONUT (Hao et al., 2024). Below, we highlight the key distinctions our ACoT training setup and recall relevant aspects of the original training procedure for completeness.
The primary distinction our approach is the use [ANT] tokens to guide the generation of reasoning traces and final answers. Unlike COCONUT, we do not remove intermediate reasoning steps when inserting [ANT] tokens. Instead, we treat anchors as auxiliary indicators that help the model focus on important context of the reasoning trace.
For instance, in Question of Table we identify the tokens 16, 3, 4, 2, and 18 important due to their role in the arithmetic operations leading to the final answer. When using two [ANT] tokens, we select the initial subset of these important tokens in this case, and 3 serve as anchors. These anchors act planning cues that guide the model's 's reasoning trajectory.
Following COCONUT, adopt multi- training strategy shown in Figure 6 and train all models for a total of 25 epochs. We report results from the best checkpoint among these 25 epochs. The first stage is equivalent standard CoT pre-training, where [ANT] tokens inserted between the question and the geeason,answe. tokens. In the second stage (epochs 1-3), we introduce one [ANT] token and continue training for 3 more epochs. In the third stage (epochs 4-6), add one more [ANT] token and train for another 3 epochs. After a maximum of 2 [ANT] tokens have been introduced the span epochs, we continue training for the remaining epochs (up to 25) using supervised fine-tuning.
PAontoQA. We follow a similar multi-stage training procedure for the ProntoQA benchmark, with total of 50 training epochs. Given the longer context and deeper reasoning chains in this dataset, we introduce [ANT] tokens progressively over six stages, following an initial stage of CoT fine-tuning.
On this benchmark, we consider the valid nodes in the reasoning trace as important tokens, since they anchor the generation of the ground truth reasoning steps that ultimately lead to the correct answer.
48

---

Table 9: Anchoring improves autoregressive modeling OWT. Test perplexities (PPL; for standard AR models and our anchored variant (A2R) at various training scales. Results from (Sahoo
al., 2024). A2R consistently improves perplexity by introducing a two-stage prediction process: anchor tokens are first predicted, then used to guide next-token prediction.
Model
PPL (4)
Tokens
AR (retrained)
17.94
110B
ART
17.54
262B
AR (retrained)
17.53
262B
AR (retrained)
17.26
524B
A2R (ours)
17.29
110B
A2R (ours)
16.23
262B
A2R (ours)
15.86
524B
For example, in Question of Table 11, we treat the tokens Alex, Tumpus, Gorpus, Wompus, Sterpus, Brimpus, and Happy as anchor tokens. When training with six [ANT] tokens, we select the first six tokens from this list as supervised labels.
The first stage consists of CoT training without any anchor tokens. In each subsequent stage, we insert one additional [ANT] token into the input and train for 5 epochs. After completing six such stages (i.e., 30 epochs in total), each involving an additional anchor token (up to 6), we continue standard SFT (without further changes to the number of [ANT] tokens) for the remaining 20 epochs.
ProsQA. For the ProsQA benchmark, we follow the same multi-stage training procedure as used for ProntoQA, with one key distinction: at each stage where new [ANT] token is introduced, we remove one reasoning step from the groundtruth reasoning steps used in SFT. This encourages the model to learn missing reasoning steps based on the guidance provided by [ANT]. As in ProntoQA, we progressively insert six [ANT] tokens over six stages, training each stage for 5 epochs, totalling up to 50 epochs. In Question of Table 13, we identify Tom, Terpus, Brimpus, and Lempus as anchoring tokens. When training with six [ANT], we use the first six valid nodes" identified in the reasoning trace as supervised labels.
C.2.4 Generative Modeling using Anchored Auto-Regressee Models
Results on OWT. We evaluate the impact of anchoring on autoregressive language models using the OWT benchmark. Similar to our approach in diffusion language models, we decompose the standard next-token prediction task into two stages. In the first stage, an anchor network predicts distribution over semantically important tokens (e.g.. low-frequency or structurally informative tokens), referred to as anchor tokens or [ANT] for short. These anchor predictions help the model identify key parts of the input that provide better context for effectively predicting the next token.
As shown in Table 9, our anchored autoregressive model (A2R) outperforms standard AR models at training scales. When trained on 262B tokens. A2R achieves perplexity of 16.23, surpassing the corresponding AR baseline (17.53). At the largest scale (524B tokens), A2R further reduces perplexity to 15.86, surpassing the best AR baseline (17.26) by margin of 1.4 PPL. Interestingly. while performance gains standard AR pretraining tend to saturate around a perplexity of 17, our A2R pretraining continues to improve, reducing perplexity from 17.29 to 15.86 as the number of training tokens increases from 10B to 524B. This consistent improvement across scales demonstrates that anchoring provides more informative latent representation, helping the model better estimate token likelihoods.
These results confirm that anchoring is not limited to diffusion models and is broadly applicable to autoregressive architectures. It enables improved context modeling through an anchor network without requiring many architectural changes to the base model.
II the number of [ANT]s is larger than the available anchors, then ignore the loss extra [ANT]s.
49

---

[BoA]
3
[EoA] [BoR] <<16- 3-4=9>3 == 1855
[EoR]
18 Tokens
Language Model Head
+ Anchor Transformer (with causal attention)
Embeddings
MaTMaaTTa [EoA] [[BOR] 2<16 3-4=9>> <= [EoR]
Embeddings
[BoA]
Tokens
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends everyday with four. She sells the remainder the farmers' market daily for $2 per fresh duck egg. How much dollars does she make every day at the farmers' market?
Figure 7: Anchored Chain-of-Thhughh (ACoT) training pipeline. In this example with two anchor tokens, we preprocess the original reasoning trace <16-3-4-99 -- <9*2=18>, and extract the numbers '16' and '3' as the anchor tokens (the first two numbers in the reasoning trace). We then create modified training trace that augmented with these anchor tokens (top line in figure). Example trace from GSM8K dataset.
C.2.5 Improved Reasoning using Anchored Chain-of-Thought
Decoding order is known to affect inference quality, as established in classical structured prediction literature (e.g., via topological or lexical ordering) (Jordan al., 1999; Bengio 2003; Koller Friedman, 2009). However, most LLMs (especially autoregressive ones) follow strictly left-to-right generation scheme, which often leads to shallow, sequential reasoning. This bias hinder tasks where intermediate computations depend on later context or global structure.
Our proposed Anchored Chain-of-Thhuggee (ACoT) addresses this issue by leveraging important tokens (such as root nodes in the underlying reasoning graph) prior to decoding. These tokens serve intermediate supervision points, enabling the model focus the most critical sub-computations early in the reasoning trace. We then apply the standard next-token prediction loss conditioned on these anchor predictions, effectively guiding the model toward more structured and globally coherent solutions. Thus, anchors allow the model to look ahead and think non-sequentially while retaining compatibility with standard autoregressive training and decoding pipelines.
During training, as illustrated in Figure 7, the loss function for the Anchor Network is supervised using future important tokens. As consequence, during inference, this results in anchor predictions having logits with higher likelihood on future important tokens. Thus, the auto-regressively generated tokens depend not only previous tokens (causal attention). but also implicitly potential future important tokens that are encoded within the anchor predictions (see Figure 2 in the main paper). This results in out-of-order reasoning discussed below.
Results on GSM8K. To evaluate the effectiveness of anchoring in complex reasoning tasks, we apply method ACoT on GSM8K (Cobbe et al., 2021). Table 10 presents qualitative examples comparing standard CoT reasoning traces with those produced by our ACoT model. Each example includes the input question, ground truth CoT trace, the model's full output (including anchor tokens), and the extracted final answer. Table 5 contains the quantitative results.
A key insight from Question is that standard CoT processes the question in purely left-to- right manner. It computes 16
9, and then 2 18, 18. following the order which quantities appear in the question. In contrast, ACoT introduces [ANT] to capture important tokens, which allows it to reason more globally. Specifically, ACoT first computes + 7 to aggregate all consumption before subtracting from the total (16 7 9). pattern more aligned with human
50

---

intuition. This demonstrates how anchoring enables a "look-ahead" planning behavior, in contrast to the left-to-right decoding bias of standard CoT.
In Question 3, ACoT diverges from the gold trace and produces an incorrect final answer. This example highlights limitation: while anchoring allows more flexible planning. it still depends correctly identifying and utilizing important tokens.
CoT and method ACoT share similar experimental setup, with the only exception of anchoring via multi-strage training. While standard multi-stage training as COCONUT reduces the performance from 42.9% 34.1%, our anchoring mechanism prvoides substantial gain of 11.1% over these latent reasoning methods (such as COCONUT and iCoT), and a notable gain of 2.3% over COT.
Overall, these examples demonstrate that anchoring provides more flexible and semantically guided framework for reasoning, enabling the model to break free from strictly left-to-right token prediction. This leads to improved planning and more human-like reasoning behaviors, especially in problems requiring multi-step arithmetic reasoning as in GSM8K.
As shown in Table 11, our ACoT model generates valid reasoning traces and correctly predicts the final answer across symbolic reasoning tasks in ProntoQA. In the first example, the question asks whether "Alex is happy" based on a complex chain of relational facts. The standard CoT trace progresses step-by-step by chaining multiple class membership relations (e.g.. "Alex is tumpus", , "Tumpuses are gorpuses", ... "Brimpuses are happy"). Our ACoT model, guided by six [ANT] tokens, learns identify and attend to these anchors in the reasoning process.
ProntoQA.
Recall from Table 5 that CoT, COCONUT, and ACoT share nearly identical training setups, with the only distinction being the use of anchor tokens in ACoT and continuous thoughts in CoCONUT. While standard multi-stage training in COCONUT leads to a marginal improvement from 98 98.8% (CoT) to 99.8%, our anchoring mechanism further enhances performance. By incorporating explicit supervision through anchor tokens, ACoT achieves a perfect accuracy of 100% on this relatively easier benchmark, demonstrating the effectiveness of anchoring logical reasoning.
ProsQA. This experiment illustrates that ACoT can internalize and reconstruct reasoning traces through anchor tokens, similar latent reasoning methods such iCoT and COCONUT. As in those prior works, our model avoids explicitly generating intermediate reasoning steps while still maintaining logical coherence in the final answer.
We evaluate ACoT under a variant the training setup where we do not remove any reason- ing steps (i.e., following the setup used in PA) In this setting, ACoT achieves 81% accu- racy-outerrormming standard CoT (77.5%) but still falling short the 97.0% achieved by COCONUT. Prior work (Hao et al., 2024) has shown that gradually removing reasoning steps during multi-stage training provides a significant performance boost on ProsQA Following this recommendation, we integrate reasoning step removal into ACoT fine-tuning. This modification improves model's accuracy from 81 % to 97.3%. Importantly, ACoT generates much less tokens compared to CoT and COCONUT while surpassing their accuracy, shown in Table 12.
In the first example from Table 13, the question asks whether "Tom lempus or scrompus", requiring multi-hop reasoning chain through symbolic class hierarchies. The ground truth reasoning trace proceeds via three steps: (1) Tom is terpus, (2) Every terpus is a brimpus, and (3) Every brimpus is lempus. Our ACoT model accurately identifies and anchors this chain using six [ANT] tokens, and directly generates the correct final inference: Tom is lempus. Notably, this is done without producing intermediate reasoning steps, demonstrating that the model has internalized the underlying inference structure via the anchor tokens. These results highlight that anchoring provides complementary benefits to existing fine-tuning strategies (such CoT and COCONUT), and further boosts the model's 's ability to solve complex reasoning problems.
51

---

Table 10: Examples of math word problems with reasoning traces from GSM8K (Cobbe et al., 2021). Each row shows the input question, groundtruth reasoning trace (CoT), answer, model's full output sequence, and the final extracted answer. Our model implicitly reasons through anchoring tokens ([ANT]) and produces reasonable traces before computing the final answer.
Question 1
Janet's 's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much dollars does she make every day at the farmers' market? 116-3-499> <9*2=18>
CoT
Answer
18
Full Output
[ANT] [ANT] 18
<3+4=7> <16-7=9> <9*2=18>
Extracted Output
Question
robe takes 2 bolts of blue fiber and half that much white. How many bolts in total?
CoT
<2/2=1> <2+1=3>
Answer
Full Output
[ANT] [ANT] <2/2=1> <2+1=3>
Extracted Output
Question 3
Josh decides to try flipping house. He buys house for $80,000 and then puts in $50,00 000 in repairs. This increased the value of the house by 150%. How much profit did he make?
CoT
480000+5000-130000
180000+1.5=120000
<120000-80000=2000000
2200000-130000-70000
Answer
70000
Full Output
[ANT] [ANT]
<7500+50000=575000>
057500-80000=-325000 -32500
Extracted Output
Question 4
James decides to run sprints 3 times week. He runs 60 meters each sprint. How many total meters does he run a week? <3+3-9> <9*60=540>
CoT
Answer
540
Full Output
[ANT] [ANT] <3*60=180>
Extracted Output
180
Question
Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need give her chickens in the final meal of the day if the size of Wendi' 's flock is 20 chickens?
CoT
<3* <3+20=60> <60-15-25-20>
Answer
20
Full Output
[ANT] [ANT]
<15+25=40> <40-20=20>
Extracted Output
20
52

---

Table 11:
Examples of logical reasoning tasks with symbolic reasoning traces from Pron- toQA (Saparov & He, 2023). Each row shows the input question, groundtruth reasoning trace (CoT), answer, the model's generated output sequence, and the extracted final answer. Our model implicitly reasons through anchoring tokens ([ANT]) to infer logical relationships.
Question
Tumpuses are floral. Each jompus is not melodic. Tumpuses are numpuses. Gorpuses are opaque. Each grimpus small. Sterpuses are dumpuses. Tumpuses are gorpuses. Every lorpus is not happy. Every wumpus is shumpus. Each gorpus grimpus. Shumpuses are slow, Every dumpus is overcast. Gorpuses are wimpuses. Vumpuses are dull. Sterpuses are brimpuses. Numpuses are not metallic. Jompuses are impuses. Brimpuses are happy. Sterpuses are hot. Brimpuses are vumpuses. Wumpuses are sterpuses. Brimpuses are zumpuses. Wumpuses are sour. Alex a tumpus. Alex is jompus. True false: Alex is happy.
CoT
Alex is a tumpus. Tumpuses are gorpuses. Alex gorpus. Gorpuses are wumpuses. Alex is wumpus. Wumpuses are sterpuses. Alex is a sterpus. Sterpuses are brimpuses. Alex is brimpus. Brimpuses are happy. Alex is happy.
Answer
True
Full Output
[ANT] [ANT] [ANT] [ANT] [ANT] [ANT] Alex is tumpus. Tumpuses are gorpuses. Alex is gorpus. Gorpuses are wumpuses. Alex is wumpus. Wumpuses are sterpuses. Alex is a sterpus. Sterpuses are brimpuses. Alex is brimpus. Brimpuses are happy. Alex is happy. True
Extracted Output
Question 2
Brimpuses sterpuses. Each grimpus is numpus. Lorpuses angry. Grimpuses are moderate. Each dumpus a lempus. Each lempus is cold. Gorpuses yumpuses. Every rompus is not sunny. Each dumpus is gorpus. Yumpuses are bitter. Each grimpus is a vumpus. Gorpuses are not large. Brimpuses lorpuses. Every vumpus is discordant. Every numpus is shy. Brimpuses are not brown. Dumpuses are floral. Gorpuses are grimpuses. Each shumpus are brown. Every shumpus is not metallic. Vumpuses are brimpuses. Polly is dumpus. Polly is a shumpus. True or false: Polly brown.
wumpus. Vumpuses
rompuses. Tumpuses
CoT
Polly is dumpus. Each dumpus is gorpus. Polly is gorpus. Gorpuses are grimpuses. Polly Each grimpus is vumpus. Polly is vumpus. Vumpuses are brimpuses. Polly brown. Polly is not brown. False
grimpus.
brimpus. Brimpuses are not
Answer
Full Output
[ANT] [ANT] [ANT] [ANT] [ANT] [ANT] Polly is a dumpus. Each dumpus is gorpus. Polly is gorpus. Gorpuses grimpuses. Polly is a grimpus. Each grimpus is a vumpus. Polly is vumpus. Vumpuses brimpuses. Polly is a brimpus. Brimpuses are not brown. Polly is not brown.
Extracted Output
False
53

---

Table 12:
Accuracy and Token Usage on ProsQA. ACoT outperforms prior latent reasoning methods, including COCONUT, while using fewer comparable reasoning tokens during inference. f reported COCONUT.
Method
ProsQA
Accuracy (%)
Tokens
No-CoT
76.7=1.0
8.2
Pause Token (Goyal et al., 2024)
75.9
8.2
CoTt (Wei et al., 2022)
77.5
49.4
iCoT (Deng et al., 2024)
98.2=0.3
8,2
CocoNUT (Hao et al., 2024)
97.0-0.3
14.2
Pause
96.6-
8.2
ACoT (ours)
97.3=0.2
8.2
54

---

Table 13: Examples of logical reasoning tasks with symbolic reasoning traces from ProsQA (Hao et al., 2024). Each row shows the input question, groundtruth reasoning trace (CoT), answer, model's generated output sequence, and the extracted final answer. Our model implicitly reasons through anchoring tokens ([ANT]) to infer logical relationships. In this experiment, we employ the gradual CoT removal scheme used in prior works (Deng et al., 2024; Hao et al., 2024) to demonstrate the reasoning ability in the anchored latent space without producing word tokens.
Question 1
Every shumpus is rempus. Every shumpuus is yimpus. Every terpus is a fompus. Every terpus is a gerpus. Every gerpus is a brimpus. Alex is a rempus. Every rorpus is scrompus. Every rorpus is yimpus. Every terpus is a brimpus. Every brimpus is a lempus. Tom is a terpus. Every shumpus is timpus. Every yimpus boompus. Davis is shumpuus Every gerpus a lorpus. Davis a fompus. Every shumpus is a boompus. Every shumpus is rorpus. Every terpus is lorpus. Every boompus a timpus. Every fompus is a yerpus. Tom is a dumpus. Every rempus is a rorpus. Is Tom lempus or scrompus?
CoT
Tom is a terpus. Every terpus is a brimpus. brimpus is a 1empus.
Every
Answer
Tom is lempus.
Full Output
[ANT] [ANT] [ANT] [ANT] [ANT] [ANT] Tom is lempus.
Extracted Output
Tom is a lempus.
Question 2
Sally is a zhorpus. Every yumpus is a fompus. Every zhorpus is a rempus.
Every rompus is a sterpus. Every kerpus timpus. Stella is yumpus. Every zhorpus is a zumpus. Every wumpus is yumpus. Sally is a rempus. Stella is a wumpus. Every zumpus is a rorpus. Sally is a rompus. Every numpus is a bompus. Every zumpus is a scrompus. Every rempus is kerpus. Every zumpus a vumpus. Every timpus a yerpus. Every rempus is numpus. Every vumpus is worpus. Every rompus is a felpus. Every wumpus is a sterpus. Every rompus is a kerpus. Every zumpus a rempus. Every rempus is chorpus. Bob is rorpus. Every wumpus is a fompus. Sally is kerpus. Every zhorpus rompus. Is Sally a fompus or worpus? Sally is a zhorpus. Every zhorpus is a zumpus. Every zumpus is a vumpus. Every vumpus is a worpus.
CoT
Answer
Sally is a worpus.
Full Output Extracted Output
[ANT] [ANT] [ANT] [ANT] [ANT] [ANT] Sally is worpus.
Sally is a worpus.
Question 3
Every shumpus is yumpus. Every worpus is yimpus. Every shumpus is gwompus. Every tumpus is a boompus. Every worpus a shumpus. Every storpus is a terpus. Max is yimpus. Every shumpus is rompus. Every wumpus a jelpus. Every boompus
terpus. Fae a tumpus. Every tumpus is a worpus. Every rompus is gorpus. Every timpus is impus. Everyjompus is a gerpus. Every boompus a rompus. Fae is a boompus. Every boompus is kerpus. Every zumpus is bompus. Max is a rempus. Every rompus is a kerpus. Max is a impus. Every rempus is a impus. Every wumpus is yumpus. Every grimpus is a terpus. Every tumpus is jompus. Every yumpus is a felpus. Every jelpus is a felpus. Every shumpus is a felpus. Every rempus a timpus. Every storpus is jompus. Every rompus is a storpus. Every tumpus is a wumpus. Every wumpus is a jompus. Every boompus is worpus. Fae is storpus. Every worpus is a jelpus. Every grimpus is a felpus. Every worpus is a yumpus. Every rempus is a zumpus. Every kerpus is a grimpus. Is Fae gwompus bompus?
CoT
Fae is tumpus. Every tumpus is a worpus. Every
worpus is shumpus. Every shumpus is gwompus.
Answer
Fae is a gwompus.
Full Output
[ANT] [ANT] [ANT] [ANT] [ANT] [ANT] Fae is bompus.
Extracted Output
Fae a bompus.
55


Reviews:
[{"review_id": "X2cIxx66xv", "reviewer": "Reviewer_mP11", "summary": "This paper observes that the performance of masked diffusion models can be improved by generating certain “special” tokens—called anchor tokens—early in the sampling process. (This isn't a novelty as a similar observation was made in Shi et al., 2024 and He et al., 2022)\n\nModified forward process:  To enforce such behavior, they train their model where these anchor tokens are masked later in the forward diffusion process. \n\nModified reverse process: The reverse sampling process is redesigned so that anchor tokens are predicted first. Additionally, the parameterization of the denoising model is adjusted: instead of mapping noisy sequences directly to clean data, part of the network predicts a distribution over the anchor tokens, which is then used by another network to map to the clean data distribution.\n\nModified NELBO: To support this training strategy, the authors propose a modified NELBO objective tailored for the anchoring mechanism.\n\nThe authors claim that this approach improves the likelihood performance of masked diffusion models. Moreover, anchoring also boosts the performance of autoregressive models and enhances reasoning capabilities in mathematical and logical tasks, outperforming existing chain-of-thought methods.\n\n\n### References\nHe et al., 2022 \"DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models\"\nSahoo et al., 2025 \"The diffusion duality\", ICML 2025", "questions": "See above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\nThe overall results, particularly on the reasoning tasks, are very promising.\n\n### Weaknesses\n\n1.\tThe observation that prioritizing important words during the diffusion process can improve likelihood was previously proposed in Shi et al. (2024) and He et al. (2022). The authors must compare their method against Gen-MD4 from Shi et al. (2024). Notably, Gen-MD4 does not use a specialized network to identify important tokens; it simply introduces a learnable weight for each token in the vocabulary. Since the proposed method involves two networks, I suspect it may require significantly more parameters (see my concern below).\n\n2.\tModel size clarity: It is unclear how large the denoising model is and what transformer sizes are used for the anchor network and the denoising network. What is the total parameter count for both networks combined? Given that the embedding and final projection layers in transformers are typically large, I would not be surprised if the proposed algorithm uses substantially more parameters than a standard autoregressive transformer.\n\n3.\tMAUVE score: The MAUVE scores reported using GPT-2 embeddings are often unreliable. Could the authors recompute the MAUVE scores using modern BERT embeddings?\n\n\n### Minor comments \n* For completeness, the authors should consider including additional baselines such as DUO and UDLM (Uniform-State Diffusion) in Tables 1 and 3.\n\n### References\nHe et al., 2022. \"DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models\"\n\nSahoo et al., 2025. \"The diffusion duality\", ICML 2025", "was_revised": true, "final_justification": "After the rebuttal, I choose to retain my rating", "rebuttal": "> **Q1.1 The observation that prioritizing important words during the diffusion process can improve likelihood was previously proposed in Shi et al. (2024) and He et al. (2022).**\n\nA1.1. The importance of specific tokens has been well-explored in classic NLP literature —e.g., via TF-IDF weighting (Salton & Buckley, 1987), conditional random fields (Lafferty et al., 2001), and neural attention mechanisms (Bahdanau et al., 2014; Vaswani et al., 2017). While recent works such as GenMD4 (Shi et al., 2024) and DiffusionBERT (He et al., 2022) focus on static global importance, our proposed ADLM introduces a fundamentally different approach based on **sequence-level contextual anchoring** and a novel, **fully differentiable training objective (ANELBO)**. We highlight the key differences below. \n\n---\n\n**GenMD4 vs. ADLM**\n\n- **Local vs. Global Importance.**  \n  GenMD4 assigns global, static importance to tokens via learnable weights across the vocabulary. In contrast, ADLM selects important tokens dynamically **per input sequence** via anchored transition function (Eq. 5). For instance, \"cat\" and \"dog\" may be important in a given context but overlooked by GenMD4 due to low global weights. ADLM adaptively anchors these tokens via our anchor transition function.\n\n- **Objective Function.**  \n  GenMD4 requires REINFORCE-style gradient estimators because it cannot backpropagate through discrete samples (which introduces high variance and instability). ADLM introduces an **Anchored NELBO** that supports end-to-end differentiable training using an adaptive anchor transition (see Sec 3), leading to more stable optimization.\n\n- **Generalization vs. Overfitting.**  \n  As acknowledged in Shi et al. (2024, Sec 7.1), GenMD4 can overfit to training data, achieving low perplexity but poor generalization. ADLM achieves both strong **generative perplexity** and **zero-shot generalization**, as shown in Sec 5.1.1 in the main draft and Appendix C.1.3-C.1.4 in the supplementary material. .\n\n---\n\n**DiffusionBERT vs. ADLM**\n\nDiffusionBERT also assigns static importance via frequency-based \"spindle\" noise schedules. This global assignment, similar to GenMD4, does not adapt to sequence-level context and suffers from the same **hard unmasking** problem.  \nIn contrast, ADLM introduces **anchoring as a latent reasoning process** to model contextual importance via anchor transition function and adaptively decode tokens important for each sequence.\n\n---\n\nOur anchoring mechanism opens up other intriguing directions. Words corresponding to verbs and nouns relate to the concepts of **terminals and syntactic categories** in the **Chomsky Hierarchy** (formal language theory). With this preface, anchoring can be thought of as another means of bringing classical NLP back into statistical language models. This view opens up other grammar-based mechanisms for anchor network design (e.g., multi-level hierarchical anchor network), which is an exciting direction for future study.\n\n\n> **Q1.2. The authors must compare with GenMD4 from Shi et al. 2024.**\n\nA1.2. We now include GenMD4 and MD4 in Table 1 (b). Repeating other results for clarity.\n\n### \n\nTest Perplexity on OWT\n\n| **Method**                | **PPL (↓)** |\n|---------------------------|--------------------|\n| SEDD Absorb    | 24.10              |\n| `MD4`        | 22.13 |\n| `GenMD4`      | 21.80|\n| MDLM                      | 22.98              |\n| **ADLM (ours)** | **20.14**         |\n\n---\n\n### Model Size vs Performance \nRepeating results from Table 7 for clarity. \nNewly added: SEDD-medium (T=1000), MD4-medium, SEDD-small and MD4-small. \n\n| **Model Type**       | **Evaluated Model**                                            | **Params** | **Gen. PPL (↓)** |\n|----------------------|----------------------------------------------------------------|------------|------------------|\n| *Diffusion*    | Plaid (Gulrajani & Hashimoto, 2023) (*T=4096*)          | 1.3B     | 19.7    |\n|                      | SEDD-medium (Lou et al., 2024) (*T=2048*)                      | 424M       | 27.3 |\n|                      | `SEDD-medium` (Lou et al., 2024) (*T=1000*)                      | 424M       | 31.95 |\n|                      | `MD4-medium` (Shi et al., 2024)  (*T=1000*)                    | 416M       |27.51 |\n|                      | `SEDD-small` (Lou et al., 2024) (*T=1000*)                      | 165M      | 42.94 |\n|                      | `MD4-small` (Shi et al., 2024) (*T=1000*)                    | 165M      |33.16|\n|                      | MDLM (Sahoo et al., 2024) (*locked-in, T=1000*)                | 170M  | 44.2  |\n|                      | ReMDM (Wang et al., 2025a) (*remasking, T=1024*)  | 170M  | 28.6   |\n|                      | GGM (Varma et al., 2025) (*T=4096*)                            | 387M | 19.5  |\n|                      | **ADLM (ours)** (*locked-in, T=1000*)                          | 293M | 32.9  |\n|                      | **ADLM (ours)** (*remasking, T=1000*)             | 293M       | 26.8             |\n|                      | **ADLM (ours)** (*remasking, T=1024*)             | 293M       | 25.1             |\n|                      | **ADLM (ours)** (*remasking, T=2048*)             | 293M       | 20.1             |\n|                      | **ADLM (ours)** (*remasking, T=4096*)             | 293M       | **15.7**         |\n\n---\n\n### Zero-shot Evaluation\n\nRepeating results from Table 4 for clarity. Newly added: MD4\n\n| **Model**                 | **Lambada** | **PTB**  | **WikiText** | **LM1B** | **AG News** | **PubMed** | **ArXiv** |\n|--------------------------|-------------|----------|--------------|----------|-------------|------------|-----------|\n| MDLM                     | 47.52       | 95.26    | 32.83        | 67.01    | 61.15       | 41.89      | 37.37     |\n| `MD4`                       | 48.43       |102.26|35.42| —        | —           | —          | —         |\n| **ADLM (ours)** (262B)   | 44.93       | 98.16    | 32.45        | 65.59    | 57.10       | 38.29      | 35.08     |\n| **ADLM (ours)** (524B)   | **44.32**   | **95.37**| **31.94**    | **64.43**| **55.72**   | **37.56**  | **33.69** |\n\n---\n\n> **Q2.1. Model size clarity: It is unclear how large … network.**\n\nA2.1. We will expand architecture details in Lines **247–250** and Appendix **C.1.3 (Lines 1471–1479)** as follows:\n\n- Both networks follow the DiT-Base architecture [Peebles & Xie, 2023], consistent with SEDD and MDLM.\n- The anchor network uses 12 DiT blocks.\n- The denoising network uses 6 DiT blocks.\n- Each block has hidden dimension = 768 and 12 attention heads.\n- The input sequence length is 1024 for OWT and 128 for LM1B.\n\n---\n\n>**Q2.2. What is the total parameter count for both networks combined?**\n\nA2.2. ADLM has **293M** total As given  in **Table 7 in Appendix C.1.3**, . For comparison:\n\n- **Plaid** has **1.3B** parameters  \n\n- **SEDD-Medium** has **424M** parameters  \n\n- **GGM** has **387M** parameters\n\n- **MD4-medium** has **424M** parameters\n\nThus, ADLM achieves superior results while using **fewer parameters** (293M) than these strong baselines.\n\n---\n\n>**Q2.3. Given … the proposed algorithm uses substantially more parameters than a standard AR transformer.**\n\nA2.3. Our theoretical results (Sec 4) show that anchoring requires less parameters. This translates into practice: the **denoiser uses ~123M parameters**, while the anchor network uses ~170M, totaling **293M**—still smaller than DLMs with comparable performance, please refer to Table 7 in Appendix C.1.3 (and new results in A1.2). \n\nADLM is **parameter-efficient** because:\n\n- The denoiser has **half the transformer layers** than that of the anchor network.\n\n- We **share the tokenizer and vocabulary** across the anchor and the denoiser.\n\n- We use a **lightweight linear projection** to pass anchor logits to the denoiser (Appendix C.1.3), avoiding additional parameters due to re-embedding layers.\n\n---\n\n>**Q3. MAUVE score … Could the authors recompute the MAUVE scores using modern BERT embeddings?**\n\nWe have now recomputed the MAUVE scores using BERT embeddings (bert-base-uncased) with max sequence length=512 and T=4096. We compare with the best performing baseline ReMDM from Table 3 and obtain a **7% improvement** in MAUVE score. We will add these results in the revision.  \n\n### Evaluation on OWT. \n\n| **Method**      | **MAUVE (↑)** | **Gen PPL. (↓)** | **Entropy (↑)** |\n|----------------|---------------|------------------|-----------------|\n| ReMDM          |   0.702      | 17.6             | 5.20            |\n| **ADLM (ours)**| **0.754**     | **15.9**         | **5.19**        |\n\n---\n\n>**Q4. For completeness, compare with DUO and UDLM (Uniform-State Diffusion).**\n\nA4. While our primary focus is on **masked diffusion**, we now include comparison with **uniform diffusion** methods: DUO and UDLM.\n\n---\n\n### Test PPL\n\n| **Method**                  | **LM1B** | **OWT** |\n|-----------------------------|---------:|--------:|\n| `UDLM (Schiff et al., 2025)`  | 31.3     | 27.4    |\n| `DUO (Sahoo et al., 2025)`    | 29.9     | 25.2    |\n| **ADLM (Ours)**             | **26.4** | **20.6** |\n\n---\n\n### Zero-Shot Validation PPL\n\n| **Model**               | **Lambada** | **PTB**  | **WikiText** | **LM1B** | **AG News** | **PubMed** | **ArXiv** |\n|-------------------------|-------------|----------|--------------|----------|-------------|------------|-----------|\n| `UDLM (Schiff et al., 2025)` | 53.57     | 112.82   | 39.42        | 77.59    | 80.96       | 50.98      | 44.08     |\n| `DUO (Sahoo et al., 2025)`   | 49.78     | **89.35**| 33.57        | 73.86    | 67.81       | 44.48      | 40.39     |\n| **ADLM (Ours)** (262B)     | 44.93     | 98.16    | 32.45        | 65.59    | 57.10       | 38.29      | 35.08     |\n| **ADLM (Ours)** (524B)     | **44.32** | 95.37    | **31.94**    | **64.43**| **55.72**   | **37.56**  | **33.69** |\n\nWe hope this addresses the reviewer’s concern, and we appreciate the opportunity to clarify."}, {"review_id": "v4xCupfCKI", "reviewer": "Reviewer_oJvZ", "summary": "This paper identifies a key limitation of current diffusion language models: important tokens are often masked early in the forward process, which prevents the model from accessing sufficient context to accurately reconstruct the original sequence. To address this, the authors propose the Anchored Diffusion Language Model (ADLM), which introduces an anchor network alongside the standard denoising network to predict the likelihoods of important tokens. They derive an Anchored Negative Evidence Lower Bound (ANELBO) to jointly train the two networks. ADLM achieves strong performance on language modeling perplexities and MAUVE. Additionally, the proposed anchoring mechanism can be integrated into autoregressive models to enhance reasoning ability.", "questions": "- Have you explored applying the proposed method to stronger language models such as LLaMA or Qwen? What are the results in those settings?\n- What optimizer settings did you use during training? Do different optimizer hyperparameters for the anchor network and the denoising network lead to better performance?\n- I find Eq. (5) somewhat confusing: the right-hand side appears to describe a distribution over $z_s$ ​, but the left-hand side uses $r(y_s)$. Could you provide further clarification on this point?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The motivation is clear and novel. It addresses an interesting question that may help explain why diffusion language models underperform autoregressive models in likelihood.\n- The paper is well-structured.\n- The experimental results are strong: ADLM outperforms previous diffusion language models on both perplexity and MAUVE.\n\nWeaknesses\n- The paper does not report architectural details of the anchor and denoising networks, such as layer counts or dimensions. This makes it unclear whether ADLM requires more compute than MDLM under the same training data. It also leaves open how ADLM compares to baselines under equal pre-training or inference cost.\n- As noted in [1], using GPT-2 perplexity to measure sentence quality is questionable. The authors are encouraged to report entropy in Table 2 alongside GPT-2-Large perplexity.\n- In autoregressive experiments, the authors use GPT-2 as the base model. GPT-2 is outdated and performs poorly on reasoning tasks, making it hard to validate the claimed improvements.\n- The motivation is to use an anchor network to predict important tokens, as illustrated in Figure 1. However, the paper does not visualize the learned anchor outputs after training to verify that the model captures such tokens. This weakens the connection between the method and its original motivation.\n[1] Zheng et al. Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. ICLR2025.", "was_revised": true, "final_justification": "I increased my score from 3 to 4 after the rebuttal because most of my concerns have been addressed. The authors provided detailed architectural descriptions, clarified compute comparisons via controlled ablations, and demonstrated the benefits of their two-stage anchoring mechanism. However, I did not raise my score higher because I still have reservations regarding the significance of the method. In particular, there is a lack of evaluations on stronger language models. Overall, the paper represents a technically sound and well-motivated contribution to the emerging area of diffusion language modeling, and I tend to accept it now.", "rebuttal": ">**W1.1. The paper does not report architectural details of the anchor and denoising networks, such as layer counts or dimensions.**\n\nA1.1. We appreciate the reviewer’s suggestion to improve clarity. We will expand the discussion in Lines **247–250** and Appendix **C.1.3 (Lines 1471–1479)** to incorporate the following architecture details: \n\n- Both networks follow the DiT-Base architecture [Peebles & Xie, ICCV 2023], consistent with SEDD (Lou et al., ICML 2024) and MDLM [Sahoo et al., NeurIPS 2024].\n- The anchor network uses 12 DiT blocks.\n- The denoising network uses 6 DiT blocks.\n- Each block has hidden dimension = 768 and 12 attention heads.\n- The input sequence length is 1024 for OWT and 128 for LM1B.\n\n---\n\n\n>**W1.2 This makes it unclear whether ADLM requires more compute than MDLM under the same training data. It also leaves open how ADLM compares to baselines under equal pre-training or inference cost.**\n\nA1.2. Thank you for highlighting this important point. As discussed in **Lines 276–278**, we perform a controlled ablation to compare **ADLM** against **MDLM** under equal compute and data budgets.\n\nWe introduce **ADLM***, a variant of our model that uses the same two-stage architecture as ADLM but **disables the anchor loss** (i.e., sets $\\gamma = 0$). This allows a fair comparison with **MDLM**, using the same training compute.\n\n- As shown in **Table 1(b)** of the main paper:\n  - **MDLM** achieves a test perplexity of **23.17**.\n  - **ADLM*** improves this to **21.79**, validating the benefit of our two-stage design.\n  - Full **ADLM**, trained with our proposed **ANELBO** loss, further reduces perplexity to **20.62** \n\nIn addition, **Table 7** compares ADLM with other baselines under comparable pre-training or inference cost.\n\n- **ADLM** achieves **better generation quality** than several larger models:\n- Outperforms Plaid (1.3B), SEDD-medium (424M), and GGM (387M),\n- While using only 293M parameters and comparable sampling steps—thus requiring significantly **less pre-training and inference cost**.\n\n---\n\n### Model Size vs Performance \nRepeating our results from Table 7 for clarity. \nNewly added results: SEDD-medium (T=1000), MD4-medium, SEDD-small, MD4-small. \n\n| **Model Type**       | **Evaluated Model**                                            | **Params** | **Gen. PPL (↓)** |\n|----------------------|----------------------------------------------------------------|------------|------------------|\n| *Diffusion*    | Plaid (Gulrajani & Hashimoto, 2023) (*T=4096*)          | 1.3B     | 19.7    |\n|                      | SEDD-medium (Lou et al., 2024) (*T=2048*)                      | 424M       | 27.3 |\n|                      | `SEDD-medium` (Lou et al., 2024) (*T=1000*)                      | 424M       | 31.95 |\n|                      | `MD4-medium` (Shi et al., 2024)  (*T=1000*)                    | 416M       |27.51 |\n|                      | `SEDD-small` (Lou et al., 2024) (*T=1000*)                      | 165M      | 42.94 |\n|                      | `MD4-small` (Shi et al., 2024) (*T=1000*)                    | 165M      |33.16|\n|                      | MDLM (Sahoo et al., 2024) (*locked-in, T=1000*)                | 170M  | 44.2  |\n|                      | ReMDM (Wang et al., 2025a) (*remasking, T=1024*)  | 170M  | 28.6   |\n|                      | GGM (Varma et al., 2025) (*T=4096*)                            | 387M | 19.5  |\n|                      | **ADLM (ours)** (*locked-in, T=1000*)                          | 293M | 32.9  |\n|                      | **ADLM (ours)** (*remasking, T=1000*)             | 293M       | 26.8             |\n|                      | **ADLM (ours)** (*remasking, T=1024*)             | 293M       | 25.1             |\n|                      | **ADLM (ours)** (*remasking, T=2048*)             | 293M       | 20.1             |\n|                      | **ADLM (ours)** (*remasking, T=4096*)             | 293M       | **15.7**         |\n\n\n---\n\n\n\n>**W2. As noted in [1], using GPT-2 perplexity to measure sentence quality is questionable. The authors are encouraged to report entropy in Table 2 alongside GPT-2-Large perplexity.**\n\n\nA2. Thank you for the suggestion. We agree that GPT-2 perplexity alone may not fully capture generation quality. For this reason, we already report both **GPT-2-Large perplexity and entropy** in **Table 3**, along with a range of sampling steps. We kindly refer the reviewer to Table 3 and Lines 313–318 for detailed discussion. Table 3 results indicate that the proposed method ADLM not only generates high quality text but also preserves diversity. \n\n---\n\n\n>**W3. In autoregressive experiments, the authors use GPT-2 as the base model. GPT-2 is outdated and performs poorly on reasoning tasks, making it hard to validate the claimed improvements.**\n\n\nA3. We thank the reviewer for pointing this out. As noted in Lines **320–322**, our primary focus in this paper is on **diffusion language models (DLMs)**. However, we find it noteworthy that the benefits of our **anchoring mechanism** extend beyond diffusion, unlike the prior DLMs that are restricted to only diffusion settings.\n\nTo support this claim, we include experiments on **autoregressive models using GPT-2** (a widely adopted baseline), ensuring **compatibility with recent works** such as **BoLT, iCoT, and Coconut**.  Our goal is not to compete with modern LLMs, but to show **consistent gains from anchoring** under comparable compute. All baselines use the same architecture and training setup for a fair analysis, which otherwise becomes difficult given the cost associated with training large language models. \n\nWe hope the reviewer will consider our anchored AR experiments as a proof-of-concept, highlighting the generality of our approach across diffusion and AR model families.\n\n---\n\n\n>**W4. The motivation is to use an anchor network to predict important tokens, as illustrated in Figure 1. However, the paper does not visualize the learned anchor outputs after training to verify that the model captures such tokens. This weakens the connection between the method and its original motivation.**\n\nA4. Thank you for the insightful question. We note that anchor tokens are not explicitly decoded to minimize the number of extra tokens generated  during inference. This offers 2 crucial benefits during training: (1) ANELBO remains end-to-end differentiable with respect to the parameters of the anchor and the denoiser network (discussed in Lines 1471-1479); and (2) This further enables implicit reasoning via latent distribution over anchors as opposed to a single sampled token. \n\nBelow, we recall **qualitative and visual evidence** from the paper showing that the model learns to focus on important tokens. **Appendix C.1.4** shows that anchoring leads to the generation of **more human-like, structured text**, supporting the **qualitative claims** in Figure 1 and the **quantitative gains** reported in **Table 3** (e.g., improved MAUVE, Gen PPL and entropy scores). \n\nImportantly, we **visualize intermediate tokens** generated during inference in **Appendix C.2.5**, specifically in the section *“Results on GSM8K”* (Lines **1740–1746**) where anchoring can be better interpreted. This example demonstrates how the anchor network **selects semantically important tokens**, enabling **out-of-order reasoning** that is different from the conventional left-to-right decoding of standard CoT, and thus, anchoring reflects better planning behavior.\n\nFurthermore, we provide theoretical results justifying the benefits of anchoring in improved sample complexity (Sec 4.2) and likelihood modeling (Appendix A3.4).  These findings support our core motivation: that explicitly modeling token importance via anchor transition enhances both **latent reasoning** and **generation quality**. \n\n---\n\n>**W5 (Q1). Have you explored applying the proposed method to stronger language models such as LLaMA or Qwen?**\n\n\nA5. We appreciate the reviewer’s interest in scaling our method to stronger LLMs. As noted in Lines 320 - 322, the **primary focus** of this paper is on **diffusion language models**, a line of research that is still in its early stages compared to LLMs. Our goal is to improve the modeling, training, inference, and zero-shot validation capabilities of diffusion language models through our proposed **anchoring mechanism**.\n\nTo demonstrate the generality of our approach, we apply anchoring to GPT-2 scale models in a controlled autoregressive setting. While GPT-2 is not state-of-the-art, it provides a fair and transparent testbed to isolate the effects of our algorithm under similar compute budgets. Our method is modular and can be integrated into any transformer-based diffusion or AR backbone. \n\nWe thank the reviewer for showing this promising direction for our future study.\n\n---\n\n>**W6.1 (Q2). What optimizer settings did you use during training?** \n\nA6.1. We used the same **AdamW optimizer** for both anchor and denoiser with learning rate = 3e-4, $\\beta_1=0.9$ and $\\beta_2=0.999$ and no weight decay. We will expand the implementation details in Appendix C.1.3 to include these details.\n\n>**W6.2. Do different optimizer hyperparameters for the anchor network and the denoising network lead to better performance?**\n\nA6.2. Our goal is to **isolate the impact of anchoring** without extreme hyper-parameter tuning, so we use the same optimizer and hyperparameters for both the anchor and denoiser networks. Using **different optimizers or hyperparameters** may lead to better performance. Our current results already show strong gains without much hyper-parameter tuning. \n\n---\n\n>**W7 (Q3). I find Eq. (5) somewhat confusing.... Could you provide further clarification on this point?**\n\nW7. Thank you for catching this typo. In **Eq (5)**, the **right-hand side describes a categorical distribution over $y_s$**, not $z_s$. The left-hand side correctly denotes the distribution over $y_s$. We appreciate the reviewer’s attention to detail.\n\n---"}, {"review_id": "xOtbBfVxd0", "reviewer": "Reviewer_gd1Y", "summary": "The paper introduces the Anchored Diffusion Language Model (ADLM), which is a two stage framework that first predicts anchor tokens, and then predicts the likelihoods of missing tokens conditioned on the anchored predictions. This paper aims to tackle the issue of masking important tokens early in the forward process through use of anchoring. The paper compares to prior Diffusion Language Models (DLMs) on LM1B and OWT datasets and shows lower test perplexities. The paper also benchmarks against autoregressive models and shows better perplexity and MAUVE scores. The paper also demonstrates the utility of anchoring for AR models and compares to chain-of-thought approaches.", "questions": "The main questions are highlighted in the weakness section and are related to:\n1. The cost of the additional anchor network.\n2. The anchor token selection. \n\nAdditionally:\n\n* Could the authors provide more details on the approach to labelling the anchor tokens for the training. There is no algorithm containing training details or how the online anchor labelling is done within the training loop. Algo. 1 seems to be pseudo-code for the sampling implementation and not the training.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* Experimentation: The paper includes significant experimentation where they compare to the existing DLMs and show improved performance. The paper includes experiments that vary the time parameter which is also helpful for demonstrating how all DLMs perform, with different time schedules.\n* Results: The results of the experiments show that ADLMs tend to outperform the baselines/previous approaches.\n* Theory: The theory seems to be correct.\n* Novelty: The idea to build an additional anchoring network seems novel.\n### Weaknesses\n* Anchor token selection: The requirement to pre-label tokens (supervision) via a frequency-based analysis seems a weakness of the approach. The relative frequency of a token in a sequence might not be directly related to importance in some scenarios and it seems like it might require significant domain expertise. For example, in A.3.4, how would the anchor token selection operate within the sequence (0, 1, 1) and (1, 0, 0), since the main argument of that section is on the middle token $x^2$ is the anchor token. However, in both cases the middle variable is represented by a token that is more frequent (and potentially labelled as the less important token for the supervision).  \n* Requirement of large anchor network transformer: The paper writes that the anchor network adopts the transformer architecture from SEDD, with the denoising architecture using the same base architecture, but with half the number of layers. How easy is it to distinguish between the potential performance gain associated with adding significantly more parameters to the overall model and the introduction of the anchoring approach. For example, if you half the number of layers of the anchor network, (or get a similar total number of parameters to the SEDD network for ADLM), then does it still outperform the previous approaches. Overall, the question is whether the associated computational cost of the additional anchor network is work the performance gain?\n* Reproducibility: The paper asserts that it “provides open access to the data and code, with sufficient instructions to reproduce the main experimental results”.  However, given the complexity of the work, the pseudo-code would not be enough to reproduce the results of the paper. The supplementary materials did not include code.", "was_revised": true, "final_justification": "Repeat of response below: \"Thank you for your response. I will keep my score as I my concern over reproducibility is still present without a code release.\"", "rebuttal": ">**W1.1.  Anchor token selection: The requirement to pre-label tokens (supervision) via a frequency-based analysis seems a weakness of the approach. The relative frequency of a token in a sequence might not be directly related to importance in some scenarios and it seems like it might require significant domain expertise.**\n\nA1.1. Thank you for the thoughtful question. We actually have experiments with three different anchor token selection mechanisms: (i) relative frequency for LM1B and OWT (generative modeling), (ii) digit extraction for GSM8K (math reasoning), and (iii) verbs and nouns (exclude articles) in ProntoQA and ProsQA (logical reasoning). These were not well highlighted in the main draft before; in the revision we will bring all three up front.\n\n**Relative frequency based anchoring)**  As noted in our **limitations section (Lines 378–380)**, we acknowledge that using token frequency as a proxy for importance is a simplification and may not always align with semantic relevance. However, our primary goal in this paper is to demonstrate that **even this simple strategy**, when integrated into diffusion training via our **ANELBO** objective, yields substantial performance gains (e.g., up to 25\\% in PPL and outperform AR in MAUVE score).\n\n---\n\n**(Digit extraction anchoring)** In Sec 5.2.1 (and Appendix C.2.5), we have studied the use of anchoring in math reasoning using the GSM8K dataset. Here, we used the numerical digits as the anchors, and this enabled out-of-order reasoning (as shown in Lines 1740 - 1746 in Appendix C.2.5; pasted an example question below for quick access). \n\n---\n\n### Question 1 from Table 9\n\n**Prompt**  \nJanet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for \\$2 per fresh duck egg.  \n\n**Question:** How much in dollars does she make every day at the farmers' market?\n\n---\n\n**Chain-of-Thought (CoT)**  \n`«16 - 3 - 4 = 9»  «9 * 2 = 18»`  \n**Answer:** `18`\n\n---\n\n**Anchored Chain-of-Thought (ACoT) -- ours**  \n`[ANT] [ANT] «3 + 4 = 7» «16 - 7 = 9» «9 * 2 = 18»`  \n**Extracted Answer:** `18`\n\nInterestingly, ACoT first computes 3 + 4 = 7 to aggregate all consumption before subtracting from the total (16 - 7 = 9), a pattern that seems more aligned with human intuition. Moreover, anchoring enables a **look-ahead planning behavior**, which is in contrast to the commonly observed left-to-right decoding bias of standard chain-of-thought (CoT) reasoning.\n\n---\n\n**(Verbs and nouns (exclude articles) anchoring)** For logical reasoning tasks with symbolic reasoning traces from ProntoQA, we extracted verbs and nouns (excluded articles such as ‘a’, “an’, ‘the’, etc.) as the anchors. While standard multi-stage training such as in COCONUT leads to a marginal improvement from 98.8% (CoT) to 99.8%, our anchoring mechanism further enhances performance. By incorporating explicit supervision through [ANT], ACoT achieves a perfect accuracy of 100% on this relatively easier benchmark, demonstrating the effectiveness of anchoring in logical reasoning. We refer to Appendix C.2.5 for further discussion and results from ProsQA. \n\nThis particular study also opens up other intriguing questions and potential directions. Words corresponding to verbs and nouns relate to the concepts of **terminals and syntactic categories** in the **Chomsky Hierarchy** (formal language theory). With this preface, anchoring can be thought of as another means of bringing classical NLP back into statistical language models. This view opens up other grammar-based mechanisms to explore for anchor network design (e.g., multi-level hierarchical anchor network), which is an exciting direction for future study.\n  \n---\n\n>**W1.2. Regarding the Example in A.3.4…**\n\nA1.2. This example is designed as a controlled synthetic setting to **theoretically demonstrate the benefit of anchoring** when the important token is known. It is not meant to rely on frequency-based selection. Note that our main experiments are conducted on real-world datasets, where **token frequency provides a practical and scalable proxy** for importance, especially when combined with pretrained tokenizers from GPT-2 or BERT that capture semantic priors. While this proxy is admittedly simple, it consistently yields gains across diverse benchmarks. Learning anchor tokens through attention, gradients, or saliency would further enhance performance of our model. \n\n---\n\n>**W2. Requirement of large anchor network transformer: The paper writes … half the number of layers. How easy is it to distinguish between the potential performance gain associated with adding significantly more parameters to the overall model and the introduction of the anchoring approach.**\n\n\nA2. Thank you for raising this important point. We add clarification below:\n\n1. **ADLM does **not** significantly increase parameter count.** As detailed in **Lines 1526–1540** and Table 7 in Appendix C.1.3:\n   - ADLM uses **293M total parameters** (Anchor: 170M, Denoiser: 123M),\n   - This is **~5X smaller** than **Plaid (1.3B)** and **~130M smaller** than **SEDD-medium (424M)**,\n   - Yet ADLM outperforms both in Gen PPL by **4–11 points**, demonstrating high performance with lower cost.\n\n---\n2. **Ablation on architectural gains vs. anchoring.** In **Lines 276–278**, we introduce **ADLM***, which uses the same two-stage architecture as ADLM but disables the anchor loss (i.e., $\\gamma=0$). This allows a fair comparison to **MDLM** under **equal training compute**.\n   - MDLM: 23.17 PPL\n   - ADLM*: 21.79 PPL\n   - ADLM (with ANELBO): **20.62 PPL**\n\n   This shows that:\n   - The **two-stage design alone** provides improvement (MDLM → ADLM*),\n   - But **anchoring via ANELBO** is essential for achieving state-of-the-art.\n\n---\n\n3. **Summary of compute-vs-performance tradeoff:**\n\nRepeating the results from Table 7 for clarity. \n\n| **Model**         | **Params** | **Sampling Steps** | **Gen. PPL ↓** |\n|------------------|------------|--------------------|----------------|\n| Plaid            | 1.3B       | 4096               | 19.7           |\n| SEDD-medium      | 424M       | 2048               | 27.3           |\n| GGM              | 387M       | 4096               | 19.5           |\n| **ADLM (ours)**  | 293M       | 2048               | **20.1**       |\n| **ADLM (ours)**  | 293M       | 4096               | **15.7**       |\n\nWe will make this discussion more explicit in the revision.\n\n---\n\n>**W3. Reproducibility: The paper asserts that it “provides open access to the data and code, with sufficient instructions to reproduce the main experimental results”. However, given the complexity of the work, the pseudo-code would not be enough to reproduce the results of the paper. The supplementary materials did not include code.**\n\n\nA3. Our implementation builds on open-source codebases from SEDD (https://github.com/louaaron/Score-Entropy-Discrete-Diffusion) and MDLM (https://github.com/kuleshov-group/mdlm), with training done using our ANELBO objective (Eq. 7). All training hyperparameters are detailed, and ablations are included to justify their choice in Appendix C. \n\nFor evaluation, we use the source code of MDLM to compute the test perplexity. We use ReMDM (https://github.com/kuleshov-group/remdm) to compute MAUVE, Gen PPL, and entropy. Appendix C.1.1 provides links for all baselines, and Appendix C.1.2 describes our training/evaluation datasets.\n\nWe believe these details are sufficient for reproducibility and will release our full code after incorporating the reviewers’ comments to ensure clarity and accessibility.\n\n---\n\n>**Q1. The cost of the additional anchor network.**\n\nPlease refer to W2 above. \n\n---\n\n> **Q2. The anchor token selection.**\n\nPlease refer to W1 above. \n\n---\n\n> **Q3. Could the authors provide more details on the approach to labelling the anchor tokens for the training. There is no algorithm containing training details or how the online anchor labelling is done within the training loop. Algo. 1 seems to be pseudo-code for the sampling implementation and not the training.**\n\nThank you for the helpful suggestion. We clarify the anchor labelling and training process below.\n\n- **Anchor Token Selection**: As described in Lines **183–190** of the main paper, we adopt a frequency-based strategy to identify important (anchor) tokens during training. These are tokens that occur less frequently within a sequence and are likely to carry more semantic content.\n\n---\n\n- **Additional Implementation Details**: Appendix **C.1.3** elaborates on the implementation of anchor supervision and how these labels are integrated into the training loss (ANELBO). During each training step, anchor tokens are dynamically selected for each input sequence based on their token frequency, and only the tokens with frequency lower than a threshold contribute to the anchor loss term. We ablate on this threshold in Lines 1489 - 1501.\n\n---\n\n- **Code Release**: We will release the full source code—including anchor token selection, model training, and evaluation—after incorporating reviewers’ feedback. \n\nWe hope this addresses the reviewer’s concern, and we appreciate the opportunity to clarify.\n\n---"}, {"review_id": "yBoE4CtOQU", "reviewer": "Reviewer_zAqx", "summary": "The paper \"Anchored Diffusion Language Model\" introduces a novel framework, the Anchored Diffusion Language Model (ADLM), to address the performance gap between diffusion language models (DLMs) and autoregressive (AR) models in likelihood modeling and text generation quality. ADLM employs a two-stage approach: an anchor network predicts distributions over important tokens (e.g., low-frequency or semantically key words), which then guide a denoising network to reconstruct missing tokens more accurately. The method is trained end-to-end using a new Anchored Negative Evidence Lower Bound (ANELBO) objective, which enhances likelihood modeling by leveraging structured guidance from anchor predictions.\nADLM significantly improves test perplexity on benchmarks like LM1B (9.54–25.4% over prior DLMs) and OpenWebText (12.3% over MDLM), narrowing the gap with AR models. It achieves state-of-the-art zero-shot generalization on six out of seven language modeling benchmarks, excelling in long-context and domain-specific datasets (e.g., PubMed, ArXiv). Using the remasking sampler, ADLM surpasses AR models in MAUVE score, indicating superior human-like text generation. The anchoring mechanism also enhances AR models, improving reasoning in math (GSM8K) and logic tasks (ProntoQA, ProsQA) beyond chain-of-thought approaches. Theoretically, the paper demonstrates that anchoring reduces sample complexity exponentially in directed graphical models.\nContributions:\n- Introduction of ADLM, a two-stage DLM framework with anchor-guided denoising.\n- Derivation of the ANELBO objective for joint training of anchor and denoising networks.\n- Significant perplexity improvements on LM1B and OpenWebText, with strong zero-shot generalization.\n- First DLM to outperform AR models in MAUVE score for human-like text generation.\n- Application of anchoring to AR models, enhancing reasoning in math and logic tasks.", "questions": "- Why were comparisons limited to MDLM, SEDD, and BD3LM, and can you evaluate ADLM against other diffusion models (e.g., DiffusionLM, DiffusionBERT) or token prioritization methods (e.g., attention-based salience)?\n\n- How does the frequency-based anchor selection ($\\mu(\\mathbf{x}^l) \\leq \\tau$) perform compared to semantic or syntactic criteria, and can you provide an ablation study?\n\n- Can you quantify the computational cost (e.g., memory, runtime) of ADLM’s training and inference, particularly for the anchor and denoising networks?\n\n- What are the quantitative improvements for AR models with anchoring on reasoning tasks (e.g., GSM8K, ProntoQA), and can you provide specific metrics?\n\n- How does ADLM perform on larger models (e.g., GPT-3 scale) or diverse datasets (e.g., multilingual), and can you provide such evaluations?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "*Quality*\n\nTheoretical Rigor: The paper introduces the Anchored Negative Evidence Lower Bound (ANELBO) objective (Theorem 4.1, Page 5), which is rigorously derived to integrate anchor and denoising networks:\n\n$\\mathcal{L}_{\\text{ANELBO}}(\\mathbf{x}, \\mathbf{y}; \\varphi, \\psi) = \\mathbb{E}_{Z_0 \\sim q(\\cdot|\\mathbf{x})} \\left[-\\log p_\\psi(\\mathbf{x} | \\mathbf{y}_\\varphi(Z_0))\\right] + \\sum_{i=1}^T \\mathbb{E}_{Z_{t(i)} \\sim q(\\cdot|\\mathbf{x})} \\left[\\lambda_{t(i)} \\sum_{l=1}^L \\left( \\log \\langle \\mathbf{x}_\\varphi^l(\\mathbf{y}_\\varphi(Z_{t(i)})), \\mathbf{x}^l \\rangle + \\gamma \\log \\langle \\mathbf{y}_\\varphi^l(Z_{t(i)}), \\mathbf{y}^l \\rangle \\right)\\right]$\n\nThe proof, supported by Assumption 4.3 and Proposition 4.4, demonstrates reduced sample complexity ($\\mathcal{O}(K^L) \\to \\mathcal{O}(L K^{d+1})$) in anchored models, grounding the method in graphical model theory.\nComprehensive Experiments: ADLM is evaluated on LM1B and OpenWebText, achieving significant perplexity improvements (9.54–25.4% over MDLM and SEDD on LM1B, 12.3% on OpenWebText). Zero-shot generalization is tested on seven benchmarks, with ADLM leading in six (e.g., PubMed, ArXiv). The paper also assesses text quality via MAUVE scores, where ADLM outperforms AR models, and reasoning tasks (GSM8K, ProntoQA, ProsQA) with AR integration.\nStatistical Robustness: The paper reports results from two independent runs with fixed seeds, matching up to the first decimal point, indicating low variability. Additional statistical details in §C (not provided in the excerpt) further support robustness.\nLimited Baseline Comparisons: While ADLM is compared to AR, MDLM, SEDD, and BD3LM, other relevant diffusion-based models (e.g., DiffusionBERT [He et al., 2023], DiffusionLM [Li et al., 2022b]) or recent discrete diffusion advancements (e.g., Liu et al., 2025) are not included. This narrows the context for assessing ADLM’s superiority.\nAnchor Selection Simplification: The frequency-based anchor token selection ($\\mu(\\mathbf{x}^l) \\leq \\tau$) is simplistic and may not capture semantic importance in all contexts. The paper acknowledges alternatives (e.g., syntactic importance) but lacks ablation studies to justify the choice.\n\n*Clarity*\n\nClear Problem Statement: The introduction articulates the limitations of DLMs, particularly the loss of context when important tokens are masked early, and positions ADLM as a solution to narrow the gap with AR models. The motivation is compelling and grounded in prior work.\nWell-Structured Methodology: Section 3 (Page 3) clearly outlines ADLM’s two-stage framework: anchor network prediction and denoising conditioned on anchors. Figure 1 (Page 2) visually illustrates the process, enhancing understanding of token pathways (e.g., ‘cat’, ‘dog’).\nAccessible Theory: The ANELBO derivation (Page 5) is detailed, with clear explanations of terms like the anchor loss ($\\mathcal{L}_{\\text{Anchor}}$) and weighting ($\\lambda_{t(i)}$). The graphical model analysis (Page 6) connects anchoring to sample complexity, making theoretical implications accessible:\n\n$\\mathcal{L}_{\\text{Anchor}}(\\mathbf{x}; \\varphi) = \\mathbb{E}_{q(Z_{0:1}|\\mathbf{x})} \\left[ \\sum_{i=0}^T D_{\\text{KL}}(r(Y_{s(i)} | Z_{t(i)}, \\mathbf{y}) \\| r_\\varphi(Y_{s(i)} | Z_{t(i)})) \\right]$\n\nUnder-explained Anchor Selection: The frequency-based anchor selection (Page 5) is described briefly without justifying the threshold $\\tau$ or discussing its impact on performance. A short example or ablation would clarify its practical effect.\n\n*Significance*\n\nSubstantial Performance Gains: ADLM’s perplexity improvements (up to 25.4% on LM1B, 12.3% on OpenWebText) and state-of-the-art zero-shot results on six benchmarks demonstrate its potential to advance DLM pre-training, a critical area in NLP.\n\nBreakthrough in Text Quality: Achieving the first DLM to surpass AR models in MAUVE score is a significant milestone, suggesting ADLM generates more human-like text, with implications for applications like chatbots and content generation.\n\nBroad Applicability: The anchoring mechanism’s success in both DLMs and AR models (improving reasoning on GSM8K, ProntoQA, ProsQA) indicates versatility, potentially influencing diverse tasks like reasoning and controllable generation.\n\nTheoretical Impact: The exponential reduction in sample complexity ($\\mathcal{O}(L K^{d+1})$) via anchoring offers a new lens for analyzing generative models, likely inspiring further research in diffusion and graphical models.\n\nScalability Concerns: The lack of evaluation on larger models (e.g., GPT-3 scale) or computational cost details raises questions about ADLM’s scalability, potentially limiting its adoption in industry settings.\n\n*Originality*\n\nNovel Framework: ADLM’s two-stage approach—anchor network predicting important tokens followed by guided denoising—is a new paradigm for DLMs. The ANELBO objective is a novel training strategy, distinct from standard NELBO.\n\nTheoretical Novelty: The graphical model analysis and sample complexity reduction via anchoring are original contributions, offering a new theoretical lens for diffusion models:\n\n$$\\text{Sample complexity: } \\mathcal{O}(K^L) \\to \\mathcal{O}(L K^{d+1})$$\n\nCross-Model Application: Extending anchoring to AR models for improved reasoning is a creative application, distinguishing ADLM from prior DLM-focused work like MDLM [Sahoo et al., 2024] or SEDD [Lou et al., 2024].\n\nFrequency-Based Anchoring: The anchor selection method (Page 5) is straightforward and not entirely novel, as frequency-based criteria have been used in NLP (e.g., TF-IDF). More sophisticated selection would enhance originality.", "was_revised": true, "final_justification": "Thank you for the clarification. This is one of the strongest paper I've come across in this NeurIPS. I stand by the scores given the rebuttal by the authors.", "rebuttal": ">**W1. Missing comparison with DiffusionBERT [He et al., 2023], DiffusionLM [Li et al., 2022b] and DDPD (Liu et al., 2025). This narrows the context for assessing ADLM’s superiority.**\n\n\nA1. We thank the reviewer for the helpful suggestion.\n\nWe respectfully note that we already compare with **DiffusionBERT** (He et al., 2022) and **DiffusionLM** (Li et al., 2022) in **Table 1(a)**.  For completeness, we now also include **DDPD** (Liu et al., ICLR 2025), **DUO** (Sahoo et al., ICML 2025) and **UDLM** (Schiff et al., ICLR 2025), although these baselines use uniform noising, while our method focuses on masked diffusion.\n\n---\n\n### Generative Perplexity using GPT2-Large\n\nRepeating results from Table 7 for clarity. DDPD-medium is newly added. \n\n| **Model**                | **Params** | **Sampling Steps** | **Gen. PPL ↓** |\n|--------------------------|------------|--------------------|----------------|\n| Plaid                    | 1.3B       | 4096               | 19.7           |\n| SEDD-medium              | 424M       | 2048               | 27.3           |\n| `DDPD-medium (Liu et al., 2025)` | 514M       | 2048               | 24.5           |\n| **ADLM (Ours)**          | 293M       | 2048               | **20.1**       |\n| **ADLM (Ours)**          | 293M       | 4096               | **15.7**       |\n\n---\n\n### Test Perplexity \n\nRepeating our results from Table 1. UDLM and DUO are newly added. \n\n| **Method**                  | **LM1B** | **OWT** |\n|-----------------------------|---------:|--------:|\n| `UDLM (Schiff et al., 2025)`  |31.3     |27.4    |\n| `DUO (Sahoo et al., 2025)`    | 29.9     |25.2    |\n| **ADLM (Ours)**             | **26.4** | **20.6** |\n\n---\n\n### Zero-Shot Validation Perplexity\n\n| **Model**               | **Lambada** | **PTB**  | **WikiText** | **LM1B** | **AG News** | **PubMed** | **ArXiv** |\n|-------------------------|-------------|----------|--------------|----------|-------------|------------|-----------|\n|`UDLM (Schiff et al.)`    | 53.57       | 112.82   | 39.42        | 77.59    | 80.96       | 50.98      | 44.08|\n| `DUO (Sahoo et al.)`      | 49.78       | **89.35**| 33.57        | 73.86    | 67.81       | 44.48      | 40.39     |\n| **ADLM (Ours)** (262B)  | 44.93       | 98.16    | 32.45        | 65.59    | 57.10       | 38.29      | 35.08     |\n| **ADLM (Ours)** (524B)  | **44.32**   | 95.37    | **31.94**    | **64.43**| **55.72**   | **37.56**  | **33.69** |\n\n---\n\n\n>**W2. Anchor Selection Simplification: The frequency-based … justify the choice.**\n\n\nA2. Thank you for the thoughtful question. We actually have experiments with three different anchoring mechanisms: (i) relative frequency for LM1B and OWT (generative modeling), (ii) digit extraction for GSM8K (math reasoning), and (iii) verbs and nouns (exclude articles) in ProntoQA and ProsQA (logical reasoning). We will bring these results from the appendix to the main draft to better highlight these contributions. \n\n---\n\n**(Relative frequency based anchoring)**  As noted in **Lines 378–380**, we acknowledge that using token frequency as a proxy for importance is a simplification and may not capture semantic importance in all contexts. However, our primary goal in this paper is to demonstrate that **even this simple strategy**, when integrated into DLM training via our **ANELBO** objective, yields substantial improvements (e.g., up to 25\\% in PPL and outperforms AR in MAUVE score).\n\n---\n\n**(Digit extraction anchoring)** In Sec 5.2.1 (and Appendix C.2.5), we study the use of anchoring in math reasoning using the GSM8K dataset. Here, we use the numerical digits as the anchors, and this enabled out-of-order reasoning (as shown in Lines 1740 - 1746 in Appendix C.2.5). Please see for example Question 1 from Table 9).\n\nInterestingly, ACoT first computes 3 + 4 = 7 to aggregate all consumption before subtracting from the total (16 - 7 = 9), a pattern that seems more aligned with human intuition. Anchoring thus enables a **look-ahead planning behavior**, which is in contrast to the commonly observed left-to-right decoding bias of standard chain-of-thought (CoT) reasoning.\n\n---\n\n**(Verbs and nouns (exclude articles) anchoring)** For logical reasoning tasks on ProntoQA and ProsQA, we extract verbs and nouns (excluding articles such as ‘a’, “an’, ‘the’, etc.) as the anchors. By incorporating explicit supervision through [ANT], our method ACoT achieves a perfect accuracy of 100% on the relatively easier ProntoQA and 16.3\\% improvement over CoT on ProsQA, demonstrating the effectiveness of anchoring in logical reasoning. We refer to Appendix C.2.5 for further details. \n\nThis particular study also opens up other intriguing questions and potential directions. Words corresponding to verbs and nouns relate to the concepts of **terminals and syntactic categories** in the **Chomsky Hierarchy** (formal language theory). With this preface, anchoring can be thought of as another means of bringing classical NLP back into statistical language models. This view opens up other grammar-based mechanisms for anchor network design (e.g., multi-level hierarchical anchor network), which is an exciting direction for future study.\n  \n---\n\n\n\n>**W3. Ablation study on anchor threshold.**\n\n\nA3. We appreciate the reviewer’s suggestion to clarify the anchor selection threshold and its effect on performance. As noted in **Line 266**, we defer the ablation study of the threshold $\\tau$ to **Appendix C.1.3 (Lines 1489–1501)**. There, we present a detailed empirical analysis showing how different values of $\\tau$ influence performance (measured by NLL, PPL and BPD metrics). This justifies our default choice of $\\tau = 5$ used across experiments.\n\n---\n\n>**W4. Frequency-Based Anchoring: The anchor … originality.**\n\n\nA4. As acknowledged in **Lines 378–380**, the frequency-based anchoring mechanism is indeed a simplification and not claimed as a contribution. Our **primary focus is to demonstrate the impact of anchoring in DLMs**, rather than to propose methods for anchoring. Indeed, more sophisticated anchor selection would enhance performance as we have demonstrated in math and logical reasoning tasks. Please refer to Sec 5.2.1, Appendix C.2.5 and also our detailed response in A2.\n\n---\n\n\n>** W5(Q1). Why were comparisons limited to MDLM, SEDD, and BD3LM, and can you evaluate ADLM against other diffusion models (e.g., DiffusionLM, DiffusionBERT) or token prioritization methods (e.g., attention-based salience)?**\n\nA5. We already include comparisons with **DiffusionLM** and **DiffusionBERT** in **Table 1(a)**. For completeness, we now include **DUO** (Sahoo et al., ICML 2025), **UDLM** (Schiff et al., ICLR 2025), and **DDPD** (Liu et al., ICLR 2025), as detailed in our **response A1** above.\n\n---\n\n>**W6(Q2). How does… provide an ablation study?**\n\nA6. Please refer to our response in A4.\n\n---\n\n>**W7(Q3). Can you quantify the computational cost (e.g., memory, runtime) …?**\n\nA7. As already reported in **Table 7 (Appendix C.1.3)**,  ADLM has **293M** parameters for both the **anchor** and **denoising** networks combined.\n  - For comparison:\n    - **Plaid**: 1.3B parameters\n    - **SEDD-Medium**: 424M parameters\n    - **GGM**: 387M parameters\n    - **MD4-Medium**: 400M parameters\n    - **DDPD-Medium**: 514M parameters\n\n\nDespite having fewer parameters, ADLM achieves **significantly better perplexity** and **zero-shot performance** than these models.\n\n---\n\n#### Runtime Analysis (1024-token generation)\n\n| **Model** | **Sampling Steps** | **Time per Sample (s)** | **PPL ↓** | **Gen. PPL ↓** |\n|-----------|--------------------|--------------------------|-----------|----------------|\n| ReMDM      | 1024               | 19 sec                   | 23.17     | 28.6           |\n| ADLM (ours)      | 1024               | 28 sec                   | 20.62     | 25.4           |\n\nWith just **9 additional seconds per sequence**, ADLM delivers:\n- 11.0% improvement in **test perplexity**\n- 11.1% improvement in **generative perplexity**\n- Improved performance across 6 of 7 **zero-shot benchmarks**\n\n---\n\n>**W8(Q4). What are the quantitative improvements for AR models with anchoring on reasoning tasks (e.g., GSM8K, ProntoQA), and can you provide specific metrics?**\n\n\n\nA8. **Table 5** already shows quantitative improvements for AR models with anchoring on reasoning tasks:\n\n- **GSM8K** (math reasoning),\n\n- **ProntoQA** (logical reasoning), and\n\n- **ProsQA** (long-chain logical reasoning).\n\nFollowing prior works, we use **accuracy** as the standard evaluation metric. For clarity, we repeat our observation from Table 5 below: \n\nAnchored Chain-of-Thought (ACoT) consistently outperforms standard CoT and latent reasoning methods (e.g., COCONUT, iCoT). For example:\n\n- On **GSM8K**, ACoT improves over standard CoT by **+2.3%**.\n\n- On **ProsQA**, ACoT achieves **97.3%**, surpassing \\textsc{Coconut} (97.0%) and CoT (77.5%).\n\nPlease refer to **Appendix C.2** for more results. \n\n---\n\n\n>**W9(Q5). How does ADLM perform on larger models (e.g., GPT-3 scale) or diverse datasets (e.g., multilingual), and can you provide such evaluations?**\n\nA9. To ensure fair comparison with prior diffusion baselines, we train and evaluate our diffusion model on the same benchmarks: **LM1B** and **OpenWebText** using comparable model sizes. Remarkably, **ADLM (293M)** outperforms much larger models like **Plaid (1.3B)**, achieving better perplexity with **5X fewer parameters** (please refer to Table 7 in Appendix C.1.3). \n\nTraining models at **GPT-3 scale (175B)** or on **multilingual corpora** requires substantial compute and financial resources, which are currently beyond our academic scope. However, we believe that our **theoretical results**,  extensive evaluations on **standard pretraining**,  **zero-shot validation**, **complex planning/reasoning**, and robust performance across **varying inference steps** collectively demonstrate the **scalability and generality** of our proposed **anchored diffusion language model (ADLM)**."}]

Author rebuttal:

Meta-review: I recommend acceptance of this paper, which presents the Anchored Diffusion Language Model (ADLM), a novel two-stage framework that significantly improves diffusion language models' performance. The approach addresses a key limitation of masked diffusion models by first predicting distributions over important tokens through an anchor network, then using these to guide missing token prediction. This results in substantial improvements in perplexity (up to 25.4% over prior DLMs) and MAUVE scores that surpass autoregressive models.

All reviewers acknowledge the paper's technical merits, with three recommending acceptance. The authors have addressed concerns regarding architectural details, baseline comparisons, and anchoring strategies during the rebuttal phase.

However, I would like to remind the authors that they should discuss recent developments in diffusion language models, particularly LLaDA (https://arxiv.org/abs/2502.09992) and Dream-7B (https://hkunlp.github.io/blog/2025/dream/), which have significantly closed the gap between autoregressive and diffusion language models. It would strengthen the paper to discuss how anchoring techniques could contribute to these newer models, and how ADLM fits into this rapidly evolving landscape.
Decision: Accept (poster)