Gumbel Distillation for Parallel Text Generation
Authors: Chi Zhang
Xixi Hu
Bo Liu
qiang liu
Venue: ICLR 2026 Poster (2026)
Keywords: Parallel Decoding
Non-Autoregressive Generation
Knowledge Distillation
Primary area: generative models
TL;DR: A novel distillation technique that enables parallel decoders to model the complex joint distribution of token sequences.

Abstract: The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-autoregressive models often sacrifice generation quality because they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE Score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset.

Published as a conference paper at ICLR 2026
GUMBEL DISTILLATION FOR PARALLEL TEXT GENER- ATION
Chi Zhang" Xixi Hu Bo Liu Qiang Liu
Department of Computer Science
The University of Texas Austin
{chizhang, hxixi, bliu, lqiang) wtw.as.eedeeeeeooo
ABSTRACT
The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribu- tion of token sequences. To narrow this performance gap. we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distilla- tion seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LMIB and Open WebText show that Gum- bel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE Score and 10.5% in genera- tive perplexity over MDLM trained on WebTeet dataset.
INTRODUCTION
Autoregressive (AR) language models have the standard for text generation (Radford al. 2019
Brown et al. 2020; Touvron et al. (023), but their sequential, token-by-token inference process introduces significant latency, hindering their use in real-world applications. To address this bot-
tleneck, various parallel decoding methods have emerged, including Masked Diffusion Language
Models (MDLMs) (Shi et al., 2024| Sahoo et al.] 22024] Arriola et al. 2025| Ou et al.
2025 Nie
et al. 2025) and Multi-Token Prediction (MTP) (Cai al.) 2024 Gloeckle et al. 2024: Liu et al., 2024a). These non-aut aon-autoregressvve approaches accelerate inference by generating multiple tokens simultaneously. However, this speedup often comes at cost of degradation in generation quality.
This performance gap originates from a fundamental challenge in non-AR modeling (Gu et al, 2017): the difficulty of learning the joint probability distribution of entire sequence. AR models elegantly handle this by factorizing the distribution into product of conditional probabilities using the chain rule (Vaswani al. 2017). capturing the dependency of each token its predecessors. In contrast, to enable simultaneous prediction, parallel decoders operate under a naive assumption of conditional independence among the tokens generated in single step. Previous works (Xiao) et al. 2022; Liu et al., 2024b) Xu et al. 2024; Song & Zhou, 2025 Wu et al., 2025 Kim et al., 2025) have pointed out that the simplification, while necessary for parallelism, disrupts the natural sequential dependencies of language. For example, predicting the phrase "San Francisco" requires knowing that "Francisco" is highly dependent on the co-occurrence of "San" in the same block. When failing to capture these intricate local dependencies, parallel models can produce errors such as token repetition, resulting in text that less coherent or grammatically flawed.
In this work, we aim to narrow this quality gap by fundamentally improving the capacity of parallel decoders without sacrificing the speed of parallel decoding. We argue that the key issue how to make the learning problem for the non-AR student model easier. Instead of asking the model to learn the complex distribution of language from scratch, could we provide a "blueprint" from a powerful AR teacher that hints how specific output sequence was formed?
To that end, we introduce Gumbel Distillation, a novel distillation framework that enables a non- AR student to learn the complex token dependencies captured by AR teacher. The core of our method is the Gumbel-Max trick, which we use to establish a deterministic mapping from a simple
Equul contribution.

---

Published as a conference paper at ICLR 2026
Gumbel noise distribution to the target output sequence from the AR teacher. The resulting noise vector serves as the latent "blueprint" to its paired token sequence, uniquely encoding the sampling decisions from the teacher. By training the student to reconstruct the text conditioned on the Gumbel noise, we turn the difficult joint-distribution matching problem into supervised learning problem.
A key advantage of our approach is its simplicity and general applicabilliyy Because the distillation process is framed as adding a conditional input, Gumbel Distillation can be seamlessly integrated as plug-and-play module into a wide variety of existing parallel decoding architectures. It does not require complex changes to the model's structure, making it a flexible tool for enhancing different frameworks. We demonstrate this by applying Gumbel Distillation to state-of-the thrtthe-rrt architectures, including MDLM (Sahoo et al. 2024). BD3-LM (Arriola et 2025) and Medusa (Cai et al.) (024), showing serves as a simple yet powerful complement. Our experiments validate that Gumbel Distillation significantly improves generation quality, enabling high-quality multi-token decoding.
Our main contributions are as follows:
We propose Gumbel Distillation, novel framework for distilling knowledge from AR teacher into a parallel student. The method enables the student to model the distribution more accurately by learning direct mapping from Gumbel noise the teacher's output and therefore better approximating the joint token dependencies.
To demonstrate the simplicity and effectiveness of our approach, we integrate it with state- of-the-an art parallel decoding frameworks, including MDLM, BD3-LM and Medusa, improv- ing their performance while making minimal architectural changes.
Our extensive experiments validate that our method successfully alleviates the gap between autoregressive quality and parallel decoding efficiency, achieving state-of-the-ar results in high-quality, accelerated text generation with parallel decoding.
2 RELATED WORK
Diffusion Language Models Inspired by the success of diffusion models in visual domains (Ho iet al. 2020 Song et al.) 2021: Liu et al., 2022; Lipman et al. (022). a number of works have explored diffusion language models (DLMs) by extending the technique text domains. An intu- itive approach is to model text in continuous space and apply diffusion directly (Li et al.) [022] Gong et al. 2023 Han et al. 2022; Dieleman et al. 2022). To fit the discrete nature of text, an- other approach focuses
using discrete processes featuring forward and reverse dynamics, with the foundational work laid out by D3PM (Austin et al., 202 T). Numerous variants follow (Hooge-
boom et al. 1021 He et al.) 2022 Campbell et al. [022]] Sun et al. 2022; Chen et al. 2022 Wu
ct al., 2023; Zheng et al. 2023] Gat et | (024)) with masked diffusion language models being a
specific focus for recent research (Chen et al.) 2024 Shi al. 2024 Nie et al. 2024 /heng al.,
2024 Lou et al. 2024 Ou et al., [023 von Rütte et al. 2025 Kim et al. 20255 due to their strong performance. Notably, MDLM (Sahoo et al. 2024) derived simplified objective for masked diffu- sion that connects the operation of unmasking tokens to a true generative process. BD3-LM (Arriola
jt al. 2025) builds upon MDLM to introduce a hybrid approach that models sequences in block- wise manner while applying diffusion within each block, enabling flexible length generation and improved inference efficiency through kv-caching. In our paper, we apply Gumbel Distillation to these two architectures as examples to show the effectiveness and flexibility of our method.
Encouraged by the above works, recent efforts have scaled diffusion language models to billions of parameters (Nie et al. 2025 Gong et al. 2023: Ye et al.) 2025). Furthermore, commercial suc- cess like Seed Diffusion (Song et al. 2025), Mercury Coder (Labs et al.) 2025) and Gemini Dif- fusion (DeepMind 2025) have also demonstrated their practical viability, achieving performance comparable to state state-of-the-eeeeee the AR models with significantly faster speed of inference.
(MTP) strategies (Cai et al. 2024 Gloeckle et al., 2024] Liu et al.) [024a] (Samragh et al.) (025) enable faster, flexible sampling of AR lan- guage models. However, current MTP models often predict the probability of each token in a block independently. which is issue also relevant to diffusion language models. [Song& & Zhou (2025). Zhu et al. (2025) argues that this naive conditional independence assumption greatly limits the ca- pacity of the model distribution. Although the issue can be alleviated via corrections to the inference procedure (Gu et al. 2017 Leviathan et al. 2023 Santilli et al., 2023) (Shi et al.) (0024) such as spec- ulative decoding, and most notably recent works like adaptive parallel decoding (Israel et al. 2025)
CaltiiTokke-Prreiieee
Multi-token-
2

---

Published as a conference paper at ICLR 2026
Gumbel Distillation method points fundamental solution that in principle allows the model to train and infer from the true joint distribution.
Knowledge Distillation for Parallel Decoding A key application of knowledge distillation is to accelerate inference by training a parallel student model to mimic powerful AR teacher. This line of research was pioneered by Gu et al. (017)) which introduced sequence-level knowledge distillation to train a parallel student on the outputs of an AR teacher. This technique spurred the development of parallel models for machine translation (Chazvininejad et al. 2019; Gu et al., 2019). More recently, some work (Gu et al. 2023 Kou et al. 2024) have adapted the idea for general language models.
Different from these works, our proposed Gumbel Distillation distills from the AR teacher's 's internal
sampling process, which allows the parallel student to learn the full probability landscape.
BACKGROUND
The Challenge of Parallel Decoding An AR language model, denoted as p", generates se- quence of tokens 2r = (x .... xx)) one at a time. [ The probability of the sequence is factorized as p² p" = I - p" (x' ax "), where x <i represents the preceding tokens. This sequential fac- torization effectively captures the dependencies between tokens, leading to high-quality output, but at the cost of slow, iterative inference.
In contrast, parallel decoding methods accelerate this process by generating multiple tokens at once. Instead of a single token x', these models predict a subset of all tokens, denoted by x", where I is a set of token indices. The already generated tokens I, serve the context. This paradigm applies to various parallel decoding architectures; for instance, in MTP (Cai et al., 2024] Gloeckle et al. 2024 Liu al. 2024a). I is the set of indices for block of future tokens, and in MDLMMs (Shi et al. 2024 Sahoo et al.) 2024 Arriola et al., 2025 Ou et al.) 2025) Nie et al., 2025). I is the set of indices for the tokens unmask. However, enable simultaneous prediction for all indices in I, these models are often forced to adopt a conditional independence assumption. They approximate the true, sequentially dependent distribution of the AR model p* with block- wise factorized one:
po(a"" [c 7
II VET
po [x >>).
This assumption ignores the dependencies within the target set x and as a result, these architectures fail to capture the local structure essential for natural language. We aim to mitigate the quality degradation caused by this factorization.
Gumbel-Max Trick The Gumbel-Max trick (Gumbel 1954) is a reparameterization technique for sampling from categorical distribution. To sample from category X E {l...., VJ defined by the Boltzmann distribution with a vector of logits (4,, .. .v) E EV;
P(X = / exp(())
j=1
we can first draw i.i.d. standard Gumbel noise {Ex ~ g0,, 1)) and then compute:
Y arg max (4x- + -x).
&
The resulting sample Y has the exact same distribution as X. In practice, a common method to sample { ~ (O,, 1) is via inverse transform sampling, where we first sample a Uniform variable 20, 1), and then calculate the Gumbel noise as {= log log u).
One crucial insight here is that we can reframe the sampling process: the randomness is externalized into the Gumbel noise vector & (E....., RV. While categorical sampling is stochastic, the argmax function in Gumbel-Max is fully deterministic once the logits and the noise are known.
METHOD
Our primary objective is to design a parallel decoder, pe. such that its output can match the high- quality output of powerful AR teacher, p² Formally, given AR teacher model:
p² (2 1: II (a'la> where Vi, p² k|x exp (f*
Σy exp (f* (I*
Throughout the paper, we use hold symhols for vectors.
3

---

Published as a conference paper at ICLR 2026
Takens
Legits
Gumbel Noise
Gumbell-Man Sampling
Stage 1: Data Generation
Stage 2: Student Training
argmax
x²
c
CE Loss
1
g
7'
AR Teacher p (af ()
Parallel Student po(a" xt .e")
BOS
Figure 1: A conceptual overview of Gumbel Distillation. The distillation process consists of two steps: (1) Data Generation: An autoregressive teacher model's sampling process is combined with Gumbel noise to deterministically generate pairs of token sequences and their corresponding Gumbel noise. practice, one could alternatively extract {{} in parallel based on an offline training corpus (Section 4 (2) Student Training: A parallel student model is trained to predict target subset of tokens, conditioned on both the context and the Gumbel noise from the teacher.
where f (+) denotes the k-th logit predicted by the teacher, we aim to solve the following problem:
How can we train a parallel decoder po to generate a subset of tokens x. simultaneously, while ensuring its output distribution po(x: 2 x: matches the teacher's 's distribution p" (a L)?
However, this objective is notoriously difficult. A naive approach of using a neural network to di- rectly model the conditional distribution po (= (x=2) can easily become intractable. This is because the model would need to output probability distribution over a discrete space of size vizi which grows exponentially with the number of tokens [I] generated in parallel.
To overcome these challenges, key insight is to reframe the difficult distribution matching prob- lem into a more manageable supervised learning problem. We achieve this by reformulating the teacher's stochastic sampling process the AR model samples a token from a categorical distribution. The Gumbel-Max trick provides an equivalent formulation: adding random Gumbel noise the logits and taking the argmax. This reveals a crucial link: for any sequence x I:n generated by the teacher, there exists corresponding sequence of Gumbel noise El that produces it, thereby creating an explicit mapping from noise to text. Consequently, instead of directly learning the joint distribution over a block of multiple tokens, student's 's task simplified to learning this deterministic function: pa(a"" | -I eI)
fixed function of random noise. At each generation step,
4.1 GUMBEL DISTILLATION: FRAMEWORK
Building on the above intuition, we detail Gumbel Distillation as a two-stage framework (Figure The process begins with data generation to produce training instances from the teacher, followed by student training, where the parallel decoder learns the noise-conditioned mapping.
Stage 1: Data Generation In the first stage, we use the AR teacher p" to generate complete pair consisting of a full token sequence 1:n and its corresponding Gumbel noise sequence gl:n, This initial step is inherently sequential. From this complete pair, we then construct a training example for our parallel student based on its architecture. By selecting a subset of indices I C (1...., n.) to be the prediction target, partition the data into triplets (x II . x²)), where:
is the context, consisting of the tokens not in our target set. c is the conditional Gumbel noise for the target positions.
x is the target token subset that the student learns to predict.
Stage 2: Student Training Once the training data is prepared, the second stage involves training parallel student decoder P4 (e.g. masked diffusion model or multi-token-predictin model) to predict the target token subset z² simultaneously. The student model is conditioned on both the context x -I and the corresponding Gumbel noise for the target positions e and the training objective is to maximize the conditional log-likelhooo::
L (2 " (2: -I -g)].
4

---

Published as a conference paper at ICLR 2026
Algorithm Parallel Gumbel Sampling for a Sequence
Require: Ground-truth token sequence rl:= (æ) x")
Require: Sequence of logit vectors 1 1 (L' )" where each I' E RV
Ensure: Posterior Gumbel sample sequence e (E) ,...,,,)), where each e E RV
n in parallel do 2: p' +- -Softmax(())
1: for all i {1....
3: Sample auxiliary noises: G(0, and < RV 1))
4:
ç¹ < log (exp(- + exp(-Co)
v Assigns to the full vector & Overwrites a single scalar component index x²
5:
S. + So logp'.
6: end for
7:
return gl:
One crucial implementation detail in this framework is how we obtain the set of distillation targets (g!T l:)) from the AR teacher. We propose two practical methods for this extraction process. The first arises naturally from the Gumbel-Max procedure, while the second approximately equivalent method offers significant computational advantages by allowing us to parallelize the computation.
Sequential Gumbel Extraction The most straightforward approach is to perform sequential sam- pling from the teacher model p². For each position I,.... n, we first draw Gumbel noise vector g E RVV and then recursively apply the Gumbel-Mar trick to generate the next token x;,
arg max (Et f*x**))) where c = - log( log(u))); ui Uniform
By repeating this autoregressive process for the entire sequence. we can generate pairs of full se- quences (glin, _,.""), and the mapping from the noise fl:² to the text x: lm is deterministic given that the AR model fixed and deterministic. While computationally intensive, this approach allows us to generate an unlimited amount of high-fidelity distillation data, which can then be partitioned into training triplets as described above.
Parallel Gumbel Extraction While the sequential strategy is simple and robust, requires many forward passes through the teacher model to generate data. As highly efficient alternative, can leverage an existing text corpus by assuming its data was drawn from the teacher's 's distribution p". Given token sequence 20 from the corpus, we first perform single forward pass with the teacher model to get the corresponding logits [I:n The problem is then reframed as recovering the posterior x 1:n I!:)) that would reconstruct the original text.
distribution of the Gumbel noise P(g 1:n
To this end, the following theorem provides direct, analytical method for sampling from this pos- terior, enabling us to extract Gumbel noise for an entire sequence parallel. Using Algorithm [ we can sample the Gumbel noise vectors for every token in sequence via one single forward pass through the teacher model, which significantly accelerates the data generation process. A detailed version of the theorem and its proof is available in Appendix B
Theorem 4.1 (Gumbel-Mar Posterior Sampling). Assume token x is drawn from the softmax distri- bution of a logits vector 1 = (l.....,yy ,,,),, using the Gumbel-Max trick with a Gumbel noise vector E = .E.)).. i.e. x = arg (lx + Ex). A valid sample & from the posterior distribu- tion p(E]X x)) that satisfies the constraint can be drawn by the procedure which we extend for sequences in Algorithm]
4.2 GUMBEL DISTILLATION: APPLICATION
Our framework benefits from its simplicity and generality. By reformulating the learning problem as a supervised task conditioned on Gumbel noise, Gumbel Distillation can serve a versatile, plug-and and--lay enhancement for existing parallel decoders. In this section, we show how Gumbel Distillation can be seamlessly integrated into two popular families of such models: Masked Diffu- sion Language Models and Multi- Fooen--redicttoo Models.
Masked Diffusion Language Models (e.g., MDLM (Sahoo et al. 2024), BD3-LM (Arrila et al. 2025)) )) operate by training model to recover the original text from corrupted version, where masking is used for the corruption process. In our setup, given text sequence where subset of tokens xF has been replaced by [MASK], the model
Conditioning Masked Diffusion Language Models
5

---

Published as a conference paper at ICLR 2026
Start
Target
Trajectory
X Failure Point
AR Teacher Success: 100%, NFE=10
MDLM Success: 64%, NFE=3
MDLM + Gumbel Distillation Success: 94%, NFE=3
Figure 2: A toy maze problem illustrating how Gumbel Distillation helps student model learn a structured task. The goal is to generate a valid path from a start (<bos>) to a target (<eos>) using a simple vocabulary (up, down, left, right). The figure visualizes 100 generated paths from each model, slightly jittered for visualization. The baseline MDLM frequently fails, produc- ing incoherent paths, indicating its difficulty modeling the joint distribution of the sequence. In contrast, our Gunbel-conditioned model successfully finds valid paths, closely matching the perfor- mance of the AR teacher but a fraction of the generation cost (NFE 3 steps vs. steps).
learns predict the original tokens x based on the unmasked context Distillation, we simply augment the model's input. Instead of predicting the masked tokens from the context alone, the student model pe is now also conditioned on the Gumbel noise E corresponding to the target tokens. The learning objective becomes maximizing pe (x E²)
To apply Gumbel
crucial design choice is how inject the Gumbel signal. We found most effective to process the Gumbel noise for the masked tokens, E², into a vector on the model's vocabulary embedding space via a softmax normalization followed by learned linear projection. Softmax function constrains the Gumbel values to (0, 1), controlling for its large tails, while preserving the relative ranks in Gumbel that encode the teacher's sampling choice. These processed Gumbel embeddings then replace the standard [MASK] token embeddings in the input sequence. In this way, the uninformative [MASK] tokens are substituted with a rich "blueprint" from the teacher, guiding the student for prediction. Full implementation details can be found in Appendix C.I
The visualization in Figure provides a compelling toy example of this process, where we apply Gumbel Distillation to MDLM simple maze navigation task by defining the valid action quence as the language. The Gumbel-conditionen MDLM improves greatly over the baseline to match the AR teacher's performance with much lower NFE.
Models Multi-Token Prediction (MTP) models, such as Medusa (Cai et all 2024), accelerate decoding by using multiple "heads" simultaneously generate block of future tokens. At given generation step with heads, this block corresponds to the (i+ 1,..., + k.). To apply Gumbel Distillation, we train these heads to predict
Conditioning
target indices the target block æF conditioned the preceding context x and the corresponding Gumbel noise blueprint g from the teacher.
The Gumbel noise is processed into conditioning vector and provided to each prediction head. This gives the heads direct guidance on the joint distribution of the token sequence, helping them overcome the standard conditional independence assumption and propose more coherent candidate blocks for verification. Full implementation details can be found Appendix C.2
EXPERIMENTS
In this section, we empirically validate Gumbel Distillation. We demonstrate its effectiveness and generality Masked Diffusion Models (Section 5. and Multi-Tokeer Prediction Models (Section
5.2 We conclude with ablation studies (Section 5.3 to justify our method's design.

---

Published as a conference paper at ICLR 2026
Table I: Main results for unconditional text generation. AR denotes an autoregressive model with the same parameter as the student backbones (used only as a quality reference); the distillation teacher is GPT-2-Large. Models are trained for 1M steps on LMIB (length 128) and Open OpenWeeText (length 1024). Best non-AR scores are in bold.
LMIB
Web
MAUVE 1
GenPPL 1
MAUVE
GenPPL
AR (student-size)
0.465
36.42
0.691
14.10
MDLM
0.179
78.74
0.217
38.34
MDLM Gumbel Distillation
0,264
67.64
0.282
34.33
BD3-LM (L' 4) BD3-LM (I'' = 4) + Gumbel Distillation
0.193
56.98
0.251
26.40
0.291
46.06
0.304
24.37
Qualitative evaluation using Gemini-2.5-pro as LLM judge on WehTee. Scores are on 1-10 scale (higher is better). Gumbel Distillation shows clear improvements across most dimensions, with the most significant and consistent gains observed in Clarity and Factuality,
Table 2:
Model
Clarity
Grammaticality
Factuality
Style
Creativity
MDLM
2.44
2.22
2,70
2.32
2.22
MDLM+ Gumbel Distillation
2.86 (417.8)
2.57
(3330)
3.31 (2220))
2.57
2.36 (4a7s)
BD3-LM
2.89
2.95
3.21
3.34
2.75
BD3-LM Gumbel Distillation
3,41 (11829)
3,22
(19.6)
3.78 (17779)
3,35 (4224)
2.68 6259)
5.1
GUMBEL DISTILLATION FOR MASKED DIFFUSION MODELS
Experimental Setup. To evaluate Gumbel Distillation, we integrate it into two state-of-the-ar masked diffusion language model frameworks: MDLM and BD3-LM. For all distillation experi- ments, the autoregressive teacher is GPT-2-Large. For our main results, we use the efficient parallel Gumbel extraction method (Section 4.1) to generate distillation pairs online while iterating through the training data. We compare against two primary baselines: (i) an AR model with the same pa- rameter budget as the student backbones (for a fair quality reference), and (ii) students trained from scratch (MDLM//D33-LM) on raw text without distillation. Full implementation details are in Appendix
Results on Unconditional Text Generation We evaluate performance using two key metrics. Generative Perplexity (Gen. PPI., lower is better) measures token-level fluency. For a more holis- tic assessment of quality and diversity, we use the MAUVE score (Liu et al., 2021) (higher is better), which compares the generated text distribution to a human reference. As shown in Table I our method, Gumbel Distillation, consistently and significantly improves the performance of the baseline models in unconditional text generation. For instance, when applied to MDLM on Open- WebText, Gumbel Distillation reduces generative perplexity by 10.5% and boosts the MAUVE score by 30.0%. These substantial gains are consistent across other results on BD3-LM, demonstrating approach's general effectiveness.
LLM-based Evaluation To complement our statistical metrics with a qualitative assessment, we used Gemini-2.5-pro (Comanici et al.) (025) score the text samples generated by models trained on OWT. As shown in Table 2 we use Gemini-22.5pro to judge all models on five dimensions: clarity, grammaticality, factuality, style, and creativity. Full details on the evaluation prompt and setup can be found in Appendix D.3 The results are consistent with our earlier findings using Gen. PPL and MAUVE score, confirming that Gumbel Distillation leads to a clear improvement in perceived generation quality. For both MDLM and BD3-LM, we see consistent gains across all dimensions, with particularly strong improvements Factuality and Clarity. This suggests that the Gumbel conditioning helps the student model generate more coherent and grounded text.
Performance vs. Number of Sampling Steps In Figure 3 we plot MAUVE score and generative perplexity against the number of inference steps (NFE), using the efficient ancestral sampler from Sahoo et al. (2024) for all models. The results show two key advantages. First, at any given NFE, our models achieve a stronger perplexity than the baselines. Second, our models reach a high level of quality in far fewer steps compared the baseline models. This efficiency gain suggests that
7

---

Published as a conference paper at ICLR 2026
MDLM
MDLM Gumbel Disiilation
0.5
270
50-
120
NAUIEE
Gen
0.3
-20
10
32
04 123
04
220
236
512
1024
04
120
250
322
3524
LMIB
WebTex
Figure 3: Generative Perplexity and MAUVE Score vs. Number of Function Evaluations (NFE)
of MDLM on LMIB and Open WepnWeeTTe WebText Text. Our method (Gumbel Distillation) consistently outper- forms the baselines. achieving lower perplexity for given number of evaluations.
MDLM
MDLM + Gumbel Distillation
BD3-LM
BD3-LM Gumbel Distillation
60
02.688
51.7314
49.0 49,2204 52.6222
Acouracy (%)
48.
40
41
28-
32.122.4
31,02033
26.3
36.1
23
20
10.1
ARC-c
ARC-e
BoolQ
HellaSwag
PIQA
OBQA
WinoG
Avg.
Figure 4: Zero-shot performance on common-sense reasoning and question answering bench- marks. Our Gumbel Distillation consistently improves accuracy over the baseline models across suite of eight tasks. All scores are percentages (%).
the Gumbel "blueprint" provides more direct signal, enabling the student model to converge high-quality output during the iterative generation process.
Zero-shot Benchmark Performance We evaluate the language understanding capabilities of the models trained OWT wide range of question answering tasks. Following von Rütte et al. (2025) our selected benchmarks include ARC-e and ARC-c (Clark et al. 2018), BoolQ(Clark et al, al. 2018 and Winogrande (Sakaguchi et al. 2021). The results, presented in Figure 4 show that Gum- bel Distillation effectively transfers the reasoning capabilities of the AR teacher to the parallel stu- dent models. For instance, applying our method to MDLM improves its average accuracy across all benchmarks from 34.3% to 36.1 1 We observe consistent gains across the board, with a particularly notable improvement in BoolQ. Similar improvements are seen when applying our method to BD3- LM (from 38.6% to 39.4%), demonstrating the robustness of our approach. This demonstrates that the student not only mimics the teacher's fluency but also inherits its knowledge and common-sense reasoning abilities, substantially improving the performance on these challenging tasks.
2019 HellaSwag (Zellers et al. 2019), PIQA (Bisk et al., 2020), OpenBookQA (Mihaylov
5.2 GUMBEL DISTILLATION FOR MULTI-TOKER PREDICTION
We apply Gumbel Distillation to a MTP framework based on Medusa (Cai et al. 2024). This setup provides a direct, single-step test of model's 's ability to capture the joint distribution of sequence.
Experimental Setup. We follow the Medusa-1 setup, using frozen GPT--2-Small backbone and training parallel MTP proposal heads (offsets 1-3, head 0 uses the frozen base LM head) on Open- WebText. The frozen backbone serves as the verifier, and we apply Gumbel Distillation in a self- distillation setup where the backbone also provides the teacher logits. To test scalability, we ad- ditionally apply Gumbel Distillation Medusa heads on a frozen Vicuna-7B backbone. For all experiments, we use typical acceptance for speculative decoding and report the conditional accep- tance of each head: a token proposed by later head only verified if all preceding tokens in the block have already been accepted. Full implementation details are in Appendix C.2
Results and Analysis The results in Table 3 shows that Gumbel conditioning consistently im- proves pea- acceptance rates and increases the average number accepted tokens per decoding step. On GPT-2-Small, the relative gains grow with head index, from +4.5% on Head I to +22.0% on Head 3. Crucially, these gains persist when scaling to Vicuna-7B, from +8.9% on Head 1 to +37.6% Head 3 This trend provides strong evidence that Gumbel Distillation helps the MTP
8

---

Published as a conference paper at ICLR 2026
Table 3: Per-position acceptance rates Medusa MTP heads. We report conditional accep- tance rates under typical acceptance. Are denotes relative improvement over the baseline. Top: frozen GPT-2-Small trained on OpenWebTex, (self-distillation).. Bottom: scaling to frozen Vieuna- 7B trained on 60k ShareGPT samples (Medusa recipe; Head 0 uses the frozen LM head).
Baseline
Gumbel Distillation
7
GPT-2-Small backbone (OpenWebText)
Head (NTP)
1.000
1.000
Head
0.445
0.465
(+1,5%
Head
0.310
0.344
(+11.0%)
Head
0.264
0.322
(-22.0%
Accepted Length
1.609
1.691
(45.199)
Vicuna-7B backhone (ShareGPT)
Head (LM head)
1,000
1,000
Head
0.545
0.593
(+8.9)
Hcad
0,302
0.388
(-28.19)
Head
0.214
0.294
(437.69)
Accepted Length
1.745
1.891
(+8.4%)
heads learn the stronger sequential dependencies required to predict tokens further into the future, directly addressing the core challenge of modeling a joint distribution.
5.3 ABLATION STUDIES
Table 4: Comparison to outcome-based distillation baselines and APD MDLM (LMIB). Token-level and sequence-level KD are outcome-basse AR to NAR distillation baselines; APD is applied only inference time. Best MAUVE and best GenPPL are in bold.
Method
MAUVE t
GenPPL
MDLM
0.179
78.74
Token-level KD
0.166
95.88
Sequence-level KD
0.169
99.48
APD
0.203
57.61
Gumbel Distillation
0.264
67.64
Gumbel Distillation APD
0.255
49.28
Additional Comparison to Classical Distillation Baselines and APD. We compare Gumbel Dis- tillation outcome-based AR to Non-AR distillation baselines well as inference-time acceler- ation method. Specifically, we consider two forms of Knowledge Distillation (KD): (i) token-level KD (Hinton et al. 2015). which matches the teacher's per-position output distribution (e.g.. via KL divergence between teacher and student logits), and (ii) sequence-level KD (Kim & Rush 2016), which trains the student on sequences generated by the AR teacher. We also test Adaptive Paral- lel Decoding (APD) (Israel et al.) 2025), an inference-time procedure that uses lightweight AR verifier to adaptively accept the longest high-quality prefix from diffusion proposals. Unlike KD and Gumbel Distillation, APD does modify training: it is orthogonal and can be applied top of any trained model, including our Gumbel-distilled models. All methods are evaluated on the same MDLM backbone and LMIB protocol above. Results shown in Table 4 KD primarily aligns per- per-position marginals under the student's conditional-independenn assump- tions within parallel block, which does not enforce coherent joint decisions and can even conflict with the original masked-diffusion training objective, leading to worse fluency; Sequence-level KD replaces the training distribution with teacher-gemerated text, which can reduce diversity and encour- age mode collapse, yielding worse MAUVE/GeePPLL in our setting. In comparison, Applying APD on top of disiee-d MDLM yields the best GenPPL, with modest reduction MAUVE.
Token-level
Impact of Sequential Parallel Gumbel Extraction As previously discussed in Section we proposed two methods for generating the (noise, text) ) pairs for distillation: Sequential Ex- traction, which generates text via ancestral sampling from the teacher, and Parallel Extraction,

---

Published as a conference paper at ICLR 2026
Table 5: Consolidated ablation study using MDLM on LMIB. Parallel Gumbel Extraction is
superior to the sequential method. The specific Gumbel distribution is critical, as replacing it with Gaussian noise degrades performance, and Uniform noise leads to mode collapse.
Parallel
Sequential
Gumbel Noise
Gaussian Noise
Uniform Noise
MAUVE Score(t)
Gen. PPL (C)
Gumbel Extraction
Gumbel Extraction
0.264
67.64
0.189
86.38
0.242
81.43
0.097
which recovers the posterior Gumbel noise from an existing text corpus. Although we have provided theoretical support for the equivalence of these two methods, it still remains to be validated empiri- cally whether Parallel Extraction works as well as Sequential Extraction in practice, Therefore, we conduct ablation experiments by obtaining a dataset with the same size as LMIB via the sequential method, and then train MDLM with and without Gumbel Distillation on this synthetic dataset.
Table 5 shows that Parallel Gumbel Extraction yields better performance than its sequential coun- terpart, reducing generative perplexity from 86.38 to 67.64. This result may seem counterintuitive, sequential sampling generates data directly from the teacher's distribution. We suppose that susceptible to the teacher model's own errors and biases. If the teacher generates repetitive or low-quality text, the student will be trained to replicate these imperfections. In contrast, Parallel Extraction leverages a high-quality, diverse text corpus as the ground truth for w It then uses the teacher only infer the posterior Gumbel noise e that would have led the high-quality text.
Impact of Gumbel vs. Other Noise Sources Our method's core principle the deterministic link between Gumbel noise and token probabilities established by the Gumbel-Max trick. To check whether the specific properties of the Gumbel distribution are critical or harmful, we ablate our approach by replacing it with two other noise sources.
(1) Gaussian Noise: A standard random signal commonly used in deep learning. which we transform Gumbel into and then feed as input; (2) Uniform Noise: A particularly important baseline, it is the precursor to Gumbel noise in the inverse transform sampling procedure (g = - log( log(u))). This directly tests if the specific transformation the Gumbel distribution necessary, if any simple random variable is sufficient.
As shown in Table 5 both alternatives fail to match the performance of Gumbel noise. Using Gaussian noise as the conditional signal significantly degrades performance, resulting in worse Gen. PPL than the naive baseline. Using Uniform noise leads to training instability and mode collapse, where the model generates extremely low-diversity text. The choice of Gumbel noise fundamental to creating a structured "blueprint" that the student model can effectively learn from.
CONCLUSION
In this work, we introduced Gumbel Distillation, a novel distillation framework designed help parallel decoders overcome their struggle to model the joint distribution of token sequences. By using the Gumbel-Max trick to create a deterministic "blueprint" from an autoregressive teacher's sampling process, Gumbel Distillation reframes the intractable distribution matching problem into a simple supervised task. Our experiments demonstrate that it is a versatile, plug-and-play module that improves the quality, efficiency, and reasoning capabilities of diverse architectures like Mask Diffusion Models and MTPs. Future work could involve scaling this technique to larger foundation models and exploring the latent Gumbel space as a new avenue for controllable text generation.
Limitation. One limitation of this work is that as vocabulary size V grows, the dimensionality of the Gumbel noise vector grows proportionally. In our implementation this adds single pro- jection of cost OVV H) (applied once per sequence/block) and does not scale with model depth, so the relative parameter/compute overhead typically decreases as the transformer backbone scales (Appendix B.5). Still, this can lead to higher computational costs and more challenges in high- dimensional spaces. Future work could further mitigate this cost via structured or low-rank repre- sentations of the Gumbel noise. Despite this, our work paves the way for developing highly efficient generative models without compromising on quality, making powerful generative Al more accessi- ble for real-world applications.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Chaochao Yan for helpful discussions and feedback. We also thank members of the UT Machine Learning Lab for comments and for computational support. This work supported in part by the Institute for Foundations of Machine Learning (IFML), the Office Naval Research (ONR) under Grant No. 400014-25-1-2354 and a grant from Google.
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models. In International Conference Learning Representations, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, volume 33, pp. 1877- 1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. In International Conference Machine Learning, pp. 5209-5235. PMLR, 2024.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnau Doucet. A continuous time framework for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-282799 2022.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. 2014.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion foreing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 17:24481-221225 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiss Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208 04202, 2022.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of NAACL-HLT, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0803.0545, 2018
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. discourse- aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pp. 615-621, 2018
11

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0666/ 2025.
DeepMind.
Gemini diffusion.
-i-nii--iiii gemini-diffuse wwiddddfffiioo/o/ 2025.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffu- sion for categorical data. arXiv preprint S..iv:2211..50055 2022.
Nima Fathi, Torsten Scholak, and Pierre-André Noël. Unifying autoregressive and diffusion-based sequence generation. arXiv preprint arXiv:2504.0644 2025.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL wetddoeodooooooooooooo
126086602
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching, Advances in Neural Information Processing Systems,33 37: 133345-133388 2024.
Marjan Ghazvininejad, Omer Levy, Yinhan Yin, and Luke Zettlemoyer. Mask-predict: Parallel de- coding of conditional masked language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In International Conference on Machine Learning, 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http::/9kkyLooo000ggtthu OpenWebTexttorpus 2019.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. The Eleventh International Conference on Learning Representations, 2023.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. In International Conference Learning Representations, 2025.
Jiatao Gu, James Bradbury, Caiming Xiong, Vietor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprint arXiv: 1.02281, 2017
Jiatao Gu, Chang Wang, and Jianbo Zhao. Levenshtein transformer. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2019.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models. arXiv preprint 230.:23306.08433,, 2023.
F.J. Gumbel. Statistical Theory of Extreme Values and Some Practical Applications: A Series of Lectures. Applied mathematics series. U.S. Government Printing Office, 1954. URL https:
//books. wwwgle.ccmm/oooooooooooooooooooo owt.:m//AAAookksaaaaaoooAAAaa
Xiaochuang Han, Sachin Kumar, and Yulía Tsvetkov. Ssd-Im: Semi-auttoeerrsssive simplex- based diffusion language model for text generation and modular control. arXiv preprint mrXiv:2210.17432 2022.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang. and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15000 2022.
12

---

Published as a conference paper at ICLR 2026
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint Stt:::5503.25333 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. International Conference on Learning Representations, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows
and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 34: 12454-12465, 2021.
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding. arXiv preprint arXiv:2506.0044 00413, 2025.
Jacyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du Sham Kakade, Timothy Ngo- tiaoco, Sitan Chen, and Michael Albergo. Any-order flexible length masked diffusion. arXiv preprint arXiv:2509.01025 2025.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2016.
Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where find them: The gumbel-top-k trick for sampling sequences without replacement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning, 2019.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Clims: Consistency large language models. In Forty-first International Conference on Machine Learning, 2024.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang. Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, al. Mereury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506. 120006.77290 17298, 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, 2023.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprint arXiv:2410.01949 2024b.
Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Diver- gence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals. In NeurIPS, 2021.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0330 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning. 2024.
Chris J. Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances in Neural Information Processing Systems. MIT Press, 2014.
13

---

Published as a conference paper at ICLR 2026
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-3300 1993. URL
org/ (00/g//9300047
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint urXiv:1609.000 07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the Annual Conference on Empirical Methods in Natural Language Processing, 2018.
Shen Nic, Fengqi Zhu, Chao Du Tianyu Pang. Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410. 0ene:2400.888144 18514, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint 0rXiv:2502.09992 2025.
Jingyang Ou, Shen Nie. Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference Learning Representations, 2025.
D Paperno, G Kruszewski, A Lazaridou, QN Pham, Raffaella Bernardi, S Pezzelle, M Baroni, G Boleda, and Fernández. The lambada dataset: Word prediction requiring a broad discourse context. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016- Long Papers, volume 3, pp. 1525-1534. Association for Computational Linguistics (ACL), 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- genee frontiers. Advances in Neural Information Processing Systems, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Mohammad Samragh, Arnay Kundu, David Harrison, Kumari Nishu, Devang Naik, Minsik Cho, and Mehrdad Farajtabar. Your 11m knows the future: Uncovering its multi-token prediction potential. arXiv preprint arXiv:2507. 1851, 2025.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Ric- cardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via paral- lel decoding. arXiv preprint 20X5v:2305.10427, 2023.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnad Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 108131-103161 2024.
Makoto Shing and Takuya Akiba. Diffusionblocks: Blockwise training for generative models via score-based diffusion. arXiv preprint arXiv:2506. 14202, 2025.
Jiaming Song and Linqi Zhou. Ideas in inference-time scaling can benefit generative pre-training algorithms, arXiv preprint arXiw:2503.0715. 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- fional Conference on Learning Representations, 2021.
14

---

Published as a conference paper at ICLR 2026
Yuxuan Song, Zheng Zhang. Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.021933 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16755 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion. International Conference on Machine Learning. 2025.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. In ICLR Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22611 22618, 2025.
Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957-333 -99994444 2023.
Yisheng Xiao, Lijun Wu, Furu Meng, Tao Qin, and Tie-Yan Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovee, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. arXiv preprint arXi::2410.21357 2024.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguisties, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint 302iv:230..05737, 2023.
Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. DijMJo: Distilling masked diffusion models into one-step generator. arXiv preprint arXiv:2503. 2025.
15

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
Regarding paper writing, we used LLM only for text polishing and grammar correction during manuscript preparation. No LLMs were involved in the conception or design of the method, periments, or analysis. All technical content, results, and conclusions have been independently verified and validated by the authors. Apart from this, also used LLM evaluation tool in our experiments to judge the generation quality of text samples.
B ON THE VALIDITY OF THE POSTERIOR GUMBEL SAMPLING METHOD
In this section we give a formal proof of Theorem 4.1 Similar techniques on the posterior Gumbel distribution have been explored in prior works such as Kool et al. (2019). and we use some estab- lished findings from Maddison al. (2014). Before proceeding with the proof, we first restate the full theorem from the main text.
Theorem (Posterior of the Gumbel-Max Process). Assume X E [V] is sampled from a set of logits == (l,y) R using the Gumbel-Max trick, such that X arg max Ek), where
vk, &K G(O, 1) are i.idd standard Gumbel noises. Let pa Σ, exp(t,) be the softmax probability for each category k E [V] and & = (fi...... Σv). The posterior probability density function over & x given by:
expíla)
conditioned on the outcome
V
plE
II
e fu-
(1)
Ps.
k=1
where ( is the indicator function.
Furthermore, a sample can be drawn from this posterior by first sampling auxiliary i..d. variables Sy ~ G(O, 1) and then applying the construction:
So
log Pxr
if =
c
=
log exp(-<<) exp(-<o))
if +
This constructed sample g' is guaranteed to have the following two properties:
1. (Condition satisfaction) The argmax of the perturbed logits matches the conditioned outcome.
arg max(44 (k)
kE(V]
(Marginal Preservation) Each component of the sample is marginally a standard Gumbel,
Tk: EV], & ~ G(0,1)
B. PROOF OF PROPERTY
Proof. We show that for any l x, the inequality le E/2 > le + E holds. By substituting the construction formulas and the identity 1 - 1 = log Ps log Pk. the inequality becomes:
(ep((-<))
(log Pa. log.pk Co log
log
+ exp(-SSo)
Pk
exp(-Cx) + exp( (Sa)
-log Pk: + Co - -log
Pk
Pk
Co > log
exp( (c) exp(-Go)
Exponentiating both sides and rearranging yields:
e Ek
+1>10 Pk
This inequality holds because the left-hand side is always strictly greater than 1, while Pa E [0, 1].
16

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF PROPERTY
We prove that each component of the constructed sample & is marginally distributed as a standard Gumbel, ( 1). We handle the winning (A: x) and losing variables o z) separately.
CASE THE WINNING VARIABLE (Ex)
Proof. The proof for the winning variable's marginal distribution relies on the Posterior the Maximum property of the Gumbel distribution. This property states that the maximum score, M max (x). conditioned on the winning category being æ, follows a Gumbel distribution centered at the winning logit 22:
(M arg max Ex) x) G(1x:1)
KE[V]
The posterior construction is equivalent to a top-down sampling process where sample of the maximum score, call it m", first drawn from this posterior distribution. Thus, ~ G(1,, 1).
By definition, mn" must be equal to the score of the winning category from our constructed sample:
m' = l + &
By rearranging the equation, we can express the constructed Gumbel noise & in terms of m":
& m I
Since m" is randomly sampled from G(L,, 1), subtracting the location parameter L. simply shifts the distribution's 's location to zero. Therefore, the resulting random variable & follows:
&x ~ G(Le - L2 1) = G0
Thus, the marginal distribution any winning variable & is the standard Gumbel distribution.
CASE 2:
THE LOSING VARIABLE (Ex, k x))
Proof. The proof for the losing variables is more involved. We show that the cumulative distribution
function (CDF) & for any + x matches the standard Gumbel CDF, F(t) = -
The construction formula for losing variable is:
eexp(= Gx) exp(-Go)
εa -loo
Pk
The proof proceeds with change of variables. We use the property that if ( ~ 9(0, 1), then its transformation U - follows the standard Exponential distribution, U ~ Exponential(1). Let Us = - and Uo = <o. Both are i.i.d. samples from Exponential(1).
The CDF, P(Ex t), equivalent the probability P
Ui
+ Uo 2 e-t). This probability is found
Pt
by integrating the joint PDF U2 and Uo. which f(uR, uo) E us -o for uk uo 0, over the region satisfying the inequality. Letting c =e e- the integral is:
Pf S t)
+20200 ukke dug dup
While we omit the full integration here for brevity, it simplifies to the CDF.
P(Ex S e =c
Thus, the marginal distribution of any losing variable &n. is the standard Gumbel distribution.
Theorem (Posterior of the Gumbel-Max Process). Assume X [V] is sampled from a of logits I 1= (l.,..,,lv) lv) E IRY using the Gumbel-Max trick, such that X = arg max (I. + Ex), where vk, E ~ G(, 1) are ii.dd standard Gumbel noises. Let pk = exp(la) be the softmax probability exp(l,)
17

---

Published as a conference paper at ICLR 2026
for each category & E [V] and & (E1,... .vv). The posterior probability density function over conditioned on the outcome = x is given by:
V
P =
1
II
e 6
(3)
Pa.
k=
where (() is the indicator function. Furthermore, a sample E can be drawn from this posterior by first sampling auxiliary ii.. variables Co...., Sy G(,, 1) and then applying the construction:
Co llogg
if = x
4 - 1 log (exp(-Cx) + Pa exp(-Co)) ifk +
(4)
This constructed sample g' is guaranteed have the following two properties: 1. (Condition satisfaction) The argmax of the perturbed logits matches the conditioned outcome.
arg max (lx + [k) =
kE(V]
(Marginal Preservation) The unconditional marginal distribution of the constructed sample matches the standard Gumbel prior.
4k EV], E ~ 9(011)
B.3 PROOF OF PROPERTY
Proof. We show that for any k + x, the perturbed score the winning category strictly dominates,
i.e., 11 E22 le Eh- Substituting the construction formula for the winning variable (k x)),
have:
4 ++ 1 Co log P- = Co + log Z
where we applied the softmax identity Po 1, log Z.
For any losing variable (k * 2), since S4 ~ G(0, 1), its exponential is strictly positive (cx)
0). This yields the strict inequality:
exp(-Cx) pk exp( ph exp(-Co)
Because the negative logarithm is strictly decreasing function, applying log(-) both sides reverses the inequality:
& = log (exp(- + Ps exp((Co))
< log (px exp(-(o) = log pk
Adding la to both sides and applying log = 1 log Z, we obtain the score for class A:
4 + E. < 1 + Co - log Pt= = Ço + log Z
Comparing the two scores, l, + & = So + log Z > 1 + E. Thus, arg max (1s + +%) = x.
B.4 PROOF PROPERTY
Proof. To rigorously prove Marginal Preservation, must show that the unconditional distribution of each constructed component & is exactly the standard Gumbel distribution G(0, 1).
Instead of working directly with the Gumbel PDF, apply the transformation Eg exp(-E)) - Since & ~ ( 1) < exp(-E) ~ Exp(1), our objective equivalent to proving that the unconditional survival function of E, is P(E, t) -t for any 2 0.
Let Un = exp(-Gn) and U2 exp(- Since Sa- Sa ~ G(0 1) are independent, Uo and Ug are independent standard exponential variables, Uo, Uk Exp(1).
18

---

Published as a conference paper at ICLR 2026
By the Law of Total Probability, condition whether class le is the winning category (X k which occurs with probability pk) or a losing category (X * k, which occurs with probability 1 pk):
P(Ex 2)
= =
P(Ex > t X = k) + (1 pk) P(E, > x
(5)
Case 1: X (The Winning Category) From the construction formula, E = 5o log Pk- Transforming to the exponential domain:
Es exp(-(So - logp:)) = Pa exp(-Co) = -o
The conditional survival probability is:
P(pnUp > = P ( - exp
P(Es > 2
(6)
Pk.
Case 2: X * (The Losing Category) From the construction formula, & log(exp(-GA + Ph (x)(--o)). Transforming to the exponential domain:
F = exp(-Ca) + Ph exp( (n) = Un + PPUU
The conditional survival probability is P(U P&U0 t). Since UK. Uo ~ Exp(1) are independent, their joint PDF is [(ux, u0) = e uxe We integrate this over the region + Paooo > 1:
P(Ux + UuU > t) ///
dug e max(0,;t-PMOO)
dup 42
f/D
e e- e ( duo fo e
( duo
s/p2
klpp
=
uuall-ppi duo + exp
1
Pk:
1/Dk
=e-
c -up(1-pa)
+ eex
(1 pk)
-0
Pk:
-
cxx cxp
(-
t
(1 (pkpp)
pk) ))
+ exp
(-
t.
1
Pk
Pk:
pk
E
exp(-l/pp)
+ exp
Pk
1-ps Pk
Pk
Pạ exp(-t/pp)
Pk
Combining the Cases: Substitute the results from Case 1 and Case back into the Law of Total Probability:
t
--_ Pk (txp()
P(Ex > 1) pk exp
pk)
Pk:
Pk
t
t
Pk exp
+ -t
Pk exp
Pk:
Pk
-t
The result P(E, > t) e is exactly the survival function of the standard Exponential distribution. Therefore, Es ~ Exp(1), which implies & = - log Es G(0, 1). This strictly completes the proof of marginal preservation.
B.5 COMPLEXITY OF GUMBEL CONDITIONING
Gumbel Distillation conditions the student Gumbel vector E RV via single projection WE E #xV into the model hidden dimension H. This introduces VH additional parameters and extra matrix multiplication per block, independent of the number of transformer layers. In contrast, the backbone transformer cost scales roughly with OLH?). so the relative overhead typically decreases as I. and H increase.
19

---

Published as a conference paper at ICLR 2026
Table 6: Parameter overhead of the Gumbel projection matrix (We #
Overhead is
V TT/TooalParamss Large vocabularies increase overhead, while larger backbones reduce it.
Model scale
Layers L
Hidden H
Vocab V
Added params VH
Overhead
GPT-2 Small
12
768
50,257
38.6M
33.0%
GPT-2 Large
36
1,280
50,257
64.3M
8.3%
LLaMA-7B
32
4,096
32,000
131.1M
1.9%
LLaMA-13B
40
5,120
32,000
163.8M
1.3%
Qwen-7B
32
4,096
152,000
622.6M
8.9%
C EXPERIMENTAL DETAILS
C.I MASKED DIFFUSION LANGUAGE MODELS
MDLM and BD3-LM Text diffusion models (Austin al. 2021 Li et al. 2022) extend diffusion from visual domains to language modeling. The design is to reverse a forward process that gradu- ally corrupts clean text into noise, which can continuous or discrete representations. A denoising model is trained to recover the original text I from a noisy version I; where E [o. 1] is the noise level. The learning objective is typically the minimization of the Negative Evidence Lower Bound (NELBO), which can be expressed in simplified form as:
log Po (x: < CNELBO 8) E, (- [-w(t) log po mx)).
A prominent family of text diffusion models uses masking as the corruption process. Masked Diffu- sion Language Model (MDLM) (Sahoo al.) 2024) is a simple yet effective architecture that learns to predict original tokens given partially masked sequence. However, MDLM is designed only for fixed-length generation. To address this, Block Discrete Denoising Diffusion Language Model (BD3-LM) (Ariolaa et al. (025) extends the idea by interpolating between diffusion and AR models. generates text autoregressively over blocks of tokens, but within each block, it uses a diffusion process identical to that of MDLM. This hybrid approach enables variable-length generation. Be- cause the core objective within a block is the same for both models, we use the simpler MDLM architecture for illustration in Figure but use the full BD3-LM objective for our derivations below.
The BD3-LM objective factorizes the loss across blocks as follows:
LNELBO (x 1m
:m;
LNELBO (b- 1)0+ t:be 1bb-1)0;
(8)
b=1
Here, the loss for each block is conditioned the clean, previously generated blocks.
We adapt this block-wise objective to be conditioned on our Gumbel signal. The overall task is to learn the conditional distribution (0.2:imm(l and we achieve this by modifying the BD3-LM objective from Equation [ adding the corresponding slice of the Gumbel sequence as condition for each block. This results conditional NELBO:
LNELBO 1= :1:a;0)))
LNELBO (b- 1)0 (D-
((b-))) e(th (1b--))etsbee 1:he,
(9)
As mentioned in Section 4.2 a crucial implementation detail how the Gumbel signal &l.m is injected into the model. In practice, we process the raw Gumbel noise for each block, &", into dense vector on the embedding space, 9(5"). by first performing a softmax operation for normalization, followed by learned linear projection. Then we
replace the embeddings of [MASK] tokens in the noised input. Formally, let E(x²) ) be the initial token embeddings for a noised block b, let M be a binary mask where Mt- 1 the i-th token is [MASK] and 0 otherwise, and let W Wpros be the
weight matrix of the linear layer. Then, the final input to the model's transformer layers, E(a)), is:
-1 M²) E(x;) + 1 O "(E")) where 9(5") softmax (5")W proj. (10)
Here @ denotes element-wine multiplication. This design substitutes the uninformative [MASK] embeddings with Gumbel signals, which serve as a "blueprint" for what to predict at those positions. In Figure 5 give illustration of how Gumbel Distillation is integrated into MDLM.
20

---

Published as a conference paper at ICLR 2026
Datasets Following Sahoo et al. (2024); Arriola et al. (2025), conduct our experiments two datasets, The One Billion Word Dataset; LMIB (Chelba et al. 2014) and OpenWeeTent OWT (Gokaslan & Cohen, 2019). We use context length of 128 for models trained on LMIB and 1024 for models trained on OWT. For both datasets, we use the GPT-2 tokenizer to keep con- sistent with selected teacher model. For LM1B, sequences a maximum length of 128, For OWT, documents were concatenated and chunked into sequences of 1,024 tokens, with an [eos] token used a separator between documents. Since OWT does not have an official validation split, we created one by holding out the last 100,000 documents.
concatenated and wrapped to
Baseline: Masked Diffusion
Ours: Gumbel Distillation
x
MDLM trinnigg with Gumbel comdition
(ar
Po(ar
X1
&
X3
Ea
X5
Apply Musk
Inject Gumbel
x-
[MASK]
x
[MASK]
x
p"
{2t
x
Apply Mask
Get distillation pairs from AR teacher model
x²
x²
x
x²
x²
x-3
x-
x
g'
E?
g
g
Figure 5: Gumbel Distillation Integrated with MDLM. Left: original MDLM architecture, masked positions are predicted based solely on the unmasked context. Right: Gumbel-conditioned archi- tecture: distillation pairs (E, x) are obtained from an AR teacher, then MDLM is modified to take Gumbel as conditional input, learning to predict masked positions with corresponding Gumbel noise.
Model Architectures and Baselines We use GPT-2-L large as teacher model for MDLM and BD33 LM, both of which augments the diffusion transformer (Peebles & Xic (023) with rotary positional embeddings (Su et al., 2024). All model (including the AR bascline) implementation follows the transformer architecture from Arriol et al. (2025) that uses 12 layers, a hidden dimension 768, and 12 attention heads, which corresponds to 110M parameters (excluding embeddings).
Training All models were trained for I million steps using the AdamW optimizer with batch size of 512, and a constant learning rate of 3 x 10-4 with a 2,500-step linear warmup. This translates to 65B tokens and 73 epochs LMIB, 524B tokens and 60 epochs on OWT. For BD3-LM, follow the baseline by training with maximum context length first for 850K gradient steps (effectively the same MDLM) then fine-tune the target block size for 150K gradient steps. We maintain consistent random seeds to ensure fair comparisons. All experiments are run on H100 GPUs.
Nucleus Sampling For all baselines, we follow their default setup and employ nucleus sampling (Holtzman al. (020)) with p = 0.9. As discussed in many prior works (von Rütte et al. 2025 Wang et al. (022). one could get suspiciously low generative perplexity values as p becomes very small, at the cost of sacrificing diversityyennrroyy We noticed that for models trained with Gumbel Distillation, the diversity of the samples is more robust against low p values than the baseline, which could be attributed to the stochasticity brought by the randomly drawn Gumbel as the conditional input. To keep consistent with the baselines, we still use p 0.9 for Gumbel-conditionel models.
Gumbel-based Categorical Sampling for Diffusion Models Zheng al. (2024) analyzed the numerical issues of Gumbel-based categorical sampling for diffusion models and offered two meth- ods for improvement, including (1) sampling 64-bit Gumbel variables instead of 32-bit to reduce the truncation effect, and (2) using their proposed first-hitting sampler. The BD3-LM baseline adopts both methods, which we choose to follow in experiments.
However, it is worth noting that (1) Zheng et al. (2024) points out the token- boken decoding pro- cess of masked diffusion models by their first- trst-hitting sampler does not suffer from notable numerical
21

---

Published as a conference paper at ICLR 2026
issues under 32-bit precision, which makes it unnecessary to adopt both methods the same time, and (2) the first-hitting sampler effectively limits the masked diffusion model to unmask one token at each step, making its inference cost the same as AR models. Therefore, when obtain samples using fewer NFEs than the generated sequence length, first-hitting is disabled and we fall back to the efficient ancestral sampler from SShoo et al. (2024).
Inference with Calibrated Gumbel Noise During training, the Gumbel noise & is drawn from (0, 1) to fully capture the teacher's sampling variance. At inference, however, we observe that the heavy tails of Gumbel can occasionally produce rare, high-magnitude values that can lead to lower- quality samples. To ensure more stable and reliable generation, we can draw from a calibrated Gumbel distribution by introducing a temperature parameter т E (0, 1] and scaling the guidance as 5 т from the same Gumbel family but with a reduced variance, effectively controlling the influence of extreme values. Empirically, slightly reduced temperature is found to bring substantial improvement in generative perplexity with minor decline in entropy, indicating that guidance from a lower-variance distribution is more consistently reliable. In our experiments, we use 0.85.
C.2 MULTI-TOKEN-PREDICIII
Medusa Prior to Medusa C Cai et al. 2024), various speculative decoding strategies (Leviathan
et al., 2023 Chen et al., 2023) have been proposed to reduce the number of decoding steps for
LLMs, where a smaller draft model is used to generate a token sequence, which is then refined by the original larger model for acceptable continuation. Instead, Medusa proposes using multiple decoding heads on top of the backbone model to expedite inference. These are additional decoding heads appended to the last hidden states of the original model. Specifically, given the original models last hidden states he at position t, K decoding heads are added to ht. The k-th head is used to predict the token in the (++ [++++11 1-tth position (the original language model head is used to predict the (t + 1) 1-th position). A single layer of feed-forward network with residual connection is used for each head. Denote the prediction of the k-th head as Pt
(k) then the definition of the k-thh head is:
Pt softmax (W!*). (SILU(W(M) h.) + h,)) where wh( E RdxV w(()) E RRddd (11)
d is the output dimension of the LLMs last hidden layer and V is the vocabulary size. wr( is initialized identically to the original language model head, and Wr(A), to zero, which aligns the initial prediction of the heads with the original model. The heads are trained in conjunction with the origi- nal backbone model, which can be frozen (Medusa-1) I) or trained together (Medusa-2). At inference, tree- rre-structured attention mechanism is employed to process multiple candidates from each head concurrently. Empirically, Medusa found that five heads are sufficient at most.
Model Architecture For simplicity and consistency with other experiments, we choose GPT- 2-Small as the backbone and follow Medusa-1 1 's setup to freeze the backbone model during training. Since the open-souree code of Medusa mainly supports certain LLMs, we re-implemented the core framework for our controlled-scale experiments on GPT-2-Small. As shown in Figure 6 we select the number of MTP heads as 4. Since each MTP head is designed be a single linear layer, we first normalize and project Gumbel vectors via the same process as described for MDLMs, then we accu- mulate the Gumbel sequence via a small causal transformer and feed its output to the corresponding hidden state slice from the backbone as condition for each head.
Training and Inference We simulate self-distillation setup, adopting the assumption that Open- WebText (Gokaslan & Cohen, 2019) matches GPT-2's output distribution. The MTP prediction heads are trained with batch size of 256 for 20K steps as we observe further training does not improve the performance as much. At inference, we follow Medusa-1's 's setup and adopt typical acceptance as the strategy for speculative decoding, where the proposal is verified against both a hard probability threshold and a soft entropy-dependent threshold. Specifically, given æ1,..., In as context, when evaluating the candidate sequence Fn+11 Bn+k& a candidate is accepted when:
Poriginal (In+k 2 x2 ... 1n+ > min(c, c. exp( (expppH(pppspppgee (Porigsnat(-- æ Z2, In))
(12)
We select e = 0.1 and = 1.0 for our experiments. To evaluate the acceptance rates, we adopt a simple sequential strategy and verify each candidate only when all previous candidates have been
22

---

Published as a conference paper at ICLR 2026
x
Head 4
Head A
Head 2
1::4
g.
Head M
1:2
Base LM
g²
g
g
Paired Gumbels
Context
x²
x
x
Tokens To Predict
Figure 6: Gumbel Distillation integrated with Medusa. The base backbone model is frozen when training the MTP heads. We first obtain distillation pairs (E, a)) from the AR backbone as teacher, then each MTP head is modified to take Gumbel as conditional input.
accepted. We randomly select beginning slices of samples from the validation set of OWT and let MTP heads complete after the starting prompt. Reported results are averaged over 500 samples.
D EVALUATION DETAILS
D.1 ON EVALUATING GENERATIVE QUALITY OF DIFFUSION LANGUAGE MODELS
To quantitatively assess models' performance, we adopt standard language modeling metrics. For unconditional text generation, our primary metric is Generative Perplexity (Gen. PPL), evaluated using a pre-trained GPT-2-Large model. Since GPT2-Large uses a context size of 1024, we follow our baselines and compute Gen. PPL for samples longer than 1024 tokens using a sliding window with stride length of 512. The numbers reported are an average of 300 generated samples.
In addition Gen. PPL. we also follow recent works (Wang et al. 2025 Fathi al. 2025; Shing & Akiba, 2025) report the MAUVE score (L.iu et al. 2021) While Gen PPI. is commonly used metric for measuring fluency, can be an unreliable indicator of overall quality as it often fails to capture crucial aspects like diversity and creativity. A model can achieve lower perplexity score by generating repetitive or overly generic text that is highly probable under the reference model, this does not correlate well with human judgments of quality. In comparison, MAUVE score provides a more comprehensive assessment by comparing the distribution of model-generated text against the distribution of human text, offering balance between generation quality and diversity that cannot be "hacked" by tuning sampling hyperparameters. Following Pillutla et al.) (202)), we compute MAUVE score with 5000 samples from the validation of OWT as the reference data.
D.2 ZERO-SHOT SHOT BENCHMARKS
For downstream performance evaluation. we use the In-eval-harrnn Gao al. 2024 library which allows custom models to run on benchmarks by providing likelihood estimation and text generation APIs. For likelihood-based multiple-choice tasks, we compute per-token likelihood over both context and completion (excluding padding). We adopt one-step evaluation, as this tests pure generative capacity without partial information, providing deterministic low-variance estimates that give more stable and discriminative results. The models perform one forward pass on the input where only answer completion tokens are masked, and the likelihood for each token is extracted from the model's predictions at the masked positions. For BD3-LM, block boundaries are respected by sequentially masking entire aligned blocks. Sequences exceeding model length are right- right-truncated to maintain recent context, similar autoregressive context scrolling.
23

---

Published as a conference paper at ICLR 2026
D.3 LLM-AS-JUDGE
You are an expare text guality anauusarr. Flasse evaluste the grven taxt tarple fron thase cinenzions using a 1-10 scaie xitz cacisal precicion.
1. Clacity: Assuaa huw clear locical the text las Comsides:
Ie the rain muzage/tople clas??
Ideas logioslly from
the next?
Is the vay ta follox end cosprshend?
Kate: Text may be an exceept, don't pasalize incorplate contaxt,
2. Gramaticalsty: Fostatte greeatiel correctnesr; Consider:
Proper senterce stracture and syntax
punstoasion, supitalisasion
Suafect-vara agresset,, tanss consistancy
ibsence of granratioal errors
5. Factuxlity: Assess aceuracy of verifiable infomation; Considar:
Are stated facte ristes, nanes, places, numbers) acourste? information
to varifiable fasts present, acoce bazed an plausibility.
Fozuz internal consiatancy rathez than pariect
4, Styler Evaluste the quility of vriting style and flueney: Consdiert Sentence veriery and thyths
iporopriata vocabulary
Natural flow and reaobility
Lane for the centent
5. Creativity 1 Ascess Ive and engaging qualsties) Consideri
Neiqinal idosa, unique prrspactives, intereoting apgeschee Enasging language
tesssive use of lunguaee, sutaphess, os deacripsions sontent
Arstiance cverly The bexs to be qeaded is as followa:
(taxt)
Figure 7: Prompt used for LLM-based evaluation using Gemini-2.5-pro.
Following previous work (von Rütte et al. 2025). evaluate the quality of the unconditionally generated text samples by using LLM as a judge. Specifically, we employ the Gemini 2.5 pro API (Comanici et al. 2025) rate the samples based on clarity, diversity and fluency on a scale of to 10. We provide the sample text along with instructions for the LLM to first give a justification and then grade each dimension accordingly. The final output from the LLM is restricted a json format for parsing. See [ the prompt template we used to instruct the LLM for the evaluation.
E ADDITIONAL RESULTS
E.1 MAZE NAVIGATION ToY
Here we present more details on the maze navigation toy's setup and results.
As shown in Figure D, we used 5 x 4 map for the maze. Denote the coordinate of the upper left point as (0, O). our start point is set to (3, 1) and target point is set to (3, (). Given a vocabulary of, {<bos>, <eos>, up, down, left, zi right), }. we define simple language as a sequence of actions that form a valid path from start to target in the maze, and set the maximum path length is to 10. We used BFS to generate 2000 valid paths as the training dataset.
For the AR model, its architecture is a transformer with 8 layers, 8 heads and a dimension of 128. We trained it with batch size of 256 and learning rate of 3e-4 for 200 epochs to obtain an "ideal" teacher that almost always generates valid paths. Then, we implemented simplified version of MDLM as student and trained it with/without Gumbel Distillation using the AR model as teacher. The student model is a transformer with 4 layers, 4 heads and dimension of 128, and was also trained with batch size of 256 and learning rate of 3e-4 for 200 epochs. At inference, we generate 100 paths using different number of steps and count the number of successful generations. For AR, NFE is fixed at 10 whereas we try NFF=1, 1,2,48 and 16 for MDLM. The success rates are below:
Table 7: Success rates vs. NFEs for MDLM and MDLM + Gumbel Distillation the maze toy.
NFE
=1
=2
=4
=8
=16
MDLM
$3.0
64.0
69.0
81.0
94.0
MDLM Gumbel Distillation
82.0
86.0
97.0
99.0
98.0
24

---

Published as a conference paper at ICLR 2026
E.2 LIKELIHOOD ESTIMATION
In Table 8 we also report the zero-shot validation perplexities. We evaluate the likelihood of mod- els trained with OWT on Penn Tree Bank (PTB; (Marcus et al. 1993). WikiText (Merity et al.) 2016). LMIB, Lambada (Paperno et al.) 2016), AG News (Zhang et al. 2013) and Scientific Pa- pers (Pubmed and Arxiv subsets) (Cohan et al., 2018). Since the zeroshot datasets have different conventions for sequence segmentation, we wrap sequences to 1024 and do not add cos tokens.
Note that the results here are heavily biased towards models trained with Gumbel Distillation as they are conditioned on the Gumbel vectors corresponding to the ground truth sequences. Since the model has learned mapping from Gumbel noises to coherent AR outputs, it can generate good text samples using random Gumbel at inference, but this does not guarantee the prediction of the exact sequence on the validation set. However, we believe these results demonstrate how well the model has learned to estimate the conditional likelihood when the exact corresponding Gumbel is given.
Table 8: Validation perplexities (L) of models trained on OWT.
PTB
Wikitext
LMIB
Lambada
News
Pubmed
Arxiv
MDLM
92.54
34.52
66.10
48.63
65.32
43.67
37.47
MDLM Gumbel Distillation
35.12
15.57
26.01
15.56
20.03
20.81
16.85
Block Diffusion
96.27
30.82
61.14
49.51
63.09
42.59
39.39
Block Diffusion Gumbel Distillation
37.54
13.86
22.69
18.19
19.78
17.02
E.3 INFILLING EVALUATION
We evaluate whether Gumbel conditioning preserves the versatility of masked diffusion models beyond unconditional generation. Following the setup in our rebuttal, we sample sequences of length 128 from the LMIB validation set, uniformly mask 32 tokens, and use the same MDLM sampler as in the main experiments infill the masked positions. We report GenPPL computed only on the masked tokens using the same AR evaluator as in Table averaged over 300 samples.
Table 9: Infilling on LMIB. Gumbel conditioning improves GenPPL non-AR infilling task, suggesting it does not overfit to AR-style sampling.
Model
GenPPLJ
MDLM
55.1
MDLM + Gumbel Distillation
47.8
E.4 MODEL-SIZE SCALING
We also study how Gumbel Distillation behaves when scaling MDLM. We compare a small model (12 layers, H=768) and a medium model (24 layers, H=1024), where the medium models are only partially trained (40% of tokens seen compared to small models), and the trend persists.
Table 10: Gumbel Distillation improves both MAUVE and GenPPL. for different model scales.
Method
MAUVE
GenPPLJ
MDLM (Small)
0.179
78.74 67.64
MDLM (Small) + Gumbel Distillation
0.264
MDLM (Medium, partial)
74.22 0.6
MDLM (Medium, partial) + Gumbel Distillation
0.253
61.34
F GENERATION SAMPLES
25

---

Published as a conference paper at ICLR 2026
"The two Houses have adepted bipartican appcoach, inviting now rules licensing reieases for rederal agencies. Currently, thay requice the agencies Lo have la-Septh internal zeviss of the riles. Thay agreed zules and dideft atop until Banking and Budges Chaimen ahared the Senate Firance ket thzough Senate Finance BLT 2012, though chere no innediate support for any reform Democrats not united in venting a near-complete return to the roles, which would requize monthe of political pressure to rein requlators. While Republicans Virginia) has seftened his staace. Bo said the House will continue support commities work on a bill to help break the rules but also create complex ovechaul But Cantur said that, I aors of implicitly Leuat, Lhat we will pasa the bills in a bipastisan way with uni Repubtican leadership. s.ill, Denocrala concinue Lo be caul ious about. wha i. would cost for the 2012 rinancial bill pass.
pre-oxisting differmncas, Hajority Cheirman Jim Cantor (R-Mest
Cantors ie to be the tina. one that cleared the Senate next week, The House will return it on the tlooz tor consideration it passes roles, Me. Dehner seentred..ennaorr and Repoblicane eay they rever expressed aupport for much nore oversicht of federal requiators because of the controvarsial bodd-Prank overhaul late 2013. Dancerata had resisted plans for several other Dodd-Frank initiatives to go bayond belsway and create Line for Federal overaight. They feared that Financial regulatora would set up revolving dooe and andless finercsal related regulat could slow data lect lon and satiaffy people.But Senator Dansel Costs (8-0hio) offered flat tax increases to create the height-line zpproach. Now, the idea is are regulatory overtauls, Financial Services Chairman John Thune said Adsitional reporting Benjamin Reevess (endorteet
Edward Snoween: Suean Grigh Kara CORRECTION: Joar. Tones says **Uboa
pills"* on record. Party to 2, Edward Bnowden, BiricaMM Broadeast A tav days aco, Frank Geithner spoke on The Friands Bouz of Havs Bloorberg about decending ground line the right aqainat Psagnatian and seying that the only alternative in she matter requires reading of the Flements Policy of Medics Nerr though, should drew liee inportart essential fasues sard an alternstive recessaryafter an important decision stago for Russia. Vitally Golditkin carlies had outlined set of decisions with nzjor implications. Ruscias actions will risk a layer of canctions and the ire of many Republicane in Congraas. Be varned that oegional actuss are in confliet cla powes and influenew grandly coee dictate stzatagic posture, ecommiu, milllasy, and political determination, Be passed on saying except tranquility with zegards to the nuclear weapona regine, there could be a result of direct, direct disruption of the Ruesian econony, But atter just enough deaate and explanation, it had energeds The treaty most renew and be upheld. Ultinately, the United States nest roly upon our allies to sift through the fects. Rassia made the nucicar comnitmet publicly, on the possibility pessible foint military strike. Hevershaloss, any type of action, through physicz1 and through the enforcement of the treasy, is a priority of the United States, Rochani said. The Ruropean Onion have aloeady indinated Dhey would panish Russia with retallatory messure, Bussia would impose other sanctions, and far China China look to develop an immediste utter supprice oz retallation, considering Chinas economic policy, which applies Putin seeke it. Putin talked like way about potential resauttle ot Saudi Azabia and Iran, The Saucis lvenseives could just show their belligerence toward Russia or encourage the Uniced states to use fozce to quickly squeazo Russia. But cbana has civen the Saudis an excuse to berate then diplonttiaally for slippade, while rolish to suggest that the United states is consurnste engeged senot ions againet Buxsia, That situation cont inues with the Russiar responze. The report that Ruxsia is about to install nuclear miasile shield on U.S. the waves the Twitter pronpting of unnaned Ressian statosnan Androi Lavrov. Othere chimed ronstully by scunding the tollowing bell. Its not grined from you, Auther Vashemi Kopacak stepped in for Putins explanation. That dicreçards what you sent over crose borders with your hardo, what you chose to eat and lssten to strangers while people at or asving intel Leak your enenies. Why they cant you pat. your weapona the eneny and calculate St.No"
wide level, seying that Lhe Beat percelows
Figure 8: Sample from MDLM of length 1024 and T= 1024 steps. The generative perplexity of the sample under GPT-2-Large is 62.36 and its entropy is 5.56.
26

---

Published as a conference paper at ICLR 2026
'The Japan Galaztic Institute has ancounced that it Nill bocome the vorld's ever-growing space lao. The company will ze develeping a comeercial syston used to nove Japan't opace caip into the next toontier. The narket is booning and the central Japanese nacket Iorecasts zevenues of 92.6 billion to $3.8 billion 2013. Japan Galactic the biggest of the world, "n ssid dean Phillips, calef executive of European Salence Systens (ASES), an
big step forward in getting used an Japan, which has the ability to
international research company. **A for the Jepanese government's is success in foreign policy, we're proud that the CMSS partner. We look like the Japan Galactic Institute a world-type seience laberatory. "KKashisci and its partacrs have been developing Japan as a globsl hub for industiies such solar pover and courisn. Starting cities, villages, Lowns, tural aceas and village a quarler Lo a quaeter and a third of tie world's cop. 2010, the Nordie country recurded ammuel GD2 of $1.11 billion and yoow by $14.1 billion faster than originally projessed. dapan's global GDP grew to $14.0 bi billion, $101 million in 2013, and 5130 million, with Asian nations siovly adopting new varieties ot seed for arops and inoigenous seede tor their growing acanomies and economies. That's bacause Capan has managad to use the technology to get a vebsite for online courses and new narketing technolooy, too. As is being developed at Japan. Galactin, the CMAS will be able to use software planforn, directt ly from senxors, chips and comguter aguipeent, to connect the lab to the Internet, so anyone can have to connect the technology to a recwork in order to provo thenseiver The work behind the ecience study will to bit harder. Once Origin Incospersted, the in vitzo technclogy, the cornersione of Japenese research, will be patented. this point in time, the company will have been seaking to tly to and around the Europsan Union. Now it is soon moving snall Buzopean zesearch fism Into the institure'a aystens and hardwaze to make it the lareaat possible narket in the next years. "Me want to co take the load in scientific Lab that extenda ita capabilities all over the world. He want to see isig iiv research in pursuit of our own lives. And if this heppens, it's huge breakchrough
step to somcthing of
early 1990s, Japan building towns,
homen race and it's very exciting opportunity for inrovation, "* seid James Smith, the head of the Europoan Union's mein scientiric reszarch organization, in his remarks l8st Wednerday. First, the World Congress xill launch similar technelogy July and apacoShapOne's fleat public flight in June. But it's the Cirst in jas launch aves in the US. Dne fleat year to Lake US ixperiments Logetliez was announced by a consortiun that had nade head'ines. Allhoagh the Buzopean nacket is already the first to use the technology, not all companiea are doing the same, SABC madical group whose works show the potential to provide the treatment for cancer is moving to use the Earopean Union, government, agenstes and government scientists to set up its research laboratory or CNAS, This is trip up to space, and both exporinents will carry a druq and brain thsted. first stage is that it lets the US taks by plane, The technology also lets the DS to interact with the anvi including in places like the Intecnet ional Space Station, Thas probably leads a second stage is shere the FDA is looking to cind ways to make comparisons and communicate to the pualic about how to deliver drugs. It's way that the ottects on the buran body have beer. tar less neseured. CNAS still needs approval at its patent, Using the technology that could produce therspeutic etticacy in these with paychiateic disorders broad vein of neuroscience, nedicina and data vould violace ssveral of the agency's guidelinas. "*We seen to find that a few years progress on the case would ba slow and hard to if becsuse it radiets these government and the need for higghe quelity, Smith wrote in hia remarks, The U.S. Senstw in Joly has suggested is will not approve any proposala that boost zales.A sinllar neasure in Kay led Beropsans to vota on whether or not drugs should face poor-to-peor trading, The news dars ott signs in 2015 that the European Parliancat passed the Suropoan Conmiscion's e next tinancial year, inhibiting the FDM's s scientitic review ot its potentinl ottect on certain drugo.d pilot for ciniler study expested cost about 62,030 each and is espeated to launch the UK, the and of 2016. slendofooott >Tha Bank of Bogland announced the intereat rates on Dacumbar 7, zecomending a discouns zate of from 28
orbit. Uniike the first spaceflight process, it must be transited and overseen
own bigger push into the robot budy.
Figure 9: Sample from MDLM Gumbel Distillation of length 1024 and T = 1024 steps. The generative perplexity of the sample under GPT-2-Large is 35.19 and its entropy is 5.32.
27

---

Published as a conference paper at ICLR 2026
"Iuo years then signed hin, and be :old his Felenda. Be vas one of the best soccer playera ever. helped take out the England national tean, which la vacant at this atage, Iufa been a long journay, said Rodrigues, who's new gone beyond the borders of the world, going may I love England, "* out to said England, and to have fun, honviooosls hefe very close. T have no éoubt he in a highly respected mele partner my country, "1 said Roiriquez, who siened anbassador. rrsut not wanting that much beat the player, bocause, 1ike a pascior. Hot's going to be here for a lcng time just to go there."* Redrigusz, huge player get top flight Lean the top
like the England Americans and others, has would be Lough, "MBe's been on the sight side, Bodk iguex sald, Hetss England squnes than I nex. The last Linw with Burupwan Lean, Befa been really Lough, whether get the Courth sound, Ne have been corpetitixe, the capital at the Bouthwest region, ""Denophia peillen, who waz a tirs:-round pick, one best playere the world, who has played seavily hopes of winning the national championship in. the past two years, even as part of a loan by the United States to Colerado "MBu: he's not an idoi. don't have that many other guya in the soccez world with the highest-lovel play Chicago but ha knows his side, an bei Tlen swid. said in statenent after Rnglansfx World Cop titler "Mile has helged a tear that signed a numbers like season in the Nat ioral Songer leegue. That's excitinng football. nenesn care"" wonzn's soccer playars is in the United States," "* International Divector Brien K, Senjelstra told. The JIA, "rI an happy to win Morld Cup and have person able represent the women'e e League in England and with women'' Soceer at the top of ny heart, and inspize ae to defand my country, Uaited Secoas Association ion currently selected women's fivst tean compacition xinners, and wishas competition organizera prepare the best young players beyond the age of Meanwhhh,,,,, the scadeny is in Australia and begins in New Zealand. It's played. "fTe's a challlngee for
at Kanchestor City than anymere in the European Lesgue of ""everyone
new biggest biggest tournement the wonen tean in Americe tans sez women's Seceer playing in the Morld cup tinals," tn pellion thid the Eaçlard Americans had the players they signed. lot of then have seamed to may that they sonts stand these clabs, and this time thay are playera Lhenaelows, M"Taa, would be cunowen, I Lhink you luck at some Leans, one year and 16, soow of then hoping Lo get then four yeass," she said. Bo Casy the England players appear have nationalistie In the prelininary atage, there are no signs that the two national players left the team. They have no aay on who is now, and the tean ie still with the England players. I know there's s yet another team play for and that's really confidence, "Nevaad spokeswenan Lacra White later fasued stazement to the Federation doclaring this "ta new caeet betwean the two sides, which will play in Leaden, the kind of spet where the top tvo in contencion by the federations Mhito spoke with the federat ian at 9:40 patti and they said the players didn't be ready to take part before, a ststement Chat indisales the way the woren's Iedera ion intends
Whide ssid the norning before the Lournerent said he could not be with that team, and that he wed prepared to make hinsel: unavailable. Re has been contacted by the United States Erbaasy tor corment the price had shade said she is *"yhere the players are tor whi "* but added that she will continue to be connected to the team <lensottext kelke Carrington (SMS chiof) nade his case to Clazke Hall (S8P), who heads Royal North-C. Wales for the west west Kales University Hoapital, saying the tvo aze "*vary closs asseciates on the NESTY as a result of phone call. myy rirst react fon is I'n not happy at the noment, " seid Royal's a deputy chief,, nuree who was traveling lest week in his elinie to neet Hall and hospital, .on Toestay norning, spokesman for the chief "fcolled into similar situstion with our partners, who wore working with the NHS, whore they approsched our otticers the camo convereation with the epokesman, ** he North-wstt Wales chief direstor or the aospital posted otticial webeite, saying: "*Ie taere are no good answered on the NES's mobile phone then
Figure 10: Sample from BD3-LM of length 1024 and T == 1024 steps. The generative perplexity of the sample under GPT-2-Large is 37.7 and its entropy is 5.21.
28

---

Published as a conference paper at ICLR 2026
"How nany times in ny lifa have been vonan? Irve alwaya Zolt like vonan in everyching Irve done to be proud of, just being great pezsoc, so I am zoceasidaring. My gifter vants to be raned the "Santa" and we will have to see her on Chrisgnes Day-T have received nany gifta over the years, but an a soppin! and beno in body, I. feel like I an wonan! All I want to throw away this thing is to do happy and productive job like Soneshing has boen happering a lot lately, and it's a very trustrating time today, The gitting is just part of series of eveats in the past year, The location is orlando, and a relatively long way away crom home, The photos are the ones I want to give to the communtty, not the my aendera not getting. Buz I know the I' sending donating to asaisting the fans in. these gits. My gifter will be thanking ay Santa and giving na hags! If are you, you are in ny heast. You at my Funeral, This is not giving denn about the people that I put to then, Teta beaut but the pisture sen't about my aeart? If you are ne, it would moze than what do.And itts not something that a I'm. show in a vidco, I just see some of the nows pleces that have boon pested and on. I want for people
doing, don't feel like being a hero,
literally
poople do. If
to alow their support, for (ny gifleet and (hey will tell de. I' not doing what do.I have Lo givw away the gift, I'n not "aatisu or arcial at all, and I' not Metaa creator"" of anything leave ny okn name, and don't want to wonan to get any aupport. want to get name on the internet for this thing. You aren't getting the nane out at people deserve be given name. aith that sald,
at least one of the people in this incividual's s cirele have been kind through me. I have never thought while doing zounds with the cifter I rould give avay. jest laughed. thought gifs exchanges would nake no reel better. oz naybe they would have. But I'n not leaving it up to the gifter on it. It's just sccept
friend was ny nother, but said that she likes helping people, and ahe said vanta to get away from things thas have been done. Rowever, she didn't i make any plan to gift hin, He was completely avare of and just thought and vent with it. vant things like this to get away from all of ny gite, I think I have only ore thing to do. Both or those things are bad and I cannot gain access then. These to gifters and I take it to the audionce to show ne that I can do what I could do. I want to ba abie to get sonsthing to give to then, and give it to othera to give away. I' just then goine and then capping into the pressure they are usnelly getting it exay aryway from all of then, I don't t know how all did, kwan't thinking about it, went my way, I have a very happy time. Iets besscally all about ne. was going to give it on to somcone who was one of the ones aster and I dida't ever have give then the support because tols right for ne, as I went to what I alceady did and I just never got the tool that I had. Ifa so happy with it. spent couple of nonata gecting myself. I have not really defired myself as a real artist for being e, So that is a good feeling for all me. The people do not want ne ges through the process loving as artist, not wanting to me be the person I can be. I need to reach out my fans, to xnow shat the people have zqainst na. But I go through this. need to show my support for then end gifting. The fact that those are all bencrito the people have, of the commuity itswls. That mean I havw lot of Liny swid help projects and project that are uneocme..
bad things and when read about then, tind then justitsed. I then give the gict
that is scogping then from helging ne, This isn't simply gift. Wher Iive done it,
this opportunity for people that was going to consider and
going. It didn't
that power bask and back to the world to write about
Figure 11: Sample from BD3-LM + Gumbel Distillation of length 1024 and T == 1024 steps. The generative perplexity of the sample under GPT-2-Large is 27.2 and its entropy is 5.
29

---

Published as a conference paper at ICLR 2026
"The numbers game to ne, the most inportant takcoway is the modia wants to sensitionalize the story, it malles. The seal stosy here is what have Lo expert
average cepoztes, have expect that nost chings you say en the aix will blow up and overshadov the fact that you are segular citizen and could sell have eaid aryshing you want people it they catch on, But tais the case. The average anchor on ON has otanged the story on a regular basis, If you dont Lister to the medie, you will lose,"n "n the name being unbiased, the meinet media wil". rerain relatively y unknown group. I think they are having an affect on the perception of the public. In my view, everyone on the aiz is opar-eeprcsented on netwerk tsiovision. If they are not correct, they can win over viewers who may not aecesseslly be paying allention. À good example of this would be when CNN featured a big acory about the Nocth Korea nucleas Lest and the rallnge went down. Pox some season, the covezage led Lu feaes vievers CHN, and that cetinitely what Iapponed. I taink pieces, The result is that wher they present a big story, and when they get the impression they are going vin, people night not
U.S. media more adept maxing propagends
CMI, exsect that my expactation that navs nedia will nore viswers on CMS and possibly even win nore vievars for cable networks (not to nention nexscasta and prine time shavs). But it they dont, I think wave of viawers night be swayad the accond tier. An thThe outcore for ne is unglear. CMN has big story, and I do expecs is to draw in a large percentage of vievers, whill the anchors loxing vievers. This could
entertaining topic that najor aetworks want to have on sir, Maybe they vant to vin over vicwers for those networks? The next stop I see is for then to stop showing old news it they dont think the American public is paying actention.You can follow @Baaro on Twitter, Comment you step insido The Collectors Mini towez on the grounds Dianeylanda Bellywood atudioa. As the world and park nstures, peoplee are returning visit Bianeyland for the firet time Tene.Corsttune rootior the 10, 300 square foot ster-studied attraction for began in 2001, hianey deve loped the idea to inclade the area as hiseeyl and Nenorial Hall in the park. August 201 3, The Dol lectors Mini Tower was revealed to the public during a virit, During the virit, questo SEN the incredible view that was incleded inside the tower and loarned about how nuch cost to construct this unique ietraction.tcoordie to Disney, fiveraooryy building is 300 squaze feel and featurea mere than 1, 500 zouns, with wirdows powslooking the street level of the exterior the inside,The size of the atzuctuse nakes it unique expesience for touriets. The attraction itselt takes guesta to special toyer Lined with board gane and ride setups. The Tower was constmucted anusement park property. //addtttxxt |>Both the Canada Goose and the Canadian flag may soon have now colours. rederal geverneet is charging its policy that says any bill to chance Canada's a flaq must be approved by parliament, making the Canada-U.s. trilatcral trada agrcenent less of reality. An AnLast nonch, Pinance Minister Bill Morneau isxoed a menorandum to all government stristers liiiini what he sald would be changea Lo the copyright and Leadenerks laxs 10 the Canada U.S. Lrade agoeement becamen Uhe Lwo countries is not appreved by Parlsament Under the nemorandem, the gevernment is proposing to replace the country's Canadian halt with an intergovermmentll and unionized version or Canada's ting, the tlng the Donfedoration. The new agreement betweon Canada and the U.S. will add a now label to the Canadian #laq called the Treasy. It narks change from the governnent's a Conservative geverrment of prine minister Staphan Harper's a to a Liberal governmen: of prine ninister Jastin Trudcau. is estinated that 12 millicn Canadians sould lose their currens U.S.-ce. Canadian helf, Tt alnoss impossible to pick colour withous buying one, Patrick Sinon, retired Ostava journel iat. To ges your product into the C Treaty, the country most submit its now government agreement, called the Treaty, to the U.S. Congress and ask the acministration to pacs it.When the C Treaty is appreved, it will be trade limeralization, which will start with sixemonth noticw peziod from October 1, 2018, through October 31, 2019. That means the new treaty will be in effect for three rccordngg to the Finance Ministar'
ftdot (Dissneyladd will have you believe the first Line
cost of \560 million includee several areas of state owned
Sample from AR of length 1024 and T= 1024 steps. The generative perplexity of the Figure 12:
sample under GPT-2-Larre is 15.7 and its entropy is 5.29.
30


Reviews:
[{"review_id": "ZUxo0b0L8T", "reviewer": "Reviewer_7yqW", "summary": "This paper addresses a critical trade-off in language model decoding: autoregressive (AR) models achieve high generation quality by modeling sequential token dependencies via chain-rule factorization but suffer from slow, token-by-token inference; parallel (non-autoregressive) decoders (e.g., MDLM, BD3-LM, Medusa) accelerate inference by generating multiple tokens simultaneously but struggle to capture the complex joint distribution of token sequences, leading to degraded coherence, repetition, or grammatical errors.\n\nTo bridge this gap, the authors propose Gumbel Distillation, a model-agnostic knowledge distillation framework that enables parallel decoders to learn the joint distribution from a high-performance AR \"teacher\" model. The core insight leverages the Gumbel-Max trick—a re-parameterization tool for categorical sampling—to establish a deterministic mapping between latent Gumbel noise vectors and the teacher’s output token sequences.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper directly addresses the long-standing trade-off in language model decoding: parallel decoders sacrifice generation quality (due to poor joint token distribution modeling) for speed, while autoregressive (AR) models excel at quality but are slow. \n2. The introduced knowledge distillation mechanism to transfer the AR teacher’s sequential dependency knowledge to parallel students. This focus on \"fixing the joint distribution defect\" aligns with the most critical unmet need in parallel decoding research, making the work problem-driven and practically relevant.\n3. The paper’s experimental design provides controllable evidence for the effectiveness of Gumbel Distillation.", "weaknesses": "1. A notable omission is the lack of targeted discussion on the classical \"AR-supervised NAR distillation\" baseline, a well-established method in non-autoregressive decoding where NAR models are trained to mimic the sampled outputs of AR teachers via cross-entropy (CE) loss or sequence-level losses. This gap may lead readers to question whether the authors have overlooked a foundational approach in the field.\n2. While the Gumbel-Max trick and knowledge distillation are individually well-established in NLP and CV, the paper does not clearly articulate why their integration for transferring AR teachers’ joint token distributions to parallel decoders is non-trivial.\n3. The paper’s framing of results overpromises, while failing to contextualize the persistent quality gap between AR teachers and Gumbel-enhanced parallel models. The paper uses language like \"bridge this gap\" and \"resolve the quality-efficiency trade-off,\" but the remaining performance difference between AR (e.g., GPT-2-Large) and optimized parallel models (e.g., Gumbel-MDLM) is still large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ip5ETQ8dnj", "reviewer": "Reviewer_pK43", "summary": "This paper proposes Gumbel Distillation, a knowledge distillation framework that improves the learning of parallel decoders such as Masked Diffusion Language Models (MDLM) and Multi-Token Prediction (MTP) models. The key idea is to convert the stochastic sampling process of an autoregressive (AR) teacher into a deterministic mapping via the Gumbel-Max trick, allowing the student model to condition on the teacher’s latent Gumbel noise. This effectively transforms the hard problem of matching a complex joint token distribution into a simpler supervised learning problem. The framework is architecture-agnostic and can be plugged into various diffusion or parallel decoding models.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper correctly identifies that the main challenge of parallel decoding lies in learning the joint token dependencies. By introducing Gumbel noise as an explicit conditioning variable, the proposed approach offers a conceptually clean way to transfer dependency structure from AR to parallel models.\n2. The idea of externalizing stochasticity to simplify the learning problem is an important insight likely to inspire future work in distillation and non-AR training.\n3. The method is designed to integrate with different architectures (MDLM, BD3-LM, Medusa) without architectural redesign, demonstrating good generality.", "weaknesses": "1. Limited and marginal empirical gains. In particular, Figure 3 shows that the NFE–quality trade-off is not clearly improved—on one dataset the curve even slightly underperforms the baseline, and on the others the advantages are only marginal. The improvements on reasoning and QA benchmarks (e.g., BoolQ, ARC) are also small.\n2. The paper does not compare with few-step DLM acceleration baselines such as APD (Adaptive Parallel Decoding, arXiv:2506.00413).\n3. Although the approach simplifies distribution matching conceptually, the student still faces a harder supervised task: mapping high-dimensional Gumbel noise vectors to text sequences. As the vocabulary grows, the conditioning dimension scales linearly, and the paper acknowledges this as a limitation (§6). The work could benefit from more discussion or experiments on how this affects optimization stability and convergence.\n4. While distillation transfers AR dependencies, it is unclear whether the Gumbel-conditioned student retains the versatility of standard masked diffusion models — e.g., controllability, flexible conditional generation. Conditioning on a fixed “blueprint” may overfit to AR-like sampling behavior and reduce adaptability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0vKpx4nnFL", "reviewer": "Reviewer_NcsQ", "summary": "This paper proposes a method for distilling the knowledge of an autoregressive model into a parallel/multi-token model via gumbel distillation. This key idea allows for a deterministic supervised setting for the parallel student, and effectively reframes the parallel learning task. Through various experiments, the authors demonstrate how gumbel distillation improves the generation quality of parallel decoding language models, closing the gap with the slower but more high-quality autoregressive models.\n\nThe authors validate their approach by integrating it into masked diffusion language models and multi-token prediction architectures. Quantitative results demonstrate significant and consistent gain over the original methods, on datasets such as OpenWebText and LM1B. This, along with a strong increase in the acceptance rate for the MTP setting, clearly indicates the benefits of the proposed approach.", "questions": "What are your intuitions regarding the scalability of this approach? In regards to model parameters, context length, dataset size etc?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written, and it is clear that the authors have put a lot of time and care into the presentation. This extends to figures, tables and the appendix as well. \n\nThe method is (to at least my understanding) sound, and the explanation is pedagogical with consistent notation. The empirical results are persuasive in that they are both consistent and strong. The tasks selected for evaluation seem relevant, and give a fairly comprehensive overview of the expected gains from the gumbel distillation approach.", "weaknesses": "The authors currently don’t demonstrate any results or arguments for the scalability of this approach. Considering that a major component of how well LLM works is their ability to scale, such an addition would further strengthen this paper. The only part relevant to this seems to be the brief discussion regarding the scalability of vocabulary size. Personally, I’d prefer to see an experiment where the number of model parameters are scaled in the main paper, rather than the existing ablation studies. \n\nLess an issue and more a suggestion of structure. The current layout flow of: 2. Background, 3. Method, 4. Related Work, seems a bit unconventional? As a reader I found it a bit jarring to suddenly be exposed to a related work section after having just read the method. I would suggest considering moving the related work section before the method section or to the end of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Rv59cuA1c", "reviewer": "Reviewer_B2R6", "summary": "The paper introduces a method called Gumbel Distillation that turns the teacher model’s sequential sampling process into a deterministic \"blueprint\" of noise values which a parallel student model can condition on, enabling high-quality parallel decoding. \n\nThe authors show how to extract these noise \"blueprint\" from teacher samples or recover them from existing text using a parallel posterior sampling procedure, and they explain how to inject the resulting signal into masked or multi-head parallel decoders so the student learns to reproduce the teacher’s joint decisions. \n\nEmpirically, this approach improves generation quality compared to prior parallel decoding methods across multiple datasets and model architectures, with gains on metrics that capture both distributional similarity and sample fluency, and with ablations that demonstrate the importance of the Gumbel formulation and the posterior extraction step. \n\nThe paper also provides practical algorithmic details and theoretical justification for the posterior extraction, and it discusses limitations and directions for scaling the technique to larger vocabularies and more structured noise representations.", "questions": "1. The authors acknowledge in the Limitation that computational complexity increases with vocabulary size, but current models often use very large vocabulary. For example, the commonly used Qwen is nearly 152K. I'd like to know how many times more computational effort is theoretically required compared to GPT-2?\n2. I'm not sure why Head 0 is also trained in Table 3. I understand that Head 0 is essentially the LM head, and Medusa doesn't modify it, otherwise it wouldn't be lossless. Is this because gumbels must be input continuously? Are there any experimental results demonstrating true losslessness?\n3. Comparing Table 4 with Table 1, I find that Sequential Gumbel Extraction's MAUVE score is better than the baseline, while PPL is worse. It seems that the introduction of gumbels only improves the Parallel model, but the two are theoretically equivalent. What are the specific implementation details of Parallel and Sequential? Does Sequential use a Casual Mask for multiple forward passes, while Parallel performs a single forward pass without a mask? If yes, are the Logits very different between the two? I feel more explanation is needed here.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper raises a good question: improving the accuracy (MTP) and quality (DLM) of parallel generation is a meaningful endeavor that will help improve the inference speed of autoregressive models.\n2. The paper is clearly written and very detailed, presenting comprehensive methods, experimental details, and theoretical proofs, along with specific case studies.\n3. The paper proposes a simple, commonly used, yet effective method that improves generation quality in DLM and increases acceptance rate during MTP.", "weaknesses": "1. I think the experiments are still not solid enough. You should compare them with other existing methods, such as other distillation methods. While it's good to see improvements when adding a new method to the original model, you need to compare it with similar methods to see if it outperforms them.\n2. I'm not sure how the green gains in Table 3 are calculated. The difference between the two doesn't equal the green value. I'd also like to know the average generation length, as the current improvement in acceptance rate is minimal. It would be even better if you could provide the variance over 3-5 runs.\n3. I think MTP has greater potential, as few people would retrain pre-training and DLM just to add a new method. Therefore, I think you could conduct several similar experiments with related methods to make your results more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)