Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
Authors: Marianne Arriola
Yair Schiff
Hao Phung
Aaron Gokaslan
Volodymyr Kuleshov
Venue: NeurIPS 2025 poster ()
Keywords: Text Diffusion
Diffusion Models
Language Models
Generative Models
Primary area: deep_learning

Abstract: Discrete diffusion models enable parallel token sampling for faster inference than autoregressive approaches. However, prior diffusion models use a decoder-only architecture, which requires sampling algorithms that invoke the full network at every denoising step and incur high computational cost. Our key insight is that discrete diffusion models perform two types of computation: 1) representing clean tokens and 2) denoising corrupted tokens, which enables us to use separate modules for each task. We propose an encoder-decoder architecture to accelerate discrete diffusion inference, which relies on an encoder to represent clean tokens and a lightweight decoder to iteratively refine a noised sequence. We also show that this architecture enables faster training of block diffusion models, which partition sequences into blocks for better quality and are commonly used in diffusion language model inference. We introduce a framework for **E**fficient **E**ncoder-**D**ecoder **D**iffusion (E2D2), consisting of an architecture with specialized training and sampling algorithms, and we show that E2D2 achieves superior trade-offs between generation quality and inference throughput on summarization, translation, and mathematical reasoning tasks.

Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
Marianne Arriola'
Yair Schiff
Hao Phung
Aaron Gokaslan
Volodymyr Kuleshov
Department Computer Science, Cornell University
aarriola,yairschii hao] cornell. edu, {akg87, kuleshov] kuleshoov@corneell. (ww.oeleoooeeccccooooooc..c...ee
Equal contribution; corresponding authors
Abstract
Discrete diffusion models enable parallel token sampling for faster inference than autoregressive approaches. However, prior diffusion models use a decoder-only architecture, which requires sampling algorithms that invoke the full network at every denoising step and incur high computational cost. Our key insight is that discrete diffusion models perform two types of computation: 1) representing clean tokens and denoising corrupted tokens, which enables us use separate modules for each task. We propose an encoder-decoder architecture to accelerate discrete diffusion inference, which relies an encoder to represent clean tokens and a lightweight decoder to iteratively refine noised sequence. We also show that this architecture enables faster training of block diffusion models, which partition sequences into blocks for better quality and are commonly used in diffusion lan- guage model inference. We introduce framework for Efficient Diffusion (E2D2), consisting of an architecture with specialized training and sam- pling algorithms,, and we show that F2D2 achieves superior trade-offs between generation quality and inference throughput mathematical reasoning tasks. We provide the code'. model weights, and blog post the project page: https: //mm amaarolrlaa.......a...a. com/ com/e2d2.
-eecoder
summarization, translation, and
Introduction
Discrete diffusion models [4]. originally proposed extension of Gaussian diffusion [28, 59], have steadily improved in quality on tasks, such as language modeling [24, 40, 55), music generation [62], and biological sequence design [56, 68]. Recent successes include diffusion models the 7-8B parameter scale, which outperform or match comparably sized autoregressive (AR) models [46, 78].
To generate samples, diffusion models iteratively refine sequence consisting of both clean and corrupted tokens. We observe that this process consists of: 1) producing useful representations of clean tokens and 2) using these representations as context for denoising corrupted tokens. However, prior diffusion language models jointly perform both tasks within the same decoder-only architecture. As result, these models must invoke the full network every denoising step, incurring high computational cost.
In this work, propose encoder-decoder transformer architecture separate the computation for representing clean tokens and denoising corrupted tokens. In particular, we use an encoder to represent clean tokens and a lightweight decoder to iteratively refine noised sequence conditioned on the encoder's 's representation. This enables faster inference, as we call the lightweight decoder multiple times to iteratively denoise tokens and invoke the encoder only periodically update its
Code: http:////gichabb..ccc/lleeeeeeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

take
add
[ASKK]
Encoder representation
Faster generation
tokene without invoking encoder
Decoding stops
compared to decoder- only AR and diffusion
Small Decoder denoises tokens while attending encoder representations
Large Encoder processes clean tokens
Extends Pareto
frontier for quality- throughput trade-off
More efficient
training than decoder- only block diffusion
What
[MASK]
add
[MASK]
Prompt
Noisy input block
(+ accumulated samples)
Figure 1: Efficient Encoder-Decoder Diffusion (E2D2) enables faster generation than decoder-only architectures. accelerate inference by using lightweight decoder to iteratively denoise for fixed number of sampling steps, without invoking the encoder. The encoder processes the newly generated tokens periodically to update its representations.
representations. Our novel framework for Efficient Encoder- of encoder-decoder transformer architecture complemented with efficient training and sampling algorithms that enable both faster inference and KV caching support.
Encoder-Decoder Decoder Diffusion (E2D2) consists
In addition to faster inference, our proposed encoder-decoder architecture enables faster training of block diffusion models [1], which partition sequences into blocks to improve generation quality and support KV caching. Block diffusion is widely used for generating with large diffusion language models, even those trained with standard full-sequence diffusion, such as LLaDA [46], Seed Diffusion [61], MM [77]. However, decoder-only block diffusion incurs higher training costs, with forward passes that are 2 more expensive than standard diffusion, as both the full clean and noised sequences must be processed in every transformer layer [1]. We derive efficient training algorithm for encoder- decoder block diffusion that uses the encoder to process the clean sequence and the decoder to process the noised sequence, which halves training costs compared
decoder-only model of equal size.
To demonstrate the efficacy of our encoder-decoder diffusion framework, we train domain-spedn diffusion models for summarization, translation, and mathematical reasoning, and show that E2D2 outperforms decoder-only diffusion models generation speed and quality.
Contributions In summary, we make the following contributions: 1) We propose an encoder-decoder architecture for discrete diffusion modeling: the encoder represents clean tokens and the decoder denoises corrupted tokens. 2) We derive efficient diffusion sampling algorithms that accelerate inference by denoising tokens using lightweight decoder. We also derive an efficient training algorithm for block diffusion modeling using our architecture, which halves training costs compared to decoder-only architecture. 3) We demonstrate E2D2's effectiveness by training task-specific models on summarization, translation, and mathematical reasoning. We also map the Pareto frontier of the quality-throughuu trade-off by varying the decoder size and demonstrate that E2D2 achieves state-of-the-art performance compared to decoder-only discrete diffusion models.
Background
2.1 Discrete Diffusion Models
Diffusion generative models [28, 59, 60] train denoising network po remove noise from latent variables 2,, for E (0, obtained from a pre-defined corruption process q. This forward process transforms data x adding noise increasing in eventually reaching a limiting distribution z

---

Discrete denoising diffusion probabilistic models (D3PM [4]) take inspiration from diffusion and define corruption processes over discrete one-hot vectors 2,. A recent successful class of discrete diffusion models focuses on masking (absorbing state) diffusion [47, 55, 58), where the limiting distribution corresponds to 2 = m, one-hot representation of a special [MASK] token. In this framework, the corruption process is defined via interpolation between clean data x and the prior m: (Zr x) Cat (zr 0;x + (1 - at) )m), where Cat(:i-) represents a categorical distribution and ar = (t)) E [0, 1] is noise schedule decreasing in t. For given corruption process of this form, the true posterior q(z, Z;, x) is known, where 2, represents less noisy latents for <
The optimal diffusion model pe will exactly recover this posterior. However, during inference, we do not have access to x. A common parameterization defines pe as (z( q(z, | x x²(2)) where the denoising model x, (z,) predicts clean tokens x given noisy latents Z,,
To model sequences of I. tokens x==2, the corruption process q is applied independently to each token in a sequence and the denoising network is assumed to factorize independently as well. This framework can easily be extended to accommodate conditional generation by conditioning on some prompt x I:P in both the forward g( (zz)"")
1:L x1: x and reverse pzz zi L.LL x x)) processes.
2.2 Block Diffusion Language Modeling
Block diffuusion language models (BD3LM [1]) improve sample quality of standard discrete diffusion and enable efficient KV caching by introducing a parameterization that interpolates between AR and fully-parallel diffusion. In block diffuuion, blocks of tokens are modeled autoregressively with diffusion applied within each block. Formally, tokens in x 1: are chunked into B blocks each of length [1], for each block b E [1, B), the tokens x (1- S:...
S: B = L.S.. Adopting the shorthand from
are denoted as x" and all preceding tokens x xl:(b-1).5 1).8 are denoted as x x<h The likelihood of this
model is defined as log (x) = Σx= log pe (x xxb),, where each pe xx²) is modeled using discrete diffusion. In the reverse process within each block po (z)) 2,, x<"),, we account for already decoded blocks: X9 = xo(2"; x )). As with other diffusion language models, BD3LM uses a decoder-only architecture to parameterize pa. See Appendix A for an extended overview.
Models that are trained with standard full-sequence diffusion can apply block decoding inference by unmasking tokens in blocks left-to-right, such as LLaDA [46]. Seed Diffusion [61], MMaDA [77], and others [26, 81, 82]. However, since these models are trained with bidirectional context, they require bidirectional attention over the full sequence, which restricts KV caching and requires approximate caching methods [30, 72, 73]. Training with the block diffusion parameterization described above can further improve sample quality [1, 2, 19].
Naively training block diffusion model requires a separate function evaluation to predict each block, since the clean context x x is different for each block. To enable parallel training along the sequence dimension, [1] concatenate the full clean sequence x1:B with noisy sequence z,:BB. While enabling parallel training, this is difficult to scale since the entire model needs to process 2L tokens at every layer, which doubles training cost relative to previous masked diffusion models.
3 Efficient Encoder-Decoder Diffusion
Decoder-only transformers [67] are inefficient when used to parameterize po in language modeling since each inference step requires a full forward pass, motivating architectures that avoid this bottleneck. Our key insight is that discrete diffusion models perform two types of computation: (1) representing clean tokens and (2) denoising masked tokens, which suggests that we can use separate modules for each task.
We explore an encoder-decoder architecture, commonly used in sequence-to-sequeee [54, 63, 67]. to separate computation used for representing clean tokens and for denoising. Specifically, our probabilistic model py defined via the denoising network Xg. For this X9. we define 1) an encoder that extracts features from clean tokens, and 2) a decoder that iteratively denoises sequence of noisy tokens conditioned on these features. Crucially, the decoder can be evaluated multiple times to denoise sequence of tokens, without needing to invoke the encoder. After predetermined number of decoder function evaluations, we can then pass newly generated tokens to the encoder, which adds them to its running representation. Unlike existing decoder-only models, our approach
modeling
3

---

allows parameters and compute to be reallocated between the encoder and decoder (e.g.. by using small decoder and a larger encoder), amortizing the encoder's cost over multiple lightweight decoding steps. We call this approach Efficient Sacoder-Deecoee Diffusion (E2D2), consisting of our proposed encoder-decoder architecture complemented with efficient training and sampling algorithms.
We note that E2D2 can be applied to any discrete diffusion parameterization, c.g., both the standard masked diffusion formulation presented in Section 2.1 and the block-wise factorization from Section 2.2. However, in the figures and experimental results, we focus on the block diffusion parameterization for the following reasons: 1) block diffusion attains superior language modeling performance compared to full-sequence masked diffusion [1], 2) block diffusion enables key-value (KV) caching which significantly accelerates inference [[], and 3) even recent state-of of-the--rtt large diffusion language models which are trained with the full- full-sequence masked diffusion parameterization (e.g.. LLLDA [46]) rely on block autoregressive decoding at inference due to its improved performance.
3.1 Architecture
Encoder We parameterize the encoder with a transformer model and denote the number of layers as NEae- The input to the encoder is sequence of clean tokens comprised of both prompt tokens and previously decoded tokens, which we denote as Xr.Fne. The encoder outputs features h, = ANCODER(Xx,EEd) We explore different choices of encoder feature representations in Section 3.2.
Decoder As with the encoder, we use transformer model for the decoder and denote the number of layers as Dee. The decoder's 's imput, Dr.e corresponds to the portion of the entire sequence z!: that is "actively being denoised." The decoder outputs logits that are conditioned on the encoder output via cross-attention Xlugi DECODER Der, ht), where the positions of Xlugi correspond to Dr,D
3.2 Design Decisions
Below, we detail an efficient implementation of the decoder's attention module using kernel fusion. We also explore two ways of connecting the encoder outputs to the decoder: 1) a "last hidden state" version, where the last encoder hidden representation is used input to the decoder (see left-hand side of Figure 5) and 2) a "shared KV cache" version, where the KV cache is shared between corresponding layers (see right-hand side of Figure 5).
Fused Attention Kernel Previous encoder-decoder implementations, e.g., T5 [54], apply self- and -otes-atttntoonn as separate decoder operations. In contrast, we perform single fused attention call that jointly attends to both the encoder outputs and the decoder's own hidden states. While this forces the decoder to distribute its attention between the encoder representation h, and its own input 2, De, enables a fused attention kernel that reduces memory access and kernel launch overhead.
Last Hidden State The "last hidden state" version follows T5-style encoder-decoder models [54] where we provide the final encoder layer's output features as input to each layer of the decoder. Formally, let g,, f, be the output of the 5th layer of the encoder and decoder, respectively, with fy = EMBEDDING (Z, Dece. Then we define the encoder output as h, = gNe.. Before each decoder layer, we concatenate this representation to the decoder's 's input token embeddings. Thus, the input to each decoder layer h, @ f,-1, where denotes concatenation along the sequence dimension.
Shared KV Cache The "shared KV cache" variant conditions the decoder on the encoder's intermedi- ate representations by reusing the encoder's 's KV's from each layer. Unlike the "last hidden state" design where every decoder layer attends to the same h,, the 7th decoder layer uses the KVs from some jth encoder layer, where the pairing of and j is user-specified. This design suits fine-tuning decoder-only models as encoder-decoders. as reusing the encoder's 's cached KVs keeps the decoder's cross- tross-attention inputs aligned with those expected by the base model and thus improves fine-tuning stability. We define the encoder output as a list of KVs computed per layer h, = [K Boe [L]. K Eoo [Nee]]l where K Emc [j] the KV cache of the 7th encoder layer. Let KV Dec []] be the KV's computed the
7th decoder layer using the output from the previous decoder layer. Then, the keys and values used for
the fused cross- and self self-attention in the decoder are formed via the concatenation h₂ $ KV Dee
3.3 Sampling Algorithm
F2DD2 models admit faster inference by decoupling encoder and decoder calls. Whereas prior decoder-only language models require calling the full transformer in every generation step, E2D2

---

calls a lightweight decoder multiple times throughout denoising, followed by one call to a larger encoder. Algorithm I, we present the sampling procedure for E2D2 applied the block diffusion parameterization, which generates block-by-block while enabling efficient KV caching. We begin by embedding the context or prompt of tokens through the encoder to produce encoder features h h. and we pre-fill fill the encoder KV cache. We generate new block by iteratively denoising a block of masked tokens conditioned on h invoking only the decoder.
We adjust the encoder's signature to accept the KV cache: ENCODER b KV where KV<b represents the KV cache for all blocks preceding b. We also adjust the output of the encoder so that it returns the updated KV cache for the newest block KV*. For the "last hidden state" version of E2D2, which computes separate Vss for the encoder and decoder, we also accumulate KV cache for the decoder (not currently depicted in Algorithm 1).
Algorithm 1 Saecode--eecoder Block Diffusion Sampling with KV Caching
Require: Prompt xI.P. # blocks B, of diffusion steps T, ENCODER, DECODER, algorithm SAMPLE I: I: KV <- 0,0
> Output, KV cache
2: KV <- ENCODER(x" 1:P .0)
> Encode prompt. collect encoder KV cache
3: for b= = to B do
4: z! m"
Initialize noised block
5: for 1tt=1, 1, TH
do
6:
2: +- (AAPLE(DECDD))
Denoise block: Apply small decoder only
7:
end for
8:
X + x= Φे
b Accumulate output Encode new block, compute encoder KVs b Accumulate KV cache
9: h, KV +- ENCODER(X", KV<h)
10: KV + KV** @ KV
11:
end for
12:
return x:B
Table 1: Forward-pass training FLOPs comparison. N de- notes the number of layers, the sequence length, S the block size, and 1) the hidden dimension. The expression highlighted in blue denotes the number of attention opera- tions. E2D2 refers to block diffusion parameterization. For E2D2, we separate into Non encoder and NDe decoder layers. We provide the derivation in Appendix B.
Table 2: Inference FLOPs compari- son. ((0) is cost of a decoder call and ((ø) is cost of an encoder call. For E2D2, O < ( T de- notes diffusion steps and B the num- ber of blocks. E2D2 uses the block diffusion parameterization.
Attention FLOPs
MLP FLOPs
Inference FLOPs
AR
4ND( L244) 4
2A
AR
L L-0(
MDLM
ANDL²
24NLD2
MDLM
T 0(0)
BD3LM
AND(L" + LS)
48NLD2
BD3LM
B ((0)
E2D2
B 0 BT O(0)
E2D2
4(NEsc + NEw ) 128
24(Nesc &ND)
encrder
deceder
3.4
Training Algorithm
In this section, we review the algorithm and complexity of a forward pass through the E2D2 model compute the loss of the block diffusion objective. The training procedure for F222 is presented in Algorithm/ 2. We model the posterior probabilities po 2 x for all blocks b E [I..... B] and context tokens x where C C. {I..... L} a set of their position indices. For block diffusion. context tokens for block b correspond to previous clean blocks x x Naively, would compute logits by invoking the encoder and decoder for each block in a loop, since the logits for each block are conditioned on block-specific inputs 2ł, x This entails calling the encoder hf ENCODER(x to encode the conditioning x. and the decoder to compute Xigg DECODER 2Z) hf).
Vectorized Implementation To efficiently model all posterior probabilities across B blocks using a single encoder and decoder pass, we process all clean tokens xI: through the encoder and noised tokens z 1:L through the decoder. We design custom attention masks (Figure 2) to ensure that noisy tokens attend within their noised block and to previous clean blocks, inspired by [1].
5

---

Encoder Attn.
Mask Menc
Decoder Attn. Mask Made
Self-attn winn xe 1:2.
x
Cross-atin to 1:l.
1:7. Self-attn w/in x
1 1:
Figure 2: Example attention masks used in block diffusion training with an encoder-decoder archi- tecture, for L = 6 tokens with blocks of size S = 2. Left: The encoder mask M E {0, 1 enables clean tokens to attend within their respective block and to previous blocks. Right: The decoder mask Min {0, L2L uses self-attention within noised blocks and cross-attention to previous clean blocks using the encoder's output representation h
In particular, the clean sequence x!: is passed to the encoder with a block-causal mask MEne {0, 11 to produce the repre- sentations h 1: so that each clean block conditioned itself x" and previous blocks
Algorithm 2 Encoder- -eeoddrr Training
Require: datapoint x noise process qr, ENCODER w/
params. Φ, DECODER params. diffusion loss L
repeat
x (Figure 2, left). In the decoder, for given noisy block z/. each token attends to tokens within its block zł and clean token represen- tations from preceding blocks xbb which we
2:
Sample t (4 1]
3:
L. qe( (e(- (x
4:
h.d +- ENCODER (x!:1, .MEa.
5:
I.L. <- DECODER(Z4 (z, h 1 .MDe)
denote as h <b, The decoder attention mecha- nism operates over 2T. keys and values corre- sponding to the L tokens in 1: and the L to- kens in 2 The decoder attention pattern is enforced through an attention mask Maee {0, 1 (Figure 2, right). We thus perform an encoder pass over the entire clean sequence, followed by a decoder pass over the entire noisy sequence, as follows: h EN (Mnne) and 1::-- Kogi = DECODER I:1. l:f. MDxe).
6: Take gradient step on Vo, To.ef((xipi Mogis 0,
7: until converged
Crucially, using smaller decoder significantly improves training throughput relative decoder- only BD3LM model. In Table 1, we depict the FLOPs comparison between modeling paradigms and E2D2 applied block diffusion, see Appendix for a detailed derivation. Letting N be the total number of layers in the model, which for E2D2 we have N NEne + NDe. we see that for same number of layers, E2D2 uses 2x fewer FLOPs compared to BD3LM. Moreover, for any block diffusion model of block size S < L, we achieve fewer training FLOPs relative to standard masked diffusion models (MDLM; see Appendix C) while maintaining the superior quality of the BD3LM block diffusion parameterization.
3.5
E2D2 for Standard Masked Diffusion (MDLM)
Below, we present details for training and sampling from a standard full-sequence masked diffusion model (MDLM [55]), introduced in Section 2.1, using our encoder-decoder architecture. Recall that here our denoising network predicts a joint distribution over the entire sequence pz I.I. 2 2!!²),
unlike block diffusion which requires predicting the posterior for each block of tokens.
The encoder takes as input a sequence of clean context tokens consisting of prompt tokens and clean tokens in z. The decoder takes as input the full noised sequence 2.:4 which outputs logits for each token conditioned on the encoder's output. This design facilitates faster inference, as the lightweight decoder may be invoked multiple times to generate number of tokens before calling the larger encoder to produce a strong contextual representation.
Sampling For full-sequence masked diffusion, the sampling procedure follows Algorithm 1, except that here the decoder generates groups of tokens that can be any position of the full sequence rather than in contiguous blocks. For more formal exposition see Appendix D and Algorithm 3. As above, we begin by encoding the P prompt tokens xI:P to attain the encoder representation 1.P,. The decoder's input is initialized to sequence of length I. consisting of only masked tokens. The decoder iteratively denoises this full sequence for a fixed number of steps with cross-attention to 1:P At the end of this denoising interval, the decoder has generated some new tokens, which are potentially

---

non-contiguous in the sequence dimension. These tokens are returned as a new 'block' to the encoder, which adds them to its running context and produces an updated clean token representation. This process is repeated until the sampling budget T is exhausted.
Training At every training step, sample Z.!L ((, x The input to the encoder consists of the clean tokens in this sequence. The decoder receives the entire sequence z and the encoder's representation of the clean tokens. Since we model the full sequence with diffusion, both the encoder
and decoder use bidirectional attention. See Appendix D for more detail.
4 Experiments
4.1 Experimental Design
We study the capabilities of E2D2 on summarization, translation, mathematical reasoning, and zero-shot likelihood estimation on held-out data. In each setting, we train domain-specite E2D2 and baseline models. In addition to performance metries for each task, we report inference throughput. By varying the decoder size, can map the Pareto frontier of efficiency and performance, with increasing number of layers controlling both performance (larger decoder -> higher performance) and throughput (larger decoder + lower throughput). As noted in Section 3, throughout the experiments, we use the block diffusion parameterizattoon in conjunction with E2D2.
Datasets We examine 1) text summarization MCi:: [27, 57J) for which we compute ROUGE scores [37]. 2) machine translation (WMT 14 de-en; [6]) for which we compute the BLEEU [50] score, and 3) mathematical reasoning (GSM8K; [10]) for which we compute zero-shot pass @ accuracy. We also train E2D2 on the widely used pretraining OpenWebTex, dataset [21]. We compute perplexity (PPL) the validation set of this corpus and PPL for held-out datasets (zero-shot PPL).
Baselines We compare E2D2 against decoder-only AR models and discrete diffusion models MDLM [55] and BD3LM [1]. For likelihood estimation, we also compare against SEDD [40].
Last Hidden State Design Decisions For the last hidden state version of E2D2, models parameterized using a transformer architecture based on Qwen3 [76]. Qwen3 features improvements over prior architectures, such as removing QKV biases and introducing QK normalization [14]. We maintain untied weights for the encoder and decoder. For the CNN/DailyMail, WMT, and OWT datasets, we train from scratch using this last hidden state version of E2D2. Full model and training hyperparameters are detailed in Appendix E.
pretrained Qwen3 weights to instantiate an E2D2 encoder, letting Nune be the same as the underlying Qwen3 LLM. We then copy these weights for the decoder, but reduce the number of layers, to instantiate lightweight decoder so that NDE < N In particular, we copy layers closer to the output. The KV cache from the encoder layer is provided the decoder layer from which it was copied, i.e., the 7th decoder layer we use KV [ - NDec + i]. We reduce the memory footprint of the model by weight-tying the encoder and decoder parameters.
Shared KV Cache Design Decisions For the shared KV cache variant of E2D2,
For GSMSK, we fine-tuned from a pretrained Qwen3 1.7B model using this shared KV cache architecture. Full details are in Appendix E. We find that for tasks where the downstream dataset is relatively small, e.g., GSM8K, initializing from the pretrained weights of the underlying Qwen model improves convergence and downstream performance. This is similar to previous findings that initializing diffusion training from a pretrained AR model improves convergence [22, 78].
4.2 Results
Summarization and Machine Translation For text summarization and machine translation, in Tables 3 and 4, we see that E2D2 is able match or outperform our diffusion baselines while achieving higher throughput. On the summarization task, E2D2 even outperforms an AR baseline that has the same number of layers, while increasing throughput by 75%%. We find that AR overfits on this dataset, unlike diffusion, which is robust against overfitting by training over masking rates [43, 52], Compared to MDLM, which does not support KV caching, E2D2 offers better language modeling and downstream task performance with ~3x faster inference. In Table 4, E2D2 achieves higher throughput and better BL.U scores for translation relative to the 16-layer BD3LM. While a small 12 12-layer BD3LM approaches the throughput of E2D2, its BLEU score worsens further.
7

---

Table 3: CNN/DailyMail test set ROUGE scores. Best values for our trained models are bolded. refers to number of transformer layers (Neae/Dee for E2D2). Decoding throughput (Tput) is measured in tokens / on H100 80GB machine. For all models, T= L. sampling steps, so the throughput can be higher for diffusion when < L. We report mean + standard deviation for 100 samples.
Table 4: WMT (de-en) test BLEU score. Best values for our trained models bolded. N refers to number of trans- former layers ( (Neme/ for E2D2). De- coding throughput (Tput) is measured in tokens sec on H100 80GB machine. For all models. we use so the throughput can be higher for diffu- sion when 7 < L. We report mean stan- dard deviation for 100 samples.
sampling steps,
ROUGE (t)
2
L
Past baselines
BLEU (t)
GPT-2 [53]
29.3
8.3
26.6
Past baselines
BERT-L [39]
41.7
19.4
38.8
NAT [23]
20.62
T5-L [54]
42.5
20.7
39.8
NAR autoencoders [34]
25,43
AR-Diff. (k- = 50) [74]
39.6
16.3
37.1
USMT [3]
17.43
GENIE (A= = 50) [38]
29.3
8.3
21.9
CMLM Base [20]
29.47
From scratch
N
Tput (t)
1
2
L
From scratch
N
Tput (1)
BLEU (t)
AR
28
89.1 89.1
31.7
11.7
22.1
32
77.6204
25.2
MDLM
28
49.3
30.6
12.5
22.7
MDLM
32
60.4
18.4
BD3LM
12
135.
35.8
13.7
23.7
BD3LM
12
129.6+a7
23.3
E2D2 (Ours)
20/8
155.8
36.0
14.1
23.9
BD3LM
16
102. (02.440s
24.0
E2D2 (Ours)
28/4
162.0+14
24.8
Table 5: Evaluation on GSM8K test set. Best diffu- sion value bolded. N refers to number trans- former layers (Neee/NNDee in the case of E2D2). De- coding throughput (Tput) measured in tokens on H100 80GB machine. For all models, we use = L sampling steps, so the throughput can be higher for diffusion when T < L. We report mean + standard deviation for 100 samples.
Pareto Frontier: Performance vs. Efficlency
50
40
Parformance: D-shot
30
Improved trade off
O-shot pass @ (1)
BD3LM
PPL
Tput (t)
E202
100
120
Efficiency: Throughput (tokens/sec,
Fine-tuned
AR
28
1.49
66.6
44.1
Mapping the Pareto Fron- tier: Larger models increase accuracy GSM8K at the cost of slower decoding.
Figure 3:
MDLM
28
<2.30
14.0
31.9430
BD3LM
< 1.87
33.2
86.6 86.6+a5
E2D2 (Ours)
28/14
1.80
47.9
102.8406
E2D2 improves this trade-off.
Mathematical Reasoning Table 5 contains results on the GSM8K benchmark. As above, E2D2 shows improved downstream performance and decoding throughput compared to diffusion baselines. Qualitatively, we also observe that E2D2 outperforms BD3LM models; see samples in Appendix G.
Mapping the Pareto Frontier of Performance and Efficiency By varying the depth of E2D2's decoder and that of bascline models, we can examine the trade-off between performance (which increases with larger models) and throughput (which decreases with larger models). We fine-tune models on the GSMSK dataset and compute 0-shot pass@ accuracy and decoding throughput. We select the number of decoder layers to roughly match the throughput for E2D2 and BD3LM models at various sizes, with BD3LM decoder layers varying over N E {10, 14, 17, 21, 28) and E2D2 (using NEne 28) decoder layers varying over Nox {6. 10. 14, 21, 26). At each throughput level, E2D2 features higher quality, extending the Pareto frontier of quality and speed Figure 3.
Likelihood In addition to downstream task performance, evaluate E2D2's 's language modeling capabilities. We train E2D2 model on OpenWehText (OWT; [21]) and compute zero-shot PPI. ()) on held-out benchmark datasets from [53]. E2D2 with the block diffusion parameterization outperforms

---

full-sequence diffusion baselines SEDD and MDLM on OWT perplexity. While attaining comparable perplexities BD3LM, we find that E2D2 is 40% faster train, as measured in training throughput (tokens / sec H100 80GB, with batch size 32 and context length T. 1024, measured after 100k steps) with a throughput of 8.4 x 10a for E2D2 and 5.9 x 10a for BD3LM.
Table Validation perplexities (4) for models trained on OWT. Perplexities for diffusion models are upper bounds. All models are trained on 524B tokens. All models use 170M total parameters and N 12 total layers; for E2D2, we use Nisc 10, NDe 2. indicates values taken from [1].
Zerv-shot
OWT
Wikitext
LMIB
Lambada
News
Pubmed
Arxiv
AR
17.54
81.07
25.32
51.14
52.13
52.11
48.59
41.22
SEDD
24.10
96.33
35.98
68.14 14
48.93
67.82
45.39
40.03
MDLM
22.98
90.96
33.22
64,94
48.29
62.78
43.13
37.89
BD3LM
20.73
96.81
31.31
60.88
50.03
61.67
42.52
39.20
F2D2
21.73
101.50
32.32
64.01
54.62
63.75
43.68
41.52
4.3 Ablations
We assess the impact of block size, number of diffusion steps, and choice of encoder-decoder architecture on downstream performance.
Table 7: Accuracy and decoding speed trade-off across varying block sizes for E2D2 on GSM8K. Decoding throughput (Tput) is measured in to- kens / sec on 1 A100 80GB machine. We report mean standard deviation for 100 samples.
Block size For E2D2, using larger block size increases throughput, as the encoder is invoked fewer times during generation. However, larger block sizes lead to worse quality, as the likeli- hood bound is less tight for larger blocks [1]. To explore this trade-off, we fine-tune E2D2 models with NEne/ Nec
Block size S
0-shot pass @ (f)
Tput
28/14 and varying block sizes S E {2, 4, 8, 16, 32} on the GSMSK dataset. We report results in Table 7 where we observe the anticipated trade-off, with the best quality coming from the smallest block size S = 2 and the highest throughput coming from the largest block size S 32.
32
20.9
62.2423
16
33,0
58.341.
37.4
52.7404
47.9
45.7
50.1
34.8104
Number of Diffusion Steps In Table 8, we show the effect of diffusion steps per block on sample quality using F2122 and decoder-only BD3LM. For both models, BD3LM in quality for each T. As T decreases, relative throughput gains from E2D2 diminish, since the decoder requires fewer invocations. For T 1. E2D2 achieves comparable throughput to the 16-layer BD3LM because every sampling step invokes both the encoder and decoder, which together comparable FLOPs relative the 16-layer BD3LM. The 12-layer BD3LM achieves better throughput than E2D2 for T 1, but sacrifices quality.
4. F2D2 outperforms
Table 8: Performance and decoding speed trade-off across varying number of diffusion steps T for BD3LM and E2D2 on WMT. Decoding throughput (Tput) is measured tokens sec on I A100 80GB machine. Note that for all models, we use L total sampling steps, the throughput for diffusion models can in practice increase when T < for block size S. We report mean + standard deviation for 100 samples. Best BLEU and throughput numbers bolded.
BD3LM
BD3LM
E2D2
N =
12
N
16
NEwe/ /Nex 28/
T
BLEU (1)
Tput (1)
BLEU (t)
Tput (1)
BLEU (1)
Tput (1)
1
17.2
149.2424
17.9
118.7+20
19.3
116.6
2
21.8
100.7
22.4
77.0+12
23.2
103.7
23.3
62.5
24.0
47.1 17.1
24.8
79.0+15

---

Architecture Design We compare the performance of the "last hidden state" and "shared KV cache" architecture variants (Section 3.2) when used to pretrain models for summarization and fine-tune models for math reasoning. In Table 9, we report perplexities from training models for 20k steps.
For models trained from scratch, e.g., the larger sum- marization dataset, the "last hidden state" design per- forms best. We hypothesize that conditioning on the encoder's s last hidden representation provides a richer signal than using intermediate representations. When fine-tuning from strong Qwen3 model on the smaller GSM8K dataset, the "shared KV cache" variant achieves better perplexity. We speculate that by reusing the en- coder's cached KVs, this design keeps the decoder's cross-attention inputs aligned with those expected by the base model, which facilitates more stable fine-tuning.
Table 9: Validation PPL upper bounds (L)
for last hidden state and shared KV cache versions of E2D2 on CNN/DailyMaii and GSM8K. i indicates model used in the main results. Best values are bolded.
CNN/ DailyMail
GSM8K
Last Hidden State Shared KV Cache
14.94
1.98
293.50
1.80
5 Related Work
Encoder-Decoder Models T5 [54] and follow-up works FLAN and FLAN-UL2 [9, 64] demonstrated the efficacy of encoder-decoder architectures on a wide range of natural language benchmarks. Recently, this architecture T5Gemma [79]. In contrast to our work, T5-style architectures use AR decoders and they separate out artsstattention between decoder and encoder representations into distinet modules, opposed to performing attention over both decoder and encoder outputs in one forward pass, as in our work. Moreover, in these AR T5-style models, the encoder is only used once to provide an embedding of the input context. In E2D2, the encoder is applied to the context and each newly decoded block.
applied to the more updated Gemma family of models [65] in
Diffusion Language Models Early approaches modeled word embeddings via Gaussian diffusion [24, 36]. Modern diffusion language models adopt a decoder-only architecture that is trained via denoising, similar to BERT [15]. The LLaDA model [46] scales masked diffusion models [45, 55] to the 8B parameter regime and demonstrated comparable and superior performance to a similarly-sizes AR Llama model [17]. Another large discrete diffusion model Dream7B [78], which extends the DiffuLLaMA framework [22], uses pretrained AR models to initialize their models.
Unlike E2D2, these models only train with a full-sequence diffusion parameterization instead of the performant block diffusion framework [1]. When fine-tuning tin-ttning from AR checkpoints, this requires that models, such Dream7B, carefully anneal the causal attention mask of the original pretrained AR LLM towards fully bidirectional one. In contrast, for small block sizes S < L, our work does not require this tuned annealing tokens only attend to tokens up to S positions in the future.
ENAT Highly related to our work is the discrete image generation model ENAT [44], which leverages an encoder-decoder to decouple computational budget allocated to processing clean and masked tokens. The bidirectional nature of this non-AR model, however, precludes ENAT from leveraging efficient KV caching methods explored here. While our method supports variable-length sequence modeling, the ENAT architecture is only applicable to fixed-length outputs such as images.
Efficient Inference An analogy can be drawn between E2D2 and speculative decoding methods [8, 35], which speed up generation for AR models by utilizing smaller proposal networks. Most related to our work is Medusa [7]. which uses several small MLPs to decode blocks of tokens conditioned on the final hidden state of the LLM. In contrast to our work however, these methods use accept/reject algorithms to keep proposal network tokens, while we use diffusion sampling.
Conclusion
In this work, we propose E2D2, a novel encoder-decoder architecture with efficient diffusion training and sampling algorithms. Our results demonstrate that E2D2 better trades off throughput and sample quality relative to prior state-of-the-arr masked diffusion models. While E2D2 represents promising direction to narrow both the performance and throughput gaps relative to AR models, further innovation is required increase training efficiency and sample quality. Finally, given we are working on language modeling, carry the inherent risks and opportunities of this line of research.
10

---

Acknowledgments
This work was partially funded by the National Science Foundation under award CAREER 2145577, and by the National Institute of Health under award MIRA R35GM151243. Marianne Arriola supported by NSF Graduate Research Fellowship under award DGE-21 21 39899 and Hopper- Dean/Bowers CIS Deans Excellence Fellowship. We gratefully acknowledge use of the research computing resources of the Empire AI Consortium, Inc, with support from Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation [5]. This research was also supported in part through the use of computational resources provided by Lambda (Lambda. ai) in partnership with Open Athena AI Foundation, Inc. We gratefully acknowledge their generous GPU infrastructure grants that helped make this work possible and thank Eric Czech of Open Athena for the useful discussions and feedback.
References
[1] Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025.
[2] Marianne Arriola, Naveen Venkat, Jonathan Granskog, and Anastasis Germanidis. Adapting autoregressive vision language models for parallel diffusion decoding. Runway Al Blog Post, 2025.
[3] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Unsupervised statistical machine translation. arXiv preprint 1109:18009011277 2018.
[4] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 19:17981-17999 2021.
[5] Stacie Bloom, Joshua C. Brumberg, Ian Fisk, Robert J. Harrison, Robert Hull, Melur Ramasub- ramanian, Krystyn Van Vliet, and Jeannette Wing. Empire Al: A new model for provisioning Al and HPC for academic research the public good. Practice and Experience in Advanced Research Computing (PEARC '25), page 4, Columbus, OH, USA, July 2025. ACM.
[6] Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Sorieut, Lucia Specia, and Ale s Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
[7] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.1077 2024.
[8] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.011316 2023.
[9] Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetun language models. Journal of Machine Learning Research, 25(70): 2024.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenyy Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[11] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang. and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the
11

---

Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018.
[12] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. The Twelfth International Conference Learning Representations, 2024.
[13] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941. PMLR, 2017.
[14] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
[16] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
[17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[18] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
[19] Itai Gat, Heli Ben- Hamu, Marton Havasi, Daniel Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer, and Yaron Lipman. Set block decoding is a language model inference accelerator. arXiv preprint arXiv:2509.041 2025.
[20] Marjan Ghazvininejad, Omer Levy. Yinhan Liu, and Luke Zettlemoyer. Mask-predich Parallel decoding of conditional masked language models. Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), pages (112-6121) Hong Kong, China, November 2019. Association for Computational Linguistics.
[21] Aaron Gokaslan, Vanya Cohen, Ellic Pavlick, and Stefanic Tellex. Openwebtext corpus. http://Skyliono00/ git.ub..bbuub...g ho/bpeeneeebTexttrrpuu... 2019.
[22] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025.
[23] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.0222 2017.
[24] Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
[25] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard- Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-363 September 2020.
Kevin Sheppard, Tyler
12

---

[26] Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao, Tianyi Bai. Ran Yan, Jie Fu, Chongxuan Li, and Binhang Yuan. Ultrallada: Scaling the context length to 128k for diffusion large language models. arXiv preprint arXiv:2510) 10481, 2025.
[27] Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay. Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In NIPS, pages 1693-1701, 2015.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[29] J. D. Hunter. Matplotlib: A 2d graphies environment. Computing in Science & Engineering. 9(3):90-95; 2007.
[30] Yuchu Jiang, Yue Cai, Xiangzhong Luo, Jiale Fu, Jiarui Wang, Chonghan Liu, and Xu Yang. d² cache: Accelerating diffusion-based Ilms via dual adaptive caching. arXiv preprint arXiv:2509.230099 2025.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[32] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 132161131 2013.
[33] Volodymyr Kuleshov. Fast algorithms for sparse principal component analysis based on rayleigh quotient iteration. In International Conference on Machine Learning, pages 1418-1425. PMLR, 2013.
[34] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic -on-autoreeressvve neural sequence modeling by iterative refinement. arXiv preprint arXiv:1802.0690 2018.
[35] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pages 19274-19286. PMLR, 2023.
[36] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022.
[37] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[38] Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre- mre-trannnn approach with continuous paragraph denoise. In International Conference on Machine Learning, pages 21051-21064 PMLR, 2023.
[39] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.088 08345, 2019.
[40] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023.
[41] Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):3133333 1993.
[42] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[43] Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, Zili Wang, Hang Yan, Tianyu Pang, and Michael Qizhe Shieh. Diffusion language models are super data learners, 2025. Notion Blog.
13

---

[44] Zanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han, Jiayi Guo, Zhiyuan Liu, Yuan Yao, and Gao Huang. ENAT: Rethinking spatial-temporal interactions in loken-based image synthesis. In The eiihth- Annual Conference Neural Information Processing Systems, 2024.
[45] Shen Nie, Fengqi Zhu, Chao Du Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024.
[46] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
[47] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.0377 2024.
[48]
The pandas development team. pandas-dev/pandas: Pandas, February 2020.
[49] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534, Berlin, Germany, August 2016. Association for Computational Linguistics.
[50] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performant
Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, d' Alché Buc, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[52] Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffusion beats autoregressive in data-constrained settings. arXiv preprint arXiv:2507.15837. 2025.
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[54] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
[55] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.00 075544 2024.
[56) Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple and controllable uniform discrete diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025.
[57] Abigail See, Peter Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[58] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
14

---

[59] Jascha Sohl- Dil-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMIR, 2015.
[60] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[61] Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo. Zheng Li. Yuehang Yang. Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint ar (tX.Xi:2508.029935 2025.
[62] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-timg discrete diffusion models. arXiv preprint arXiv:2211.16755 2022.
[63] Uya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[64] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gareia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint X2w::2205.0011111 2022.
[65] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
[66] The Mosaic ML Team. oonp: 2021.
https://github.
[67] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[68] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.003007 2025.
[69] Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang. Christopher De Sa, and Volodymyr Kuleshov. InfoDiffusion: Representation learning using information maximizing diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 36336 36354. PMLR, 23-29 Jul 2023.
[70] Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021, 2021.
[71] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv: 1910.0377. 2019.
[72] Chengyue Wu, Hao Zhang, Shuchen Xue, Shizhe Diao, Yonggan Fu, Zhijian Liu, Pavlo Molchanov, Ping Luo, Song Han, and Enze Xie. Fast-dllm v2: Efficient block-diffusion 1lm. arXiv preprint arXiv:2509.26322 2025.
[73] Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xic. Fast-dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding. arXiv preprint 2505:22505.226/8 2025.
[74] Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957-39974, 2023.
[75] Onry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019.
15

---

[76]
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv. Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang. Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei L.,, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu, et al. Qwen3 technical report. arXiv preprint arXi::2505.0980 093888 2025.
[77] Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15800, 2025.
[78] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025.
[79] Biao Zhang, Fedor Moiseev, Joshua Ainslie, Paul Suganthan, Min Ma, Surya Bhupatiraju, Fede Lebron, Orhan Firat, Armand Joulin, and Zhe Dong. Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation. arXiv preprint wrXiv:2504.06225, 2025.
[80] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
[81] Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504. 12216, 2025.
[82] Fengqi Zhu, Zebin You, Yipeng Xing, Zenan Huang, Lin Liu, Yihong Zhuang, Guoshan Lu, Kangyu Wang, Xudong Wang, Lanning Wei, et al. Llada-moe: A sparse moe diffusion language model, arxiv preprint 2rxiv:2509.24389 2025.
16

---

Contents
1
Introduction
2
Background
2.1
Discrete Diffusion Models
2.2
Block Diffusion Language Modeling
3
Efficient Encoder-Decoder Diffusion
3.1
Architecture
3.2
Design Decisions
3.3
Sampling Algorithm
3.4
Training Algorithm
3.5
E2D2 for Standard Masked Diffusion (MDLM)
4
Experiments
4.1
Experimental Design
4.2 Results..
4.3
Ablations
Related Work
10
6
Conclusion
10
Extended Background on Block Diffusion Language Models
18
A.1 Forward Noise Process
18
A.2 Reverse Denoising Process
19
A.3 Negative Evidence Lower Bound (NELBO)
19
B Training FLOPs derivation
20
FLOPs comparison to MDLM
D MDLM with Encoder-Decoder Architecture
21
Additional Experimental Details
22
E.1
Measuring Throughput
22
E.2 Summarization
23
E,3
Translation
23
E,4
Mathematical Reasoning
24
E.5
Language Modeling
24
Assets
25
G Generated Samples
25
17

---

G.1 E2D2 G.2 BD3LM (N 21)
25
26
A Extended Background on Block Diffusion Language Models
Below, we provide an extended overview of block diffusion language models closely following
[1]. In particular, we define the masking process (zz z 2l: and the derivation of the objective C(x I.B, 0) for parameters 0.
Recall that we factorize the sequence x 1::2 drawn from the data distribution q(x ((x1:2) over B blocks of size S. We simplify notation by denoting the tokens x(b-1)-S:b-5 in block b as x². We perform diffusion in each block over T discretization steps. We define t, to denote t(i) i/T and s(i) = (i 1)/T, /T, for all i [1, We denote the Kullback-Lein divergence as DKL|-]. Below we reproduce the negative evidence lower bound (NELBO) following [1]:
log Po())
log Pa (x" x Σ
b=1
po (2 (O( (x)
log E,,
(( (x))
b=1
Pa pa (x <h IT
Pal Po (2) [100 xx:)
log E,
II ((2br))="")
Eg log (x" 12 4. xx xx)
b=
+ E i f 1J Eo TrK KL (a(z)la x²) |I po po x²))
Camooo
+ DKL (9(2)_1(x²) ll po (2(2==))
(1)
Cyexe
A.1 Forward Noise Process
Within each block, [1] adopt the masked diffusion language modeling framework, which is the most performant [4]. They use simplified NELBO as proposed by [47,55, 58).
Following [4], we first define diffusion matrix Qe E VVV corresponding to noise level for states i E (I...., } and vocabulary size V. Recall that the prior distribution is the absorbing mask state one-hot vector centered on the special [MASK] token index. We denote this mask index as the last token in the vocabulary, arg max, m, = V. Consider the noise schedule function ag E [0, 1], which is monotonically decreasing in t, with du 1 and a 0. Then, the diffusion matrix is:
V if
Q1ly
at
if = j HV
Oz
if = ViiV
For the forward marginal Qris, we use forward marginal probabilities according to axja = a./a,. The forward noise process is applied independently for each token E [}. We first define the transition matrix Q((a) = Qun)Qu( ... Que) which is used in the forward process q(z[[x'= = Cat (zf: Qxx
18

---

A.2 Reverse Denoising Process
We now may obtain the reverse posterior q(z z/1 x²) derived in D3PM [4] as follows, where O denotes the Hadamard product between two vectors:
q(2, x (g((((())) Cat 2 Qujo2 @ Q.xx (zf) Q7 (e)) xf
(3)
((2((x))
A.3 Negative Evidence Lower Bound (NELBO)
As in block diffusion [1]. we adopt the simplified objective for masked diffusion [47, 55, 58] to obtain a tighter NELBO. We provide the sketch for the derivation, with the full proof is provided in [47, 55,58]. 58].
We first focus on simplifying the diffusion loss term in (1). To simplify the denoising model, we enforce the following constraints on the design of the denoising network by taking advantage of the fact that there only exists two possible states in the diffusion process z E {x", m) ve E {1 {I....,4} [55].
1. Zero Masking Probabilities. The clean sequence does not contain masks. So, we set (x'
z ==0=. O.
2. Carry-Over Unmasking. If a token is unmasked in the reverse denoising process, it is never remasked by definition of the masked diffusion process. The true reverse posterior for the case where z m is qz', z/|z + m) 1. Thus, set po (z' z Zzz + m 1.
With these two simplifications, we now use our denoising model to parameterize the posterior (z( x m). Let x denote token in the b-th position in block b E {1,..., B}. The diffusion loss term is derived as follows:
diiiusion
Σ E,E.T DKL [a(x") bi x*)
x b)]]
B
L' DKL ((22)112 xxx* (2) (2 x x
D E,E
Σ
b=1
{=1
DRr. is simply the discrete-time diffusion loss for the block hence, from [55] (Suppl. B.1), we get:
E'
E
Σ
ag
log
(x
Zz
be
xb)
aie
1=1
1=1
Σ E,E.T
log (x 2i, x(x))
(4)
aq
Lastly, obtain tighter approximation of the likelihood by taking the diffusion steps T 0o [55].
for which T(o a.) of:
B Σ 10.1
ai log po (xe) ZZ x x0)
Cdittusion
(5)
11-ac
6=1
When taking + 0o. [55] (Suppl. A.2.4) show the reconstruction loss becomes 0. In particular, we use the fact that 2/(1) lim Cat (:ib== Cat(:;x)). Then, we obtain:
Lrccns
Ey log po (x" (2)) xh)
log pa xx" 22(()) = =x, xb)
(6)
Also, the prior loss Spnor = DKL 1x"= || po (z) reduces to 0 since a;=1 = 0 so that q(z)_11xx) (x*) Cat(.; m) and pa (z!m1) Cat(.; m); see [55] (Suppl. A.2.4).
19

---

The final diffusion objective simply becomes a weighted average of cross-entroos terms:
B ag log bogpo(x)) x1,x=00
LBD(X; 0) Σ
10
(7)
1-ai
6=1
B Training FLOPs derivation
We derive the training FLOPs of a model forward pass in Table as follows. Let L be the sequence length, N be the number of layers, and D be the hidden dimension. We caleu- late the attention FLOPs of the forward pass following FlashAutention [12], which is 4 for bidirectional attention, and multiplying the attention output by values) requires 2N FLOPs. Assuming a gated MLP with three dense layers [13] and intermediate hidden dimension of 4D, the MLP FLOPs are given as 24NLD2 = 2NT((D)4D)) (up-projection) + 2NT(())4D)) (gate projection) 2N (DI((D)) (down-projection)
each matrix multiplication (i.e.. multiplying queries and keys
Autoregressive.
The number of attention operations in causal attention is: EE=i= +2+4 Thus, 2
AR training uses 4ND( ) attention FLOPs. The MLP FLOPs are given as 24NLD².
Full-Sequence Diffusion. Sequence diffusion models such as Masked Diffusion LMs [47, 55, 58] training uses D22 attention FLOPs from bidirectional attention, requiring L attention operations. The MLP FLOPs are also 2ANLD*
Block Diffusion. For block diffusion models [1], the sequence is split into B blocks of size S, where T, BS
The training algorithm proposed in BD3LM [1] is 2x more computationally expensive relative to both AR and full-sequence diffussin.. Below, we summarize the attention operations used compute the attention FLOPs from [1].
BD3LM uses attention mask MBDESLM {0, 1] for both the noised tokens z!!Z and clean tokens x xl This mask can visualized as a concatenation of the encoder and decoder masks from Figure 2. This mask is thus comprised of four smaller attention masks:
0
MBC
MBDALM = MOBC
MBD
where MBD and MoBc are used to update the representation of zz and MBc is used to update the representation of x 1:L. We define these masks as follows:
1. Block-causal self-attention mask to update clean x! MBC {0, 1x
(Macly = { if belongs in the same block as i, or a block before i otherwise
Thus, MBc is nonzero for Ei= 652 = BS
L2+LS
entries
2
2
2. Block-diagonal self-attention mask update noised B..B.. MaD {0, 1
MaDal = {1
i are in the same block
0 otherwise
Thus, MBD is nonzero for BSa = LS entries.
3. Offset block-causal cross-attention mask to update 27 2:BB via cross-attention to x:B
[MOBCl {
j belongs in a block before otherwise
Thus. MOBC MBC -MD such that the number of nonzero entries L2+LS
LS
2
12_L8
20

---

Thus, the total number of attention operations is 12 + LS 12-18 1,2 LS.
The number of MLP FLOPs is ASNLD' as the input sequence to each layer is of length 2L from both x 11 and 2::4
Encoder-Decoder Block Diffusion (Ours). E2112 uses half the attention FLOPS compared to BD3LM by splitting the operation into encoder and decoder layers. This is because each encoder and decoder layer uses the same number of attention FLOPs.
1. Encoder attention operations. The encoder uses mask MEne 10,11, =Mac. Thus,
the encoder uses LS attention operations
2. Decoder attention operations. The decoder uses mask MDe. E -0.1)xxx1. M
Thus, the decoder uses LS-LS + LS L*+LS attention operations.
The FLOPs of an attention pass is 4(NEne + NDee D The MLP FLOPs is 24N since the input sequence contains L tokens each layer.
C FLOPs comparison to MDLM
Below, we show that E2D2 attention FLOPs for the block diffusion parameterization are upper- bounded by decoder-only MDLM FLOPs under the same model size. Assume both models have the same number of total layers, where N NE + Ne Then,
FLOPSE2D2 I FLOPSMDLM
(8)
L + LS
4ND
S SNDDL2
2
LL + LS
S
2
TL+S
2
S
So. as long as L, E2D2 FLOPs are fewer than that of MDLM under equal model size. When $ = L,, we exactly recover MDLM FLOPs.
D MDLM with Encoder-Decoder Architecture
Notation Recall that I:P denotes P prompt tokens. We denote the set of unmasked token indices in z!! l:f. as C, {e z * m}, and we define x Drec, zf. The encoder receives inputs
Xt,E = x 1 C2 and produces representations h
Sampling In Algorithm 3, present the sampling procedure for MDLLM parameterized by encoder-decoder architecture. We denote the denoising steps budget as T, and the reverse diffusion process proceeds f 1, T-1 T We also denote the number of steps where only the decoder T
called by n. Thus, the encoder will be invoked at time steps
We begin by encoding the prompt tokens x xI: to attain the encoder representation 1:P, The decoder's input is initialized to sequence of length L consisting of only masked tokens. The decoder iteratively denoises this full sequence for fixed number of steps with cross-attentinn to h I:P Later in the decoding process, in some interval [s, t] (with s invoked over n evaluations and encoder is not invoked, we treat h as constant for all E [s, 4). At the decoder returns any new tokens decoded from the interval [s, 0.. which are incorporated into the encoder's input, and the encoder evaluated produce an updated h,. This process repeated until the sampling budget T' is exhausted.
7), during which only the decoder
21

---

Algorithm 3 Encoder-Decoder MDLM Sampling
Require: Prompt x I:P, number of steps during which only the decoder is invoked n, number diffusion steps T, ENCODER, DECODER, algorithm SAMPLE
I: f ENCODER(x PP)
b Encode prompt
2: u: 1:L m 3:4
Initialize fully noised sequence
3: for T,
T.T n, 2n, .,....771 do
4:
-+=
5:
for n steps do
6:
2/ &- SAMPLE hh) h))
Denoise sequence: Apply small decoder only
7: end for
f- Unmasked tokens 2 9: X1,Enc f- xIP $ x
Accumulate output
10: h - ENCODER(Xx,B.)
P Encode new "block"
11: end for
12:
return Z!
Encoder Attn. Mask Mare
Decoder
Mare
Self-arto w/in x
Cross-attn ht
Self-attn w/in xx!
h2 2
Figure 4: Example attention masks used in training full-sequence diffusion with an encoder-decoder architecture, for = tokens. We denote clean tokens in z! as xC*, where C, corresponds to their token indices. this example, |Cr] = 3. Left: The encoder mask MEne E {0, [0,1)iCH 1][C/Ix[C. enables clean tokens to attend to other clean tokens in the sequence. Right: The decoder mask MDee (0, 1} (x++)))))) uses bidirectional attention across the noised sequence 2!! and cross- attention to the encoder's output representation C:
Training During training, we sample Z!:L qzz.'LL x1 The input to the encoder is the sequence of unmasked tokens x from the full latent sequence z.=Z (along with the prompt tokens xI:P. if available), and it produces a representation of these tokens h, The input to the decoder is 2:22 along with h, and outputs logits for each token position in 2!:2.
In Figure 4, we present sample MEne and Mass for training MDLM with E2D2. Both the encoder and the decoder use full bidirectional self- and cross-attention. For batched training, where the number of unmasked tokens can differ between sequences, we may pad the encoder input sequences: xi + ( Edc, [PAD]) where [PAD] is special padding token. We also adjust the attention masks MEn and Miec accordingly, disable attention to padding tokens.
E
Additional Experimental Details
For all of our experiments, we used architectures based on the Qwen3 family of models [76]. In all training and fine-tuning experiments, we used the ADAM optimizer [31] with weight decay 1c-5 and (51, (32) = (0.9, 0.98). We also apply gradient clipping to 1.0. We also utilize an exponential moving average (EMA) copy of each model with decay rate of 0.9999, The EMA model was used during evaluation.
E.1 Measuring Throughput
For the main results, inference throughput was measured on single H100 (80GB) GPU using batch size of For the ablation results, we used a single A100 (80GB) GPU. We first processed 100 'warm-up" samples and then recorded decoding throughput for the subsequent 100 samples. Models generated 256 additional tokens, with no [EOS] or other early decoding stopping criteria applied. We report mean + the standard deviation of tokens per second for the 100 generated samples.
22

---

Last hidden state
Shared KV cache
Encoder
Decoder
Encoder
Decoder
DMASK
lolve
add
MAASK
Transformer Layer
Transformer Layer
Transformer Layer 4
Transformer Layer
AAlli
T' dikoodingg stapo per blok
decodding blook
K
Tranaformer
Tranaformer Layer
Transformer Layer
Layer
Tranaformer
Tranaformer Layer
Siappe
Layer?
Transformer Layer
Tranaformer Layar
Tonnsformer
Siippped
What
[.NSK]
add
IMASK
What
ts
[MASK]
TMA-KK
Accumulated samples
Noisy input block
samples
Noisy
Figure 5: Encoder-decoder architecture for discrete diffusion. The encoder embeds the clean sequence. The lightweight decoder denoises blocks over decoding steps by cross-attending to the encoder output. Left: "Last hidden state" version of E2D2: each decoder block attends the last hidden state output of the encoder. Right: "Shared KV cache" version of E2D2: Decoder layers attend to the keys and values of corresponding encoder layers. Encoder and decoder weights are tied.
E.2 Summarization
Data For this task, we use the CNN/DailyMail dataset version 3.0 [27, 57] downloaded from https:/ htops://hugat /ttugggggfacc....c/////// wwt./aaaaet//aaasseeeeeeeeeeeeee caaisse/cn__ddiymaiiiiii Data was pre-processed to add prefix to summarizations: "Summary: ". Inputs and targets were truncated to a maximum length of 512 each, ensuring a maximum sequence length of 1024 for sequences seen during training.
Hyperparameters We used the Qwen/Qwen3-00 w........-BBBe. 6B-Base tokenizer. All models had hidden size of 256 and intermediate hidden size of 768. The AR and MDLM baselines consisted of 28 transformer layers, corresponding to 80M parameters. BD3LM had 12 layers and used block size of S = corresponding to 60M parameters. E2D2 also used S = 8 and consisted of 20 encoder and 8 decoder layers, corresponding 80M parameters. We used the "last hidden state variant" of the E2D2 model.
We trained with batch size 128. Learning rate was linearly warmed-up for 1000 steps until a maximum of 3e Models were trained for maximum of 500k steps and early stopping the validation loss to select the best model.
Evaluation Models generated 256 additional tokens. Metrics were computed using the evaluate library from HuggingFace. Inputs were pre-processed as above, with the exception that input texts were truncated to 2048 tokens, and generated samples were post-pree proseproceseed truncate summaries after the end of sentence special tokens. For MDLM, BD3LM, and E2D2, we apply exponential length penalty starting at tokens with a factor 1.1 and repetition penalty with factor of 1.5. For the AR model, we do not apply the length repetition penalties as these led performance degradation.
For the BD3LM model, we found improved performance when aligning the input sequence to block length. That is. for context of length |C] the last C' mod S tokens were fixed the first decoding block. For E2D2 and MDLM models, this alignment was not necessary. Finally, for MDLM we decoded semi-auloregrrssively using block size of 32.
E.3 Translation
Data For this task, we use the WMT14 German English (de-en) dataset [6] downloaded from https://nuggingface www/aaaatts//weeeeeccceee mttt/mtiii For AR models, data pre-processed to
23

---

add prefix to the translations: "Translation: Inputs and targets were truncated to a maximum length of 128 each, ensuring maximum sequence length 256 for sequences seen during training.
Hyperparameters We used the Qxen/Qwen3-O. BB-Base tokenizer. All models had hidden size of 512 and intermediate hidden size of 1536. The AR and MDLM baselines consisted of 32 transformer layers, corresponding to 250M parameters. We trained two BD3LM models, one with 12 layers and one with 16 and used block size of S = 4. This corresponds to 140M and 160M parameters, respectively. E2D2 also used S 4 and consisted of 28 encoder and 4 decoder layers, corresponding to 250M parameters. We used the "last hidden state variant" of the E2D2 model.
We trained with batch size 128. Learning rate was linearly warmed-up for 1000 steps until maximum of 34- Models were trained for a maximum of 500k steps and we use early stopping on the validation loss to select the best model.
Evaluation Models generated 256 additional tokens. Metrics were computed using the evaluate library from HuggingFace. Inputs were pre-processed as above and generated samples were post- processed to truncate translations after the first '.' the end of sentence special token.
For the BD3LM model, we found improved performance when aligning the input sequence to block length. For E2D2 and MDLM models, this alignment was not necessary. Finally, for MDLM we decoded cerieeuloregrrssiveeyy using a block size of 32.
E,4 Mathematical Reasoning
Data For this task, we use the GSM8K dataset [10] downloaded from https:// dww.sattsoopaaaa/ggeeeeeeeeeeeee
(main version). Data was pre-processed to add a prefix to inputs: "Please reason step by step, and put your final answer within $bboxed($.." answers were prefaced with "Answer:
huggingface
and solution was wrapped in "S (0 " Inputs and targets were truncated to a maximum length of 384 each, ensuring maximum sequence length of 768 for sequences seen during training.
Hy perparameters We used the Bwen/Qwen3: 11. 7B-Base tokenizer. For AR and MDLM models, we used all the layers of the pretrained 1.7B parameter Qwen3, which has a hidden size 2048 and intermediate hidden size of 6144. For the BD3LM baselines, we used the N layers closest to the input, as opposed to the N layers closest to the output, since we found this to be the stronger of the two baselines. For our BD3LM baseline where N 21, this corresponds to 1.4B parameters. For the E2D2 decoder, we used the NDe layers closest to the output. We used the "shared KV cache variant" of the E2D2 model and the encoder and decoder layers trainable parameters. BD3LM and E2D2 models were trained using block size of 5 4.
weight-tied. Thus, E2D2 uses 1.7B
We trained with batch size 1. Learning rate linearly warmed-up for 100 steps until maximum of Ic 3 and decayed using a cosine schedule until half the maximum value. Models were trained for maximum of 30k steps and we use early stopping the validation loss select the best model.
Models generated 512 additional tokens. Inference was done using the Im-eval harness library with the 'flexible match' criteria and similar pre-processing was applied to question texts. post-processing were reverted to their original form of ### <Answer>. Inputs were pre-processed as above.
Evaluation
done to truncate text at end of sentence special characters and solutions
For both the BD3LM and E2D2 models, we found improved performance when aligning the input sequence to block length.
E.5
Language Modeling
Data We train on the Webnee [21] dataset downloaded from We use the GPT2 tokenizer. We do not pad or truncate sequences, but concatenate them and wrap them to a length of 1024. Since OWT does not have validation split, we leave the last 100k documents for validation. Following [1], we do not inject [BOS) and [FOS] tokens the beginning and end of each sequence in order to enable arbitrary-length generation.
co/
datasets (Skylloo007/openeeeeeeeee
24

---

We evaluate zero-shot likelihoods on datasets Penn Tree Bank (PTB; [41]), Wikitext [42], LMIB, Lambada [49]. AG News [80], and Scientific Papers (Pubmed and Arxiv subsets; [11]). When reporting zero-shot likelihoods on benchmark datasets from [53], we wrap all sequences to 1024 tokens and do not add [FOS] between sequences following [55].
Hyperparameters We train with a batch size of 512 and context length of 1024 for 1M steps. We trained E2D2 using a block size of S 4. All models use 12 total layers, and for E2D2 we set Nine 10, Nee 2. All models use 170M total parameters, including token embeddings. For E2D2, learning rate was linearly warmed-up for 2000 steps until constant peak of 3e-
Evaluation We report validation likelihood for OWT, and we use the test splits for reporting zero-shot likelihoods using the datasets from [53]. We use the same context length of 1024 tokens at evaluation.
F
Assets
In Table 10, we list the corresponding licenses for datasets used this work.
Table 10: Datasets and corresponding licenses.
Dataset
Licence
CNN/DaillMail
Apache-2.00 License
WMT14
GSM8K
MIT License
Open WebText [21] Creative Commons CCO license ("no rights reserved")
In Table 11, list the corresponding licenses for software packages used in this work.
Table 11: Software and corresponding licenses.
Library
License
HuggingFace [71]
Apache 2.0
Hydra [75]
MIT
Language Model Evaluation Harness [18]
MIT
Matplotlib [29]
Matplotib license
MosaicML Composer [66]
Apache 2.0
NumPy [25]
NumPy license
OmegaConf
BSD 3- 3-Clause
Pandas [48]
BSD 3- Clause "New" or "Revised"
PyTorch [51]
BSD-3 Clause
Seaborn [70]
BSD 3-Clause "New" or "Revised"
TetchMetries
Apache 2.0
G Generated Samples
Below, we provide example generations from GSM8K from E2D2 NEne/ /Ne 28/14 and BD3LM
N 21.
G.1 E2D2
Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the
the morning, she gives her flock of chickens cups of feed. afternoon, she gives her chickens another 25 cups of feed. How many cups
25

---

of feed does she need to give her chickens in the final meal of the day if the size of Wendi's flock is 20 chickens?
Answer: Wendi feeds her chickens cups of feed per day for 20 chickens for total of 3+20 <3+20=60x60 cups of feed. In the morning, she gives her chickens 15 cups of feed and in the afternoon, she gives them another 25 cups feed for a total 15+25 <15+25=40>40 cups feed. She needs 60 cups of feed and has already given 40 cups so she needs to give her chickens 60-40 260-40-20>20 cups of feed the final meal ####
Question: Josh decides to try flipping a house. He buys house for $80,000 and then puts in $50, 000 repairs. This increased the value of the house by 150%. How much profit did he make?
Answer: The repairs cost 50, 000000005550000 09.00...554000005555002222222223 house was worth for 80, 00000005500044000000000000000000 000. So he
So the
made a profit of 105, 0055::00000000001110000000000000 #*## 25000
G.2 BD3LM (N- 21)
Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon,
gives her chickens another 25 cups feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size Wendi's flock is 20 chickens?
Answer: In the morning, Wendi gives her chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. In the f final, she gives her chickens 20 * 3 = <20+3=60>60 cups of feed. Thus, in the final meal of the day, Wendi gives her chickens 60 25 15 <60-25-15-10x10 cups of feed. #### 10
Question: Josh decides try flipping a house. buys house $80,000 and then puts in 000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Answer: The value of the house increased 50000 So the value of the house was 80000~78000880000777077776078110 So he made 07000000004000000000000000000000 7500
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract correctly summarizes the main claims made in this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section 6, we discuss limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
In Appendix B we provide the derivation of the claimed FLOPs in Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All hyperparameters and relevant details are provided in Section and Ap- pendix E
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We provide a link to the code in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/Co///oooeeeeeeeeeee C ssisnsolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All the relevant details are provided Section and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: When possible, e.g., for the throughput computation, we provide mean + standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Computation resources are throughput calculation are listed in each table.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttess///eee..e.eeeeeeeeee wwwppppcc//ecciccccoocccccccpcee innes?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
Justification: Discussed in Section 6.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Citations to datasets are used throughout the work. Corresponding licenses are included in Appendix F.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs used to improve writing and for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: : //tp:r//seu.ppeeeeeee cct.c//eeeeeeence//2//////////// for what should or should not be described.
33


Reviews:
[{"review_id": "sZCAPMnawb", "reviewer": "Reviewer_KkBK", "summary": "This paper proposes an encoder-decoder architecture for block discrete diffusion models, which is called E2D2, addressing the doubled computational costs inherent in standard decoder-only approaches. By employing a lightweight decoder that iteratively refines each generated block, the method achieves improved efficiency while maintaining generation quality. Furthermore, the experiments demonstrate the effectiveness of E2D2, showing that E2D2 achieves faster inference speed and lower training costs compared to the standard decoder-only approach.", "questions": "- Regarding weakness 2, how can the optimal settings for the number of decoder layers and block size be determined?\n- In Table 4, the training and decoding speeds of the AR model are superior to those of the block diffusion models. Could the authors elaborate on this result and clarify the primary advantages of block diffusion models over AR models?", "limitations": "The paper does not include a limitations or potential negative societal impact section. I recommend adding a discussion on the performance gap that still exists compared to AR models and the challenge of effectively selecting hyperparameters like the number of decoder layers and block size.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-structured and easy to understand.\n- The proposed model is efficient in training and sampling, enabling faster inference and lower training costs compared to decoder-only block diffusion models.\n- The paper provides a thorough comparison with standard decoder-only block diffusion models, demonstrating that the proposed model matches baseline performance.\n\n**Weaknesses**\n- The placement of tables and figures could be improved. For instance, Table 4 appears after Tables 5 and 6, and its embedding within the text is unconventional.\n- Based on Tables 9 and 10, the model's decoding speed and performance appear sensitive to the number of decoder layers and the block size.\n- The related work section omits a discussion of LLaDA, which is arguably the first large-scale diffusion language model and highly relevant to this work. Furthermore, several other non-autoregressive language models are not discussed, such as SEDD and MDLM.\n- The paper lacks a performance comparison in table with other relevant discrete diffusion models, such as SEDD and MDLM.\n\n**Minor Points:**\n- There are minor typos. For instance, on line 39, 'Contributions' should be removed as it is not part of a sentence or a section title.\n- Some phrasing is informal, such as 'we make three contributions' on line 39.", "was_revised": true, "final_justification": "After rebuttal, the authors address all my concern and i increase my score to 4.", "rebuttal": "We thank the reviewer for their feedback and for recognizing the contribution of E2D2 in improving discrete diffusion models. Below we respond to the reviewer’s concerns and questions.\n\n### **Concern 1:** Missing comparison to models such as MDLM\n\nWe thank the reviewer for this suggestion. Below we provide this comparison on the GSM8k dataset. We find that **MDLM significantly underperforms E2D2 on this task.**\n\n**GSM8k Results**: (Best diffusion values are **bolded**)\n| Model | # Layers | PPL ($\\downarrow$) | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|\n| AR | 28 | 1.52 | 67.6 | 92.22 $\\pm$ 1.36 |\n| MDLM | 28 | 2.34 | 14.33 | 31.35 $\\pm$ 2.45 |\n| BD3LM | 21 | 1.86 | 30.25 | 83.57 $\\pm$ 1.20 |\n| E2D2 | 28 enc / 14 dec | **1.77** | **39.73** | **99.02** $\\pm$ 1.45 |\n\nWe also provide this comparison for models trained **from scratch** on the machine translation (WMT de-en) dataset. Here too, **E2D2 is more efficient and better performing.**\n\n**WMT results:**\n| Model | # Layers | # Params | PPL ($\\downarrow$) | BLEU ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|--|\n| MDLM | 32 | 254M | 6.80 | 20.11 | 55.95 $\\pm$ 8.80  |\n| BD3LM | 12 | 144M  | 7.92 | 23.71 | 81.02 $\\pm$ 1.10 |\n| BD3LM | 16 |166M | 7.36 | 24.11 | 63.73 $\\pm$ 0.66 |\n| E2D2 | 28 enc / 4 dec | 254 M | **6.49** | **25.01**  | **101.21** $\\pm$ 3.91 |\n\nAlthough E2D2 has more parameters than the BD3LM baseline, it is both faster and has better downstream performance, highlighting the benefit of our proposed encoder-decoder structure.\n\n\n**GSM8k Exp. details:**\n- We fine-tuned Qwen 1.7B-Base models on the gsm8k train dataset.\n- Models were fine-tuned with batch size of 1, for 30k steps, which we found to be a more useful setup than the one currently reported in the manuscript. We will update Table 7 with these improved numbers .\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the gsm8k test dataset. We report mean $\\pm$ standard deviation across samples. Note also that the throughput values reported here are superior to those in the original manuscript as we found inefficiencies in our code that we were able to streamline.\n- For MDLM, we generate semi-autoregressively using blocks of 64, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- Note, E2D2 has the same trainable parameters as AR and MDLM since we tie the weights of the models here.\n\n**WMT Exp. details:**\n- We train models with hidden dimension 512 and MLP dimension of 1536 **from scratch**. Other model architecture details follow Qwen3 models.\n- Batch size of 128, LR 3e-4, trained for 1M steps. We perform early stopping using eval loss.\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the test dataset. We report mean $\\pm$ standard deviation across samples.\n- For MDLM, we generate semi-autoregressively using blocks of 32, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- **Note**: We use a modified architecture for E2D2 here: encoder and decoder weights are untied. Rather than sharing KV cache between the encoder and decoder as above, we pass the last hidden representation from the encoder to each layer of the decoder. \n\n---\n\n### **Concern 2:** Explaining the relationship between decoding throughput and number of decoder layers and block size\n\nAs demonstrated in the asymptotic FLOPs computation of Tables 1 and 2, training and inference computational budgets are functions of both the number of layers and the block size. This is true for both E2D2 and BD3LM (and true for all models, e.g., AR and MDLM, in terms of the number of layers).\n\nIn terms of the reviewer’s question on this topic:\n> Regarding weakness 2, how can the optimal settings for the number of decoder layers and block size be determined?\n\nThe “optimal” architecture balances downstream performance and efficiency (i.e. throughput), which can be user-dependent. Models that better control this trade-off are preferable.\n\nBelow, we provide a table showing the Pareto frontier between model performance (measured by PPL and pass@1) and throughput for E2D2 vs. BD3LM models. We find that **for comparable throughput, E2D2 consistently features higher quality.** Thus, it extends forward the Pareto frontier of quality and speed. We obtain best results for high and medium throughput levels, where the gain in quality is especially high.  \n\nTo measure this frontier we vary model size (number of layers) which controls both the performance (larger model $\\rightarrow$ higher performance) and throughput (larger model $\\rightarrow$ lower throughput) axes.\n\n| Model | # Layers | PPL ($\\downarrow$) | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|\n| _High throughput_ |\n| BD3LM | 17 | 2.16 | 15.39 | 99.54 $\\pm$ 1.13 |\n| E2D2 | 28 enc / 14 dec | **1.77** | **39.73** | 99.02 $\\pm$ 1.45 |\n| _Medium throughput_ |\n| BD3LM | 21 | 1.86 | 30.25 | **83.57** $\\pm$ 1.20 |\n| E2D2 | 28 enc / 21 dec | 1.73 | **43.29** | 78.65 $\\pm$ 0.94 |\n| _Low throughput_ |\n| BD3LM | 28 | 1.65 | 44.73 | 65.93 $\\pm$ 0.75 |\n| E2D2 | 28 enc / 24 dec | 1.72 | **46.17** | **71.81** $\\pm$ 0.75 |\n\n(**Exp. details:** Same as above; Best values beyond margin of error are **bolded**)\n\n---\n\n### **Concern 3:** Missing discussion of Llada and predecessor models (MDLM)\n\nWe thank the reviewer for this suggestion, and we will correct this oversight in the updated manuscript. Specifically we edit line 275 in the “Diffusion Language Models” paragraph of the related works as follows:\n\n> Recent large-scale diffusion language models have demonstrated competitive performance with comparably sized AR LLMs on benchmark metrics. Specifically, the Llada model [2], built on the work of  MDLM [3] Nie et al., 2024 [4], scales masked diffusion models to the 8B parameter regime and demonstrated comparable and superior performance to a similarly-sized AR Llama model [5].\n\n---\n\n### **Concern 4:** Missing “limitations” section\n\nThank you for this suggestion. We will add the following text as a limitations section:\n\n> While our work represents a further step in closing the performance and efficiency gap to the dominant AR paradigm for language modeling, our experimental results demonstrate that discrete diffusion models still lag. Further innovation in training and inference algorithms for discrete diffusion models is needed. Additionally, since our work entails language modeling it carries the inherent risks of misuse of this technology. As these models improve in size and quality, care must be taken to prevent malicious use.\n\n---\n\n### **Additional concerns:** Table placement and typos\n\nThank you for these suggestions. We will update the formatting and text in our revised manuscript.\n\n---\n\n### **Question:** Clarify the primary advantages of diffusion relative to AR, which has faster throughput\n\nAs noted in the updated table above, we have removed an important inefficiency in our code which brings the throughput of the diffusion models in line with that of the AR baseline. Moveover, **we see the promise of E2D2 which is able to have the best diffusion performance with faster throughput than AR models**. We apologize that these results were not reflected in our original manuscript, but we will update them in our revised version. \n\n---\n\n**References**\n\n[1] Arriola, Marianne, et al. “Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models.” ICLR 2025\n\n[2] Nie, Shen, et al. \"Large language diffusion models.\" arXiv preprint arXiv:2502.09992 (2025).\n\n[3] Sahoo, Subham, et al. \"Simple and effective masked diffusion language models.\" Advances in Neural Information Processing Systems 37 (2024): 130136-130184.\n\n[4] Nie, Shen, et al. \"Scaling up masked diffusion models on text.\" arXiv preprint arXiv:2410.18514 (2024).\n\n[5] Dubey, Abhimanyu, et al. \"The llama 3 herd of models.\" arXiv e-prints (2024): arXiv-2407."}, {"review_id": "arZZfYLsEY", "reviewer": "Reviewer_4wvn", "summary": "The authors introduce E2D2, an encoder-decoder method for discrete diffusion. The method uses the encoder to create representations of the input that the decoder cross-attends to the Key and Value outputs of the input. The authors then train 3 domain specific models for summarization (CNN/DailyMail), translation (WMT), and math reasoning (GSM8K). Their experiments show a performance increase with E2D2 to a modified BD3LM.", "questions": "Can you share the results of an unmodified BD3LM on the tasks presented in the paper? As well as the inference and training throughput? \n\nHow is the AR model trained / what does the model size look like? Did you inference match / train match the AR model? \n\nAre the experiments in 4.4 fine tuned individually (i.e., for each parameter) or the results with the same base model?", "limitations": "I have no suggestions on discussing any negative societal impact of this particular work", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe application of Encoder-Decoder models to Diffusion LMs is interesting. \n\nThe ablation experiments are interesting showing the tradeoffs between performance and throughput of the E2D2 method. \n\n\nWeakness\nThe comparisons to BD3LM could be expanded. Currently the comparison is “apples-to-apples” wrt compute, but it would also be valuable to see the performance differences and efficiency of an unmodified model. The authors refer to a “Performance/Throughput Pareto frontier”, but it is not then obvious where E2D2 sits compared to other existing models when provided larger compute budgets.\n\nExtra result:\nI am unclear on the exact evaluation setting of these experiments (see the questions section), but the block size experiments could be potentially very interesting. It would be interesting to also see the summarization and translation experiments for this setting.", "was_revised": true, "final_justification": "The author responses alleviate concerns and increase clarity. The Pareto results added during the response makes further clear the benefits of this work. I have increased my score accordingly.", "rebuttal": "We thank the reviewer for their feedback and for the valuable additional experiment suggestions. Below we respond to the specific comments inline.\n\n---\n\n### **Concern 1:** Provide a more complete “pareto frontier” picture\n\nPer the reviewer request, below, we provide a table showing the Pareto frontier between model performance (measured by PPL and pass@1) and throughput for E2D2 vs. BD3LM models. We find that **for comparable throughput, E2D2 consistently features higher quality.** Thus, it extends forward the Pareto frontier of quality and speed. We obtain best results for high and medium throughput levels, where the gain in quality is especially high.  \n\nTo measure this frontier we vary model size (number of layers) which controls both the performance (larger model $\\rightarrow$ higher performance) and throughput (larger model $\\rightarrow$ lower throughput) axes.\n\n(Note: This table also address the reviewer’s request for presenting the  “unmodified” BD3LM, i.e., the 28 layer BD3LM).\n\n| Model | # Layers | PPL ($\\downarrow$) | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|\n| _High throughput_ |\n| BD3LM | 17 | 2.16 | 15.39 | 99.54 $\\pm$ 1.13 |\n| E2D2 | 28 enc / 14 dec | 1.77 | **39.73** | 99.02 $\\pm$ 1.45 |\n| _Medium throughput_ |\n| BD3LM | 21 | 1.86 | 30.25 | **83.57** $\\pm$ 1.20 |\n| E2D2 | 28 enc / 21 dec | 1.73 | **43.29** | 78.65 $\\pm$ 0.94 |\n| _Low throughput_ |\n| BD3LM | 28 | 1.65 | 44.73 | 65.93 $\\pm$ 0.75 |\n| E2D2 | 28 enc / 24 dec | 1.72 | **46.17** | **71.81** $\\pm$ 0.75 |\n\n**Exp. details:**\n- Best values beyond margin of error are **bolded**\n- We fine-tuned Qwen 1.7B-Base models on the gsm8k train dataset.\n- Models were fine-tuned with batch size of 1, for 30k steps, which we found to be a more useful setup than the one currently reported in the manuscript. We will update Table 7 with these[ improved numbers .\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the gsm8k test dataset. We report mean $\\pm$ standard deviation across samples. Note also that the throughput values reported here are superior to those in the original manuscript as we found inefficiencies in our code that we were able to streamline.\n- For MDLM, we generate semi-autoregressively using blocks of 64, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- Note, E2D2 has the same trainable parameters as AR and MDLM since we tie the weights of the models here.\n\n---\n\n### **Concern 2:** Explore the effect of block size more\n\nBelow we present the exploration of the effect of block size for E2D2 models. We find that E2D2 with block size of either 4 or 8 better trades-off benchmark performance with throughput.\n\n| Block size | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|\n| 4 |  39.73 | 99.02 $\\pm$ 1.45 |\n| 8 |  34.42 | 108.58 $\\pm$ 1.75 |\n| 16 |  30.48 | 103.70 $\\pm$ 1.48 |\n\nNote that all models above were fine-tuned as separate runs (see the answer to the related question below). This is an updated version of Table 10 from our paper which we will revise in our latest manuscript.\n\n**Exp details**: Same as above.\n\n---\n\n### **Question 1:** How is the AR model trained / what does the model size look like?\n\nAll of the models are based on the same backbone and are hence comparable. The AR model in particular represents an unmodified backbone, that is we use all 28 layers of the pre-trained model. This is akin to matching the number of encoder layers of E2D2. This renders the AR model a very strong baseline. We will add this information more explicitly in our revised manuscript\n\n---\n\n### **Question 2:** Are the experiments in 4.4 fine tuned individually (i.e., for each parameter) or the results with the same base model?\n\nWe apologize for the confusion. In our updated manuscript we will make it clear that each result in the ablations section (4.4) for both Table 9 and Table 10 represents a separate fine-tuning run. That is, for each row in these tables, starting from the same baseline pre-trained model, we initialize weights from the pre-trained model, modify the number or layers (Table 9) or block size (Table 10), and then fine-tune. This holds true for the updated results presented in this response as well, which we will use when revising our manuscript."}, {"review_id": "UwCWIEGwJe", "reviewer": "Reviewer_6TKj", "summary": "This paper introduces E2D2, an extension to block diffusion language models that decouples the size of the decoder from the encoder, improving performance for fixed training/inference speeds. The core idea in the paper is that the encoder context of block diffusion models—corresponding to the cross attention of the diffusing block and prior context—can be differentiated from the decoding model that generates the diffusion predictions. This equates to an encoder-decoder framework and opens up a number of avenues for exploration. Specifically, E2D2 proposes removing a number of the layers closer to the input so that the decoder is smaller than the encoder. This has the effect of either improving the cost of training as the noised examples go through fewer layers of the architecture. Alternatively, the model size can be modified to keep inference costs the same allowing for a larger model to create the KV cache with amortized cost followed by repeated steps of the diffusion model. The authors then explore a number of design decisions for the E2D2 models including weight-tying and fine-tuning datasets, before providing analyzing the results of the E2D2 framework on a variety of benchmarks such as ROUGE scores, WMT (de-en) BLEU scores, and throughput-comparisons highlighting the benefits of the E2D2 framework.", "questions": "Questions:\n1) Is the use of a KV-cache novel to this work? It is my understanding the the BD3LM paper uses a clean input processed to also generate a KV cache for further decoding. Could you expand on the differences between the methods?\n2) Related to the first limitation, but could the smaller diffusion models make more sense the larger the encoder network is?\n3) How were the “inference-matched” and “train-matched” baselines chosen? The pruning performed by choosing the first-N layers of a model seems unclear if that would be the most performant result to explore?\n\nNit:\n1) There is some inconsistencies in the naming for the Tables, e.g. (“inf_match” vs. “inference-matched”, consistency would be great).\n2) Including the direction of improvement for Figures and Tables would be helpful for readability\n3) It is worth highlighting in the main body of the paper that a block size of 4 was used for most experiments.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper highlights a core attribute of the block diffusion language model that size of the encoder and decoder can be decoupled. This allows for greater flexibility in terms of model design, in this case in terms of the number of layers of the diffusion model.\n2) There are a number of interesting ablations that are helpful for practitioners aiming to use E2D2 and resulting frameworks. For example, the tied weights is promising for limiting the memory constraints of the resulting model. The throughput comparisons may also help researchers choose what size of E2D2 model to use.\n\nLimitations:\n1) While enabling a new paradigm for decoupling the encoder-and-decoder components of the language model, there is a significant amount of flexibility in the distinctions between the two models that remains unexplored in this paper. A number of immediate ablations, such as how many layers are necessary in the decoder network are missing, limiting the conclusions that can be drawn. The paper could benefit from fully exploring the Pareto frontier of decoder-size, as it appears that the decoder could even be shrunk further. While there is some discussion of this in the appendix (with the number of decoder layers), the choice of n/2 in the main body of the paper appears limited and does not explore the range of possible options. For example, another option would be to maintain depth but shrink the model size of the decoder, use grouped attention / similar inference optimizations / alternatively modify the decoder in a way that decouples the behavior. Another direction for exploration is whether the size of the decoder affects the number of steps that the diffusion model needs to perform for a fixed target perplexity.\n2) The paper also mostly focuses on fine-tuning datasets, which is interesting in itself. However, the limited results from “From scratch” training draw concern to if the method leads to training instability or similar that would prevent training a diffusion language model from scratch with this method.", "was_revised": true, "final_justification": "I would upgrade my recommended score to a 4. Though the original paper was missing a number of key results surrounding the pareto frontier of results, I believe that the results provided in the rebuttal (when included in the final version of the paper) will make for a good paper and incite discussion into the encoder-decoder framing of block discrete diffusion models.\n\nAdditional points considered and answered:\n1) The added point about the design of the decoder framework (for example, the 4 layer decoder presented in the rebuttal or using the last layer hidden representations) helps complete the discussion around network design.\n2) The WMT experiment also helps portray a complete story of training from scratch and shows that network design is not conflated by other sources of training instability.", "rebuttal": "We thank the reviewer for their valuable feedback and for recognizing the contributions of our work. Below we respond to the specific concerns and questions\n\n---\n\n### **Concern 1:** Provide a more complete picture of the “pareto frontier”\n\nBelow, we provide a table showing the Pareto frontier between model performance (measured by PPL and pass@1) and throughput for E2D2 vs. BD3LM models. We find that **for comparable throughput, E2D2 consistently features higher quality.** Thus, it extends forward the Pareto frontier of quality and speed. We obtain best results for high and medium throughput levels, where the gain in quality is especially high.  \n\nTo measure this frontier we vary model size (number of layers) which controls both the performance (larger model $\\rightarrow$ higher performance) and throughput (larger model $\\rightarrow$ lower throughput) axes.\n\n| Model | # Layers | PPL ($\\downarrow$) | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|\n| _High throughput_ |\n| BD3LM | 17 | 2.16 | 15.39 | 99.54 $\\pm$ 1.13 |\n| E2D2 | 28 enc / 14 dec | 1.77 | **39.73** | 99.02 $\\pm$ 1.45 |\n| _Medium throughput_ |\n| BD3LM | 21 | 1.86 | 30.25 | **83.57** $\\pm$ 1.20 |\n| E2D2 | 28 enc / 21 dec | 1.73 | **43.29** | 78.65 $\\pm$ 0.94 |\n| _Low throughput_ |\n| BD3LM | 28 | 1.65 | 44.73 | 65.93 $\\pm$ 0.75 |\n| E2D2 | 28 enc / 24 dec | 1.72 | **46.17** | **71.81** $\\pm$ 0.75 |\n\n**Exp. details:**\n- Best values beyond margin of error are **bolded**\n- We fine-tuned Qwen 1.7B-Base models on the gsm8k train dataset.\n- Models were fine-tuned with batch size of 1, for 30k steps, which we found to be a more useful setup than the one currently reported in the manuscript. We will update Table 7 with these[ improved numbers .\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the gsm8k test dataset. We report mean $\\pm$ standard deviation across samples. Note also that the throughput values reported here are superior to those in the original manuscript as we found inefficiencies in our code that we were able to streamline.\n- For MDLM, we generate semi-autoregressively using blocks of 64, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- Note, E2D2 has the same trainable parameters as AR and MDLM since we tie the weights of the models here.\n\n---\n\n### **Concern 2:** Explore other design choices for decoupling Encoder-Decoder architecture design choices are underexplored\n\nHere, we explore an alternative way of connecting the encoder and decoder: rather than sharing KV cache for matched layers in the encoder decoder, we feed the last hidden layer representation of the encoder into the decoder. We conducted an experiment for this new model by training from scratch on the machine translation (WMT14 de-en) dataset. In this setup, we also find that **E2D2 is more efficient and better performing** compared to MDLM and BD3LM.\n\n**WMT results:**\n| Model | # Layers | # Params | PPL ($\\downarrow$) | BLEU ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|--|\n| MDLM | 32 | 254M | 6.80 | 20.11 | 55.95 $\\pm$ 8.80  |\n| BD3LM | 12 | 144M  | 7.92 | 23.71 | 81.02 $\\pm$ 1.10 |\n| BD3LM | 16 |166M | 7.36 | 24.11 | 63.73 $\\pm$ 0.66 |\n| E2D2 | 28 enc / 4 dec | 254 M | **6.49** | **25.01**  | **101.21** $\\pm$ 3.91 |\n\nAlthough E2D2 has more parameters than the BD3LM baseline, it is both faster and has better downstream performance, highlighting the benefit of our proposed encoder-decoder structure.\n\n**WMT Exp. details:**\n- We train models with hidden dimension 512 and MLP dimension of 1536 **from scratch**. Other model architecture details follow Qwen3 models.\n- Batch size of 128, LR 3e-4, trained for 1M steps. We perform early stopping using eval loss.\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the test dataset. We report mean $\\pm$ standard deviation across samples.\n- For MDLM, we generate semi-autoregressively using blocks of 32, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- **Note**: We use a modified architecture for E2D2 here: encoder and decoder weights are untied. Rather than sharing KV cache between the encoder and decoder as above, we pass the last hidden representation from the encoder to each layer of the decoder. \n\n\nNext, we address this comment from the reviewer:\n> Another direction for exploration is whether the size of the decoder affects the number of steps that the diffusion model needs to perform for a fixed target perplexity.\n\nBelow we present an experiment that compares varying the number of diffusion steps per block for E2D2, across different decoder sizes. The efficiency of decoupling encoder and decoder compute in E2D2 allows us to use larger models with fewer decoding steps and still outperform BD3LM models (that use more diffusion steps) in terms of both the throughput and benchmark performance axes.\n\n| Model   | Layers   |   T | Inference Throughput (Tok/sec $\\uparrow$)   |   0-shot pass @1 ($\\uparrow$) |\n|---|---|---|---|---|\n| BD3LM   | 17 |   4 | 99.54 $\\pm$ 1.13 | 15.39 |\n| BD3LM   | 21|   4 | 83.57 $\\pm$ 1.20 | 29.42 |\n| E2D2    | 28 enc / 14 dec |   2 | 152.05 $\\pm$ 3.24| 30.17 |\n| E2D2    | 28 enc / 21 dec |   2 | 126.57 $\\pm$ 2.13| 32.83 |\n| E2D2    | 28 enc / 24 dec |   2 | 119.50 $\\pm$ 1.64| 35.33 |\n\n(**Exp. details:** Same as above)\n\n---\n\n### **Concern 3:** Add a “from scratch” experiment\n\nThis comment is well received. The WMT experiment from above was indeed conducted from scratch and demonstrates that our approach is not reliant on initializing from pre trained models.\n\n---\n\n### **Formatting concerns:**\n\nThank you for these comments. We will update the revised manuscript accordingly.\n\n---\n\n### **Question 1:** “Is the use of a KV-cache novel to this work?”\n\nTo clarify, both E2D2 and BD3LM enable KV-caching of already decoded blocks and this is one of the primary motivations of building E2D2 on BD3LM models as opposed to their predecessor vanilla MDLM models, which do not support KV caching (in addition to the fact that BD3LM boasts improved performance vs. MDLM). The difference between these models is that to decode a given token one needs to make a pass through all the layers of the BD3LM model, whereas for E2D2 we only require a forward pass through the smaller number of decoder layers.\n\nThus, for the sake of illustration, assume that the E2D2 encoder is the same size as the full BD3LM model, then the KV cache for both BD3LM and E2D2 would be the same: cache any context/prompt tokens and, as each new block is generated, add it to the cache (by running it through the full network for BD3LM or through the encoder for E2D2). For E2D2, the decoder uses the KV cache from the encoder as well.\n\n---\n\n### **Question 2:** “Could the smaller diffusion models make more sense the larger the encoder network is?”\n\nGreat question. We explore this above in the response to concern 1.\n\n---\n\n### **Question 3:** “How were the ‘inference-matched’ and ‘train-matched’ baselines chosen?” \n\nThese baselines were chosen based on the FLOPs computations provided in Tables 1 and 2.\n- **Train-matched:** To match the training flops between E2D2 and BD3LM, the BD3LM model needs to have half as many layers as the combined number of encoder and decoder layers of the E2D2 model. See Table 1.\n- **Inference-matched:** We matched the inference compute empirically by seeing the number of layers of a BD3LM model that would lead to similar throughput to an E2D2 model (assuming we only decode one token per forward pass of the diffusion model, i.e. diffusion time steps $T$ per block is equal to block size $S$). See the throughput results from the response to Concern 1 above."}, {"review_id": "5cO3XWuXo7", "reviewer": "Reviewer_xEvo", "summary": "This paper introduces E2D2, a novel language model that generates text by sequentially creating \"blocks\" and filling each block in parallel using a diffusion process. This hybrid approach allows the model to achieve faster inference speeds than traditional methods while striking a better balance between text quality and computational cost.", "questions": "1. Block diffusion models inherently have the potential to reuse KV-cache in multi-turn dialogues. However, E2D2 introduces an encoder that must process the entire history, which seems to break the ability to efficiently reuse KV-cache in long conversations. It will be better to  provide a detailed explanation of E2D2's scalability in multi-turn, long-dialogue scenarios and compare it to standard block diffusion models.\n\n2. None of the results tables in the paper clearly denote which specific model size was used (0.6B or 1.7B). For the sake of reproducibility and fair assessment, it is important to provide this information for all tables.\n\n3. The authors may need to  provide a thorough analysis explaining why their model's performance on GSM-8K dropped by over 40 percentage points compared to its powerful pre-trained base model. Is it using diiferent evaluation settings?\n\nIf the authors can solve my questions, I may consider to raise my score.", "limitations": "See weakness and questions.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The core strength is the proposal of E2D2, which addresses the weaknesses between autoregressive models and non-autoregressive models like standard diffusion. The hybrid design retains the sequential logic of AR models at a macro level between blocks while harnessing the parallel efficiency of diffusion at a micro level within each block.\n2. The paper provides experiments on different benchmarks to prove the dominant performance, the strong throughput and the execellent generalization.\n\nWeakness:\n1. The core contribution is not clear enough. The proposed method appears to be merely a straightforward combination of block diffusion like BD3-LMs[1] and the encoder-decoder architecture like T5[2], rendering the limited contribution.\n2. The paper lacks the necessary discussion of some related works. For example, the authors failed to cite or compare their work with a highly relevant NLP paper on multi-token prediction[3], next block generation[4]. \n3. Though the authors state their model is fine-tuned from Qwen3-1.7B. It shows that the original Qwen3-1.7B base model achieves an accuracy of 75.44% on the GSM-8K benchmark. However, E2D2 only reports an accuracy of 33.89% in Table 7. Given that the base model's training set already includes GSM-8K, such a performance drop is  incomprehensibl.\n4. In Table 5, E2D2 shows only a little advantage over its direct counterpart, the BD3-LM baseline, which weakens the value of its contribution.", "was_revised": true, "final_justification": "4: Borderline accept", "rebuttal": "We thank the reviewer for their feedback. Please see below for responses to the specific criticisms and questions raised.\n\n---\n\n### **Concern 1:** Limited novelty\n\nWe argue that our work has novelty beyond combining a T5-style architecture with BD3-LM. Specifically, it involves the following research challenges:\n- **Architecture Design.** There are multiple ways to define encoder-decoder inputs and outputs. See e.g., ENAT [5] for an alternative architecture optimized for images. More generally, we report ablations on ways of connecting the encoder and decoder, the interplay of block size and decoder size, choices of decoder optimizations, weight-tying, and more.  \n- **Training Algorithms.** Our architecture requires specialized training algorithms that extend BD3-LM (Algorithm 1), which rely on custom attention masks relative to BD3LM (Figure 2) and a modified computation graph (Figure 1) relative to T5.\n- **Sampling Algorithms.** Similarly, we introduce sampling algorithms that extend BD3-LM (Algorithm 2) and provide hyper-parameters (e.g., block size, number of layers) needed to obtain good performance.\n\n---\n\n### **Concern 2:** Clarifying why fine-tuned models have drop in performance relative to base model\n\nThere are several factors at play here. First, although E2D2 is within 1 perplexity point of the base Qwen model [1], this gap translates to a larger drop in GSM8K exact-match accuracy which is sensitive to small mistakes in the arithmetic trace. Qualitatively, we observe that even single-token mismatches may result in an incorrect response. Additionally, the gap between even the fine-tuned AR model and the value reported in the original Qwen3 technical report [1] stems from the different evaluation setup, as detailed below.\n\n\n**GSM8k Results**: (Best diffusion values are **bolded**)\n\n|     | Fine-tuned AR | E2D2 (Ours) |\n|---------------|---------|---------|\n| PPL ($\\downarrow$) | 1.52    | 67.6 |\n| pass @ 1 accuracy  ($\\uparrow$)  | 1.77  | 39.73 |\n\n(Note: the E2D2 values here represent an improved fine-tuning setup with smaller batch size, which we will update in our revised manuscript).\n \nIn terms of the differences in evaluation setup:\n- The original Qwen model used 4-shot CoT reasoning traces. Our evaluation is 0-shot.\n- Additionally, we fine-tuned the model to just output “reasoning traces” provided in the GSM8k dataset. This differs significantly from the default output that the Qwen model produces which is much more verbose. We limited our scope of number of shots and reasoning traces for computational considerations. \n\nWe also comment that this gap is limited to the GSM8k dataset. On the translation and summarization tasks, however, E2D2 _beats_ the performance of the fine-tuned AR model (Tables 5 & 6):\n\n|     | Fine-tuned AR | E2D2 (Ours) |\n|---------------|---------|---------|\n| WMT BLEU ($\\uparrow$) |  21.45    | 21.51 |\n| CNN/DM ROUGE-1 ($\\uparrow$) | 26.4  | 36.8 |\n\n---\n\n### **Concern 3:** Limited improvement of E2D2 vs. BD3LM\n\nWhile performance relative to BD3LM is comparable in Table 5, we note that we achieve this at either improved efficiency (relative to the train-matched BD3LM) or no additional cost (relative to the inference matched BD3LM).\n\nTo make this tradeoff of efficiency and performance more concrete, we provide a table showing the Pareto frontier between model performance (measured by PPL and pass@1) and throughput for E2D2 vs. BD3LM models. We find that **for comparable throughput, E2D2 consistently features higher quality.** Thus, it extends forward the Pareto frontier of quality and speed. We obtain best results for high and medium throughput levels, where the gain in quality is especially high.  \n\nTo measure this frontier we vary model size (number of layers) which controls both the performance (larger model $\\rightarrow$ higher performance) and throughput (larger model $\\rightarrow$ lower throughput) axes.\n\n| Model | # Layers | PPL ($\\downarrow$) | 0-shot pass @1 ($\\uparrow$) | Inference Throughput (Tok/sec $\\uparrow$) |\n|--|--|--|--|--|\n| _High throughput_ |\n| BD3LM | 17 | 2.16 | 15.39 | 99.54 $\\pm$ 1.13 |\n| E2D2 | 28 enc / 14 dec | **1.77** | **39.73** | 99.02 $\\pm$ 1.45 |\n| _Medium throughput_ |\n| BD3LM | 21 | 1.86 | 30.25 | **83.57** $\\pm$ 1.20 |\n| E2D2 | 28 enc / 21 dec | 1.73 | **43.29** | 78.65 $\\pm$ 0.94 |\n| _Low throughput_ |\n| BD3LM | 28 | 1.65 | 44.73 | 65.93 $\\pm$ 0.75 |\n| E2D2 | 28 enc / 24 dec | 1.72 | **46.17** | **71.81** $\\pm$ 0.75 |\n\n**Exp. details:**\n- We fine-tuned Qwen 1.7B-Base models on the gsm8k train dataset.\n- Models were fine-tuned with batch size of 1, for 30k steps, which we found to be a more useful setup than the one currently reported in the manuscript. We will update Table 7 with these[ improved numbers .\n- Throughput calculation was done using batch size of 1 on a single H100 GPU. Models generated 256 new tokens for 50 prompts from the gsm8k test dataset. We report mean $\\pm$ standard deviation across samples. Note also that the throughput values reported here are superior to those in the original manuscript as we found inefficiencies in our code that we were able to streamline.\n- For MDLM, we generate semi-autoregressively using blocks of 64, to have a fair comparison to the other models. We did a sweep on block sizes from 4 to 256 for MDLM to find the best performing one.\n- Note, E2D2 has the same trainable parameters as AR and MDLM since we tie the weights of the models here.\n\n---\n\n### **Concern 4:** Missing related works\n\nThank you for these suggested related works. We are happy to include them in our revised manuscript. Unfortunately, the exact references for “[3]” and “[4]” suggested by the reviewer appear to have been cut off in the OpenReview UI. We therefore present discussions for what we assume are the works to which the reviewer was referring. Please let us know if we are not discussing the ones you had in mind:\n\n**“multi-token prediction [3]”**: _Gloeckle, Fabian, et al. \"Better & faster large language models via multi-token prediction.\"  (2024)._\n\nOur work differs from [3] in several important ways. First, for their non-parallel generation, within a block, [3] maintains an autoregressive decoding scheme. In contrast, our diffusion parameterization within each block enables any-order generation. Additionally, similar to our contrast with works, such as Medusa [5], our architecture represents a more expressive decoder relative to the “heads” used in [3] and [5] and E2D2 has  more sophisticated information sharing between the encoder and the decoder via cross-attention to the encoder outputs.\n\n**“next block generation [4]”** _Ho, Namgyu, et al. \"Block transformer: Global-to-local language modeling for fast inference.\" (2024)_\n\nThe key difference between E2D2 and this work are as follows: first [4] is still parameterized as an AR model, whereas E2D2 uses a block diffusion parameterization. Additionally, E2D2 employs standard global, self-attention whereas the efficiencies from [4] come from a custom hierarchical transformer model that first pools tokens to create block embeddings over which global attention is applied and then performs only local attention within blocks to decode autoregressively. These efficiencies are orthogonal to our proposed method and in fact could potentially be combined with E2D2 by replacing the standard transformer with the Block Transformer, to reduce KV cache memory storage and retrieval bottlenecks.\n\n---\n\n### **Question 1:** Can E2D2’s encoder support efficient KV caching\n\nWe clarify that E2D2 also supports efficient caching. Both the encoder and decoder use a KV cache of the already decoded / clean context blocks. More specifically, in E2D2, the encoder first processes clean context tokens. These are stored in a KV cache for the encoder (which the decoder also uses). Each newly decoded block is added to this KV cache in the same manner they would for BD3LM, except in E2D2, this is done by processing them in the encoder.\n\nThus KV-cache is not a bottleneck for E2D2 models in scaling to multi-turn, long-dialogue settings.\n\n---\n\n### **Question 2:** Clarifying the model sizes used in each experiment\n\nThank you for the suggestion. This information (reproduced in the table below) is included in the appendix, but to improve readability we will add it to the main text and corresponding tables as well.\n\n| Experiment | Base model size |\n|---|---|\n| Machine translation (WMT) | Qwen/Qwen3-0.6B-Base |\n| Summarization (CNN-DailyMail) | Qwen/Qwen3-0.6B-Base |\n| Math reasoning (GSM8k) | Qwen/Qwen3-1.7B-Base |\n\n\n---\n\n**References**:\n\n[3]  Gloeckle, Fabian, et al. \"Better & faster large language models via multi-token prediction.\" arXiv preprint arXiv:2404.19737 (2024).\n\n[4] Ho, Namgyu, et al. \"Block transformer: Global-to-local language modeling for fast inference.\" Advances in Neural Information Processing Systems 37 (2024): 48740-48783.\n\n[5] Cai, Tianle, et al. \"Medusa: Simple llm inference acceleration framework with multiple decoding heads.\" arXiv preprint arXiv:2401.10774 (2024)."}]

Author rebuttal:

Meta-review: The paper introduces E2D2, an encoder–decoder discrete diffusion model that reduces the overhead of blockwise diffusion while maintaining or improving quality. It demonstrates strong efficiency–quality trade-offs across summarization, translation, and reasoning, marking a clear advancement over prior discrete diffusion approaches.
After rebuttal, I think the authors have addressed all the concerns of reviewers. And Reviewer xEvo said that she/he acknowledges the rebuttal and would like to raise the score to 4, so it seems all scores should be positive.
Decision: Accept (poster)