Soft-Masked Diffusion Language Models
Authors: Michael Hersche
Samuel Moor-Smith
Thomas Hofmann
Abbas Rahimi
Venue: ICLR 2026 Poster (2026)
Keywords: Masked diffusion language models
continuous feedback
code generation
Primary area: generative models
TL;DR: We present soft-masking, a new method that improves masked diffusion language models by blending mask tokens with predictions from previous iterations to better preserve context.

Abstract: Diffusion models have demonstrated strong potential in language modeling, offering various advantages over traditional autoregressive approaches. Their ability to generate and revise entire responses in parallel enables faster generation and built-in self-correction mechanisms. Most modern diffusion-based language models employ masked diffusion, where decoding involves iteratively processing masked tokens based on a binary decision: either retaining the mask or replacing it with the predicted token. However, this binary choice discards valuable predictive information when the mask is retained. To address this limitation, we introduce soft-masking (SM), a novel method that dynamically blends the embedding of the mask token with the embeddings of the top-k predicted tokens from the previous decoding step, for each retained mask. This provides the model with a more informative prior, preserving context from earlier computations and allowing partial information about masked tokens to propagate beyond a single step. We propose a training methodology that efficiently adapts masked diffusion language models to incorporate SM. We demonstrate that training a 169M parameter model from scratch with SM yields superior perplexity and MAUVE scores compared to binary masking baselines. Similarly, a pretrained model can be enhanced with SM through continued pretraining. Finally, we finetune two state-of-the-art diffusion models, Dream-7B and Dream-Coder-7B, with SM. SM consistently improves performance across multiple coding benchmarks, particularly in high-throughput settings.

Published as a conference paper at ICLR 2026
SOFT-MASHE DIFFUSION LANGUAGE MODELS
Michael Hersche', Samuel Moor-Smith 1.2,, Thomas Hofmann', Abbas Rahimi
1 IBM Research - -urrchh, "Department of Computer Science, ETH Zürich
ABSTRACT
Diffusion models have demonstrated strong potential in language modeling, of- fering various advantages over traditional autoregressive approaches. Their abil- ity to generate and revise entire responses parallel enables faster generation and built-in self-correction mechanisms. Most modern diffusion-based language models employ masked diffusion, where decoding involves iteratively processing masked tokens based on a binary decision: either retaining the mask replacing it with the predicted token. However, this binary choice discards valuable predictive information when the mask is retained. To address this limitation, soft-masking (SM), a novel method that dynamically blends the embedding of the mask token with the embeddings of the top-k predicted tokens from the previous decoding step, for each retained mask. This provides the model with a more in- formative prior, preserving context from earlier computations and allowing partial information about masked tokens to propagate beyond a single step. We propose training methodology that efficiently adapts masked diffusion language models to incorporate SM. We demonstrate that training 169M parameter model from scratch with SM yields superior perplexity and MAUVE scores compared to bi- nary masking baselines. Similarly, pretrained model can be enhanced with SM through continued pretraining. Finally, we finetune two state-of-the-arr diffusion models, Dream-7B and Pream-Coder-Pe with SM. SM consistently improves performance across multiple coding benchmarks, particularly in tigh-throughput settings
introduce
(a)
Decoding with binary masking
(b) This work: Decoding with soft-masking
Prompt
add
a,
def
ade
b
F
Vn
edd
b
Decoding
iterations
def
addi
rel
add
add
F:
\n
add
FMl
def
[dd||
0,
\n
ref
umm
Response
[ddi
Figure Illustrative answer generation using masked diffusion language models (MDLMs) via iterative decoding with (a) standard binary masking or (b) our proposed soft-masking. Our soft- masking enriches the feedback for the next decoding step by superposing the masked tokens with the previously predicted top-k candidates, enabling more accurate and faster generation.
INTRODUCTION
Generative large language models (LLMs) have transformed natural language processing. LLMs typically operate in an autoregressive (AR) mode (Vaswani et al.) [2177] Brown et al. 2020), where the next token in sequence predicted based on the previously generated tokens. While AR mod- els have proven highly effective, their sequential nature makes inference computationally expensive,
"Rescarch conducted at IBM Research - Zurich. Code httphttpp... htttpa///gittbb
in lagguugg-oo-

---

Published as a conference paper at ICLR 2026
leading to high latency and costs. These inference costs particularly pronounced in large rea- soning models (OpenAI, 2024; Guo et al. 2025 Qwen Team 2024). where different solutions are validated through sequential exploration via chain-of-thouge (CoT) (Wei et al. 2022).
As a potential remedy, recent work has shown that incorporating continuous feedback, rather than relying solely discrete, sampled tokens, can improve AR model's performance (Hao et al. 2025 Zhuang et al.] 2025; ZZann et a..). Such continuous feedback encodes multiple potential solutions in superposition (Zhu et al.) 2025b), enabling simultaneous exploration of diverse paths and thereby potentially reducing the number of generated tokens. However, training AR models with continuous feedback is slow, due to the sequential reliance on previous continuous token outputs.
As a promising alternative, diffusion models ooriinally developed for continuous domains in vi- sion (Sohl-Dickstein et al. 2015 Ho et al. 22000) Song et al. have recently been adapted to natural language processing. These diffusion language models (DLMs) offer key advantages over AR models, including accelerated sampling (Inception al. (025), controllable generation (Li) et al. (022), bidirectional modeling, and built-in self-correction (Ye et al. 2024). Besides being more data-efficient than AR-models in training (Ni et al. 2025 Prabhudesai et al. 2025), they are particularly beneficial in non-causal tasks such as coding (Nie et 20.5| Gong et al. [202bb] Xie
ct al. 2025). They consist of a forward process, which gradually corrupts data, and a backward process, which iteratively reverses this corruption to generate coherent outputs.
Masked DLMs (MDLMs) have emerged as the most scalable and effective approach. They imple- ment the forward process as categorical transition function, mapping tokens an absorption state, typically represented by a [MASK] token (Austin et al., 2021a; Campbell et al., 2022] Lou et al.
2024 Sahoo et al.] 2024 Ou al.) 2025 Shi al., (024). During decoding, the model makes binary choice for each mask token: either replace it with a predicted token or retain the [MASK] (see Figure [p) This discrete formulation allows for improved training and has enabled the development
Ye et al. 2025 Xie et al. 2025) and commercial (Inception et al., Deep.ind 2025) initiatives.
of large-scale MDLMs across both open-source (Gong et al. 22025a Nie et al. 2025
Despite their scalability, MDLMs are fundamentally constrained by the binary unmasking process, which discards valuable predictive information. Likewise, AR models commit to a single discrete sampling decision, with no opportunity for refinement. Motivated by the success of continuous feed- back mechanisms in AR models, which preserve and leverage uncertainty over multiple candidates, we propose new feedback mechanism for MDLMs that propagates this rich predictive information throughout the generation process.
This work: Continuous feedback in MDLMs via soft-masking We introduce soft- masking (SM), a simple yet effective mechanism for incorporating continuous feedback into MDLMs, as illustrated in Figure Dp. During decoding, SM enriches the [MASK] state with a con- vex combination of the top-k predicted tokens, weighted by their confidence scores. This allows the model to retain and propagate partial information across decoding steps, rather than discard- ing it through binary masking decisions. Our method integrates seamlessly into existing MDLM architectures, requiring minimal adaptation. Our contributions are as follows:
We propose loft-masking (SM), a novel decoding mechanism that enhances the expressive- ness of the [MASK] token in MDLMs. SM only adds three additional parameters, which can be efficiently learned together with the MDLM parameters using a parallelizable training procedure that enables MDLMs to leverage the richer feedback.
We show that (69M-parameter MDLM with SM trained OpenWebText can enhance both validation perplexity and MAUVE scores in unconstrained generation. Further, SM can be integrated into a pretrained MDLM via pretraining continuation for only 100k steps.
We demonstrate that SM generalizes to large-scal MDLMs by applying to Dream- 7B (Ye et al.) (025) and Cream-CCdee-7B (Xie et al.) 2025). After minimal finetuning, SM improves accuracy on HumanEval and MBPP code generation benchmarks (including plus versions), particularly in high-throughput regimes with limited decoding iterations.
SM can be readily integrated into other MDLM efficiency enhancement techniques, such as unmasking and caching. We show that SM complements an advanced unmasking sched- uler, ReMDM (Wang et al.) (025). ) further improving unconstrained text generation quality. Moreover, SM can leverage caching and confidence- -anredence-aware blockwise decoding from Fast- dLLLM (Wu et al. 2026), particularly improving generations in tigh-throughhut settings.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND: MASKED DIFFUSION LANGUAGE MODELS
We begin by formulating general diffusion models (Sohl-Dickstein et al. 2015 and continue with the forward and backward diffusion processes in MDLMs (Gong et al. 2025ap. We denote scalars with lower-case letters (I), vectors with bold lower-case letters (x), sequences of length ' with colon (e.g., X1.T), matrices with bold capital letters (X), and the transpose with X
Diffusion models describe a forward diffusion process as a Markov chain that progressively cor- rupts the original data: ((x::r(xx)) II (xxx(xx-1). Here, X. ~ Iama (ana(Xo) is drawn from the data distribution and o(Xx|xx-1) describes the transition probability at step t. The marginalized target distribution (g(X))) should be stationary and cheap to generate (e.g.. Gaussian distribu- tion). reverse diffusion process aims to reconstruct the original data with a parameterized function Pe(Xo:1 = Papaaxx (XT) I Pe (x, xx).
2.1 MDLM MODELING
Forward corruption We first focus on the corruption process for a single token; the extension to sequences discussed later. We represent language tokens one-hot vectors {0, 131M,, where [V] represents the cardinality of the vocabulary. The transition function in MDLMs is defined such that, at each step, the token either remains unchanged or mapped a designated absorption state: [MASK] E V. The transition can be expressed as 9(x, (( = Cat(x;: Q x where Cat(-: p) is the categorical distribution given probability mass vector E A 1111 and [Qrk, denotes the transition probability from token token j at time The marginal distribution after steps is:
(x,, xo) Cat(x, Q xo) = 0, + 0. )
where m is the mask token, Q II= Q and ag describes the probability of retaining the original state. The schedule is typically chosen such that or = 0, ensuring that the token is absorbed into the masking state with probability at the final step T. For example, linear masking schedule
with (1 t/T) is a popular choice (Austin et al. 2021a Gong et al. 2025a Nie et al. 2025
Reverse process Decoding aims to reverse the corruption process by iteratively denoising data, starting from the absorbed (masked) state at time step T. First, note that the forward transition probability between two time steps 0 I s < S is given by:
xxx, |x,) Cat(x,;; x.)xx)):
where Q.j = Q, Q represents the transition matrix from step back to step s. Assuming access the ground-truth token Xo. the exact posterior transition from X; to X2 can be computed via Bayes:
g(xe/xe, xo) a(xx Xx (x)a(xxxxx)))
0g-01 T-a X
I-a m T-a
if m, if
qx, (X))
X0
Since Xp unknown during inference, we approximate the posterior using a learnable function fø(x;) that predicts the original token from the corrupted input (x,,/xx,x,, Here, a learnable function approximates the ground-truth (Xo); hence, imitates the denoising from step t to step O. Substituting the approximation into the closed-form expression yields the parametric backward transition:
(x Xt, fo fo (x)).
Po (x, =
Ot
fo(x;) +
1 - O'2
m. Ct
(1)
1 0+
Reverse process in natural language processing The input consists of sequences of L tokens: x] JLL Decoding begins from a fully masked sequence: xf = m...., m. At each time step t, the current sequence estimate is passed through bidirectional model (e.g., non-causal Trans- former (Vaswani et al. 2017 Peebles & Xie (023)), yielding token-wise probability distributions:
While original denoising models use time conditioning (xo(xe, t)), Ou et al. 2025 present a method without time conditioning. We omit time conditioning in our theoretical formulation. However, we show experimentally that SM improves MDI Ms with (Section 4. and without time conditioning (Section

---

Published as a conference paper at ICLR 2026
PI-1 = go (x)::),, where Pi-1 A is the predicted probability mass vector on the [V- dimensional simplex for token I. Each probability mass vector (p/_1) discretized using sampling strategy (e.g,, nucleus or argmax), yielding x Describing the sampling function with h(.), we can write the reverse process of the entire model as the functional composition of the sampling and model forward pass: fo h gö.
Training objective
Given linear schedule of at the parameters (0) are optimized by minimizing:
(()) = - -E, 1)xx0 Ram (0.0000)))))))
Σ
xx ((x)) go [x] (xxbxx)
(2)
1m1
where I xx== is the identity function. The loss L()) is an upper bound on the negative log likelihood
of the data distribution Shi et al. 2024
Ou et al. 2025). U(0, 1) is the uniform distribution.
2.2 MDLMS IN PRACTICE
Unmasking strategies Equation suggests that the model randomly decides-based on the noise schedule a; whethe not to replace masked token with the predicted value fo(x.). However, many MDLMs use additional unmasking heuristics that improve the generation quality. One ap- proach is unmask fixed number of tokens per step, guided not only by the noise schedule but also by the model's confidence. For example, at time 1, Dream-7B (Ye et al. 2025) selects n Rs L/T tokens that have the lowest entropy values. Here, T' is the integer number of diffusion steps. More recent methods introduce exploratory (remasking) and accelerated (aggressive unmasking) decoding stages (Wei al. 20255] Wang et al. 2025). Rütte al. (2025) introduce an interpolation between masked and uniform diffusion, which introduces remasking already during the training stage.
Conditional generation Conditioning the generative process on a prompt (c2:ke) is straightfor- ward. For decoding, the prompt is simply prefixed to the (partially) masked solution at each iteration, ie., p |
1:l l. go([c life, x(!:1)) where only the last L tokens are updated.
SOFT-MASKED DIFFUSION LANGUAGE MODELS
As elaborated above, the iterative decoding MDLMs makes a binary decision: selecting either the original mask or the token predicted by the denoising model (fo). This binary choice results in a loss of valuable contextual information for the masked tokens. To overcome this limitation, we propose soft-masking (SM). SM augments the mask with intermediate context from the previous denoising step, thereby preserving informative cues and providing a richer input for the next denoising step.
3.1 SOFT-MASKING (SM)
We introduce SM, illustrated in Figure which enhances the denoising process MDLMs. As shown, the overall denoising process follows the standard framework of MDLMs. However, instead of discarding previous predictions during masking or remasking, SM gently retains information from the past predictions and incorporates them into subsequent decoding steps. This provides richer feedback to guide the next denoising step. To enable this, SM relaxes the binary constraint on the feedback tokens (x,-1) provided to the denoising model, allowing them to represent a distribution of solutions, i.e., x E AIV-1 instead of the one-hot x (0.1) 1)IVI.
General formulation
The feedback is formally defined as:
(1 A(P(-1)) m + + T Vi if if m, $t/_1
x =sm( s
if 7 m,
where p is the probability mass vector and 8/_1 is the discrete output from the denoising pro- cess. SM is applied only to masked tokens; previously predicted tokens remain unchanged. SM is implemented as convex combination of the mask token and a weighted superposition of the top-k predicted tokens. Here, À [0, 1) controls the amount of feedback, v, E {0, 1)1P is one-hot
4

---

Published as a conference paper at ICLR 2026
Unmasking
X
Denoising Transformer fs
Soft-masking (SM)
Sampling/argmax
top-k & normalizationn
P
(1 - A)
Σx V
Embedding table+softmax
Confidence-bare based
weighting
Attention (bidirectionel)
xN
FFN
1 te L B x
Embedding table
Continuous feedback
Context
Answer
Figure 2: Iterative denoising in MDLMs using the proposed soft-masking (SM). Given context, the aim is to predict the answer via iterative denoising of an initially fully masked response. Here, a bidirectional Transformer (fe) performs a single denoising step. This output is passed through an unmasking function that determines which tokens remain masked. Our proposed SM enriches the masked tokens by superposing them with the normalized top-k tokens at each position, weighted by confidence parameter (A). See Appendix A for an algorithmic description.
vector representing token i, and TT is the result of the probability mass vector being normalized over the top-k tokens: n. [p)- Sa..////jpppaa Eje [p/- which ensures that Σ; m 1.
Confidence- weighting In the following, describe the dynamic weighting strategy for SM. Intuitively, higher confidence should correspond to greater weight on the model's output, while lower confidence should preserve more of the original mask token. To quantify confidence, we use the negative entropy of the probability mass vector p denoted as (H(p) To map this confidence score weight [o, we apply scaled sigmoid function:
(P w. a ((wal H( wo))
where (-)) the sigmoid function. The trainable parameters was wh, and wa control the steepness, the offset, and the amplitude, respectively.
3.2 LEARNING THE SM FEEDBACK
To teach the richer SM feedback to the model, we introduce a new training methodology that op- timizes the SM parameters (w) concurrently with the main backbone model's parameters (0). This method allows the model to dynamically learn the optimal weighting between the mask token and the predicted tokens for each position, while the backbone simultancously adapts the richer feed- back. As illustrated in Algorithm [] the training method is a two-pass process. Standard MDLMs rely on the analytical tractability of the marginal distribution q(x, (xy) to enable efficient single-step sampling during training. However, the SM introduces a dynamic dependency on the model's 's inter- mediate predictions, rendering the exact marginal distribution analytically intractable. Our two-pass approach serves as approximation of this beebaack-uggment
marginal distribution q(x, Xo). Specifically,, we define the effective input state &, sm., (x,, go (x)),, where x, q(- ((-)xo), effec- tively maximizing the standard variational lower bound (Equation] 2 using this proxy state.
First, we approximate the probability distribution of the previous denoising step by passing the corrupted data through the backbone without a gradient, yielding P.:.4. This initial pass provides the necessary cofditondiiinnnn signal (the soft-masked representation) for the second pass, as practiced in (Chen et al. 2023). This distribution is then used to compute the soft-masked representation, which is passed through the backbone for a second time. The overall loss L from this second pass used update the learnable parameters for both the backbone (0) and the SM function (w) using
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Training with soft-masking (SM)
Input: Backbone 9ø. training corpus qaata (Xo). SM function with trainable parameters sm.. sampling bounds 0 S by < by I 1, learning rates for backbone (7p²) and SM (17mm). Output: Trained parameters for backbone (f) and SM parameters (w).
repeat x L Gdmta (.):
1/ Draw samples from data distribution 11 Lime step from bounded uniform distribut
~~ U(bi, bn)
x q(- xx xx)=²);
Corrupting samples
+ detach(0)
1/ Generate detached copy of backbone parameters
p + 90 xx] 11)
// Pirst model pass without gradient
// Second model with SM and gradient // compute loas Update and w based on loss with Adam optimizer using learning rates Tob and 1/mm
l:L + g0 (sm xt
1.L p)-1))
[(0,w) - Ei= x ((x))
Pi-1):
10 until end training;
their respective learning rates, T/b and 7mm- This approach can be highly parallelized with respect to the sequence length, unlike AR training with continuous COT.
Arriola et al. 2024) showed that a narrower sampling interval for reduces the gradient variance when optimizing L with batched gradient descent. Hence, we sample from the interval [bt, bn], 0 S by < bn S 1. Moreover, following the approach of Chen et al. (2023), we activate SM with probability Psm [o, 1]. This prepares the model cope with both soft-masked and standard inputs, which is particularly necessary at the beginning of the denoising process.
3.3
SM AS AN INTERPOLATION BETWEEN ABSORPTION AND UNIFORM DIFFUSION
This section provides a conceptual interpretation of the proposed SM mechanism. To this end, we consider two extreme values of the feedback-scaling parameter (A the feedback to a single value (A: = 1). First, assuming = 0 recovers vanilla MDLM. The model always revert this behavior by setting the scaling factor w, = 0. Second, À 1 feeds the previously predicted token (based on argmax) back to the denoising model:
and X 1) and simplify
(pr) (
if x if & + m.
sm
uniform DLM (Austin al.) 2021a) would receive the same feedback. However, the unmasking strategy remains active. Hence, this corner case can be interpreted as a masked DLM with uniform feedback for the masked states. This allows the model to explore different solutions through self- correction, enabled in the masked regions. Note SM's forward corruption process (A = 1) deviates from the uniform formulation: SM determines the distribution q(x:[Xo) with the denoising model (see lines 6 and 7 in Algorithm I rather than from a uniform categorical sampling.
Relaxing the scaling factor to take intermediate values E [0, 1] can then be seen as an interpolation between MDLM and mask-aus mugk-aeg uniform DLM. Importantly, this interpolation occurs in the spatial embedding space. Why might it be beneficial to retain a portion of the mask token besides attenuating low-confidence predictions? One reason is that many MDLMs masked tokens, and the presence of the mask likely still carries useful positional or structural infor- mation. This effect is particularly relevant for denoising models that do not use time conditioning.
pretrained predict
EXPERIMENTS
General setup We begin by evaluating SM small-scale language modeling benchmark, using a 169M (59M-paramete MDLM (Shhoo et al.] 2024) to demonstrate its benefits with both standard and improved unmasking strategies. We then apply SM the large-scale Dream-7B (Ye et al., (025) and Bream-Coder-77 7 (Xie et al. 2025) models, showing improvements on downstream coding tasks. In addition to training from scratch, assess the efficiency of adapting existing models via continued

---

Published as a conference paper at ICLR 2026
Table I: Unconstrained generation after pretraining from scratch. We report MAUVE (1) and gen- erative perplexity (L) of L 1024 generated tokens using MDLM (Sahoo et al. 2024 with binary masking or our SM. Evaluations are tabulated by varying NFE budget For unmasking, we use either the standard or the more recent ReMDM (Wang al. 2025); the highest scores are bolded. Gain shows the performance improvement between the SM and the baseline MDLM. RResults of
evaluating the ground-truth data and equal- backlbaccooneeeeee AR model are taken from (Shhoo et al.
2024
Gradient updates
Forward passes
MAUVET
Generative perplexity
Unmasking
Feedback
1/8
1/4
1/2
1/1
1/8
1/4
1/2
1/1
Binary
IM
IM
0.017
0.025
0.036
0.034
60.02
54.95
52.36
50.46
Our SM (iwe compute)
0.5M
IM
0.143
0.417
0.498
0.596
41.08
31.97
27.36
24.63
Standard
Gain
+0.126
+0.392
+0.462
+0.502
-18.93
-22.98
-24.99
-25.83
Our SM (iso-apdate)
2M
0.155
0.383
0.535
0. 0.602
39 61
30.74 74
26.
23.53
Gain
+0.138
+0.358
+0.499
+0.566
-20.4
-24.21
-26.27
-26.93
Binary
1M
1M
0.075
0.199
0.292
0.411
12.53
31.05
21.75
28.62
Our SM (iso-compute)
0.5M
IM
0.316
0.667
0.559
0.706
29.90
18.08
11:40
17:29
ReMDM
Gain
+0.241
10.267
+0.355
-12.63
-12.97
-10.35
-11.33
Our SM (ise update)
1M
2M
0,263
0.626
0,511
0.774
29. 69.62
17.58
10.85
16.72
Gain
+0.189
+0.427
+0.219
+0.363
-12.91
-13:48
-10.90
-11.90
(T 1020"
0.5M
0.5M
0.760
12.1
Data"
1.0
14.8
Table 2: MAUVE (1) of unconstrained generation after pretraining continuation. Gain shows the performance improvement between the SM and the binary MDLM with pretraining continuation.
NFE budget
Gradient updates
Unmasking
Feedback
1/8
1/4
1/2
I/1
Binary
IM
0.017
0.025
0.036
0.034
Binary
IM+100k
(=0.00000000)
0.027 (+00000))
0.032
0.038 002)
Standard
Our SM (iso-compute)
IM+50k
(.004)
0.129 (100029)
0.200
0. (10.04))
Goin
+0,036
+0.10/
+0.168
+0.22/
Our SM (iso-update)
IM+100k
0. 0.059
(o.021)
0.232 (00.02))
0.211
Gain
+0,04
+0.112
+0,200
+0,173
Binary
IM
0.075
0.199
0.292
0.411
Binary
1M+100k
O. (.002
(0:80(10:030)
0.315
0.421;10.021)
ReMDM
Our SM (iso-compute)
1M+
U. 1.137
0.441;10.004)
0.610;10:020)
O. 0.693
Gain
+0, 084
+0.262
+0.295
+0,272
Our SM (iso-update)
IM+100k
0.140
(1.02
0.617(20.020
0.692;20.030
Goin
+0.094
+0.252
+0.302
+0.277
pretraining (for small-scale models) or finetuning (for large-scale models). In the pretraining con- tinuation and finetuning setup, the baseline models with binary masking are trained with the same procedure for fair comparison. Crucially, since our proposed training algorithm (Alg. D requires two model forward passes per iteration (versus one in the standard training), we evaluate SM under two distinct computational budgets: (1) Iso-update: We match the total number of gradient updates (N). This isolates learning efficiency but requires roughly twice the wall-clock time for SM. (2) Iso-compute: We match the total number of model forward passes. In this setting, SM is trained for N/2 number steps, ensuring the total computational cost remains equivalent the baseline.
4.1
LANGUAGE MODELING
Setup We evaluate SM on language modeling using a (6rammrrre MDLM (Sahoo et al.
trained on the Open WebText (OWT) Gokaslan & Cohen 2019). The model utilizes DiT- fusion Transformer (DiT) backbone (Peebles & Xie| 2023). which integrates time-step conditioning into an encoder-only transformer. We investigate two training regimes for SM (with k 3): (1) pretraining from scratch for up to 1M steps, and (2) efficient adaptation, where we apply continued pretraining to a pretrained binary MDLM for an additional 100k steps. For evaluation, we report per- plexity on the OWT validation set (computed via the two-pass SM objective). As second measure, we assess the unconstrained generation quality using both generative perplexity and the MAUVE score (Pillutla et al. 2021), the latter serving robust metric for diversity and quality. We vary "The NEF budget (Appendix B.2 the ratio hetween the diffusion steps and the max generation length.
2024

---

Published as a conference paper at ICLR 2026
Validation Perplexity (PPU)
1.00
23.0
0.75
22.5
Our MDLM+SM
scaling
0.50
0.25
0.00
1.10
1.00
1.06
1.0
Training Step
Training Step
(a) Validation perplexity on OWT.
(b) SM's learned scaling factor (wa).
Figure 3: Continuing MDLM pretraining on WebnWetText. show the average + standard de- viation (shaded) across 5 seeds. SM is configured with k 3. (a) Our SM yields better (lower) validation perplexity than binary masking. (b) The model learns to fully use SM by increasing its influence over the scaling factor wis.
the number of function evaluations (NFE) between 128 and 1024 (representing 1/8 to 1/1 of the compute budget). Both the baseline MDLM and MDLM with SM require the same number of model passes. In addition to MDLM's standard unmasking, we also evaluate the models with a recent remasking strategy (ReMDM; Wang et al. 2025 See Appendix B.I for more details.
Results Table ! shows that our SM model trained from scratch, both under the iso-compute and iso-update budgets, consistently improves the MAUVE score (by up +0.568 points) and the gen- erative perplexity (by up to -26.93 points) when using standard unmasking. Interestingly, the iso- compute SM model (with N/2-0.5M pretraining steps) even slightly outperforms the iso-update model at certain lower NFE budgets (e.g., NFE 1/4). Thus, we observe that SM is particularly ef- fective in compute-restricted training regimes. Moreover, SM can further benefit from advanced re- masking strategies, surpassing ReMDM with the binary MDLM (by up to +0.468 MAUVE points), and even outperforming the AR MAUVE score (0.760) at the highest compute budget (achieving 0.774). Table 5 in Appendix C also shows that SM maintains the entropy. Additionally, MDLM with SM achieves superior OWT validation perplexities (21.47 iso-update and 22.36 in iso-compute), shown in Appendix C.1
Next, we demonstrate that SM can be efficiently integrated into a pre-existing binary MDLM via continued pretraining for up 100k steps. As shown in Figure 3al our SM decreases the validation perplexity on OWT (from 23.14 14 to 21.63). Binary MDLM (without SM) also improves the per- plexity, but by a much smaller margin (from 23.14 to 22.88). Figure 3b shows that MDLM learns to make use of the richer SM feedback by increasing the scale from initially near-zero to close to 1. The gain in validation perplexity transfers to unconstrained generation (Table ( in pretraining continuation too, where SM consistently improves the MAUVE score across all NFE budgets and unmasking strategies. Finally, all observations in MAUVE score transfer to generative perplexities and entropy, shown in Appendix C.2
Ablations We evaluate our SM design choices on language modeling in Appendix C.3 We find that using SM 80% of the time and k = 3 superposition yields the best validation perplexities. Moreover, an alternative SM feedback with softmax (with a trainable temperature) instead of top-k achieves competitive validation perplexities, the cost of higher compute and memory demands. Even though the softmax allows for propagating the gradients to the first model pass, applying the update based both model passes does not improve the perplexities while increasing the computa- tional cost. Finally, Appendix C.5 shows that the overhead in inference by SM is small (12%).
4.2 CODE GENERATION
Setup We integrate SM into the state-of-the-ard Dream-7B (Ye et al. 2025) and
7B (Xie et al. 2025) instruction-tuned models. For finetuning, we aim to use the same SFT datasets as the original models. Dream-7B uses Tulu 3 (Lambert et al. 2025) ) and SmolLM2 (Allal et al. 2025);
uses SFT (Codefuse & Team 2025). We deploy parameter- efficient finetuning using beight-decomposees low-rank adaptation (DoRA; Liu et 20,2224) on nearly 270k curated training samples with batchsize of yielding 33.5k update steps. We test the models
8

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) on coding tasks. Evaluations are tabulated by varying NFE budgets. We fine- tune the models with seeds and report the mean accuracy (# standard deviation). SM is configured with 1=1, Gain shows the comparison between the SM model and the finetuned baseline. The best performing model is marked in bold. The learned SM parameters are given in Appendix C.10
Dream Coder 7B (instrcc))
Dream 70 (instract)
hudget
Foodback
steps
HumanEval
HunanEval+
MBPP
MBPP+
HomanEval
HumanEvale
MBPP
MBPP+
Binary
25.0
25.0
27.4
29.4
18.9
17.1
26,6
31.2
Binary
28.5
(+:7 (+:.8)
25.9
24 1777
19.0
(5.9
27.0:=1
20.2
Our SM
33.5k
29. (11..)
28. (13.7)
33.2;11.8)
29.4 29.4
24.8(11.5)
23. 23:0:11.3)
32.3
36. 36.7
Gain
+.0
+0.5
+7.3
+4.8
+588
+7./
+3..7
+7.5
Binary
54.9
50.6
51.6
51.3
31.1
29.3
42.8
45.8
1/2
Binary
33.5k
03. (1114))
49.3
49.8:+0.9)
03 (3-2(a1.)
33.0
39
(22277)
6:22.0)
2:0.7)
56.4(+14)
38.3 +5.3
+22.6)
c12 48.4 48.4
(4
Gain
+3.4
+3.3
+6,4
+3.2
+4,3
+3,3
+13,7
Binary
75.0
69.5
65.8
70.4
57.9
$3.0
57.8
61.5
Binary
33.5k
75. (11.7)
18.9
65.6:10.4)
68.1, 1,11)
59.5 59.5
53.0 53.0:
58. 38.3,
Our SM
33.5k
76.2 (e3.4)
70.4
67.0 67.0.
69.6
57.8
(0.0
56.4.
(1.9
+1.5
-0.9
on two coding tasks, HumanEval (Chen et al. 2021 and MBPP (Austin et al. 2021b), as well as on
their plus version (Liu et al. 2023). We report results in iso-update training, and show similar gains in com-uee in Appendix
C.7
See Appendix B.2 for more details on the experimental setup.
Results The mean results
given in Table 3 tabulated based on the NFE budget of the decoding. First, we see that finetuned models (without
Accuracy (9)
60
Accuracy
45
55
30
30
Bream-Cade-ff
Dream-Cadee-Ff
SM) can maintain the perfor- mance of the original models, validating our finetuning pro- cedure. Second, the bene- fits observed in small-scale lan- guage modeling transfer to per- formance gains in large-scale models: the table shows a per- formance boost on nearly all tasks (up to 15.1%). The gains are particularly prominent lower NFE budgets. Furthermore, we show that SMM complements other efficiency-enhanceee models, with and with-
120
150
120
150
180
Throughout (tokens per
Throughout (tokens sec)
(a) HumanEval
(b) MBPP
Figure 4: Integrating SM into Fast-dLLM. We plot Pream-Coder 7B performance vs. throughput with both binary feedback and our SM. SM again excels high-throughput settings.
mechanisms.
Figure 4 reports the mean performance of the finetuned out SM. when combined with Fast-dLLMP blockwise caching and confidence-awee decoding (Wul function of the token throughput, which is indirectly
al. 2026). The performance is plotted determined from the block length of the blockwise decoding- wwtt longer blocks correlating with higher throughputs, as described in Appendix B.2 The benefits of SM become more promising at higher throughputs.
Ablations To validate our design choices, we perform extensive ablation studies on the Dream models. In Appendix C.6) we evaluate SM math tasks and notice similar performance gains. In Appendix C.8 we vary the number of top-k contributions. We find that & = optimal for coding tasks and x 3 yields the best overall results. We also tested trainable selection, by replacing the top-k filtering with a softmax that uses a trainable temperature. However, this did not improve the performance. Finally, Appendix C.9 applies SM only certain periods during denoising. We find that SM is particularly beneficial in the first 20% of decoding steps.
5 RELATED WORKS
Continuous feedback in AR COCONUT (Hao et al. 2025) feeds continuous token predictions back into the model enhance reasoning capabilities. This requires training from scratch- in- herently sequential process due to its reliance on previous continuous token outputs. To reduce computational demands, many have proposed summarizing sequences of tokens into higher-level

---

Published as a conference paper at ICLR 2026
continuous representations concepts (LCM Team et al. 2024 Tack al. 2026; Geiping al. 2023). Approximate training methods such as Jacobi iterations (w et al. 2025) further mitigate sequential training bottlenecks by iteratively refining AR thought tokens. More recent methods feed weighted superpositions of token predictions back into the model without additional train- ing (Zhuang al. 2025 Zhang al.)), enabling lightweight and training-free continuous feedback. While these approaches can improve task performance, AR models often suffer from unreliable halting behavior, necessitating entropy-based heuristics to terminate decoding. In contrast, our SM introduces continuous feedback directly into MDLMs, preserving parallelism during training and inference, incurring only constant overhead during training, and avoiding halting heuristics.
Discrete vs. continuous representations in diffusion modeling Continuous DLMs maintain continuous latent space throughout the diffusion process and perform a discretization step at the final readout (Li et al.] 2022, Dieleman et al. 2022 Gong et al. 2022 Strudel et al., 2022, Gong iet al. 2023 Guirajani & Hashimoto| )023); however, they generally achieve lower performance and do not allow for adaptation from pretrained AR models. Several works explore hybrid represen- tations. HART (Tang et al. 2025) augments a discrete AR-based image predictor with a residual continuous diffusion model to correct quantization errors, but remains dependent on unidirectional AR generation, limiting self-correction. Self-conditioning (Chen et al. 2023) is a closely related approach that a similar two-pass training methodology in training. However, their use of con- catenation increases the model complexity due to the resultant higher input dimensionality, a critical architectural difference from our method. Furthermore, it does not inherently offer a mechanism for a smooth adaptation. (Shhoo et al. (2025) derive discrete uniform-state diffusion from continu- ous Gaussian models, enabling faster training and generation, though scalability and downstream performance remain unproven. Chao et al.] (2025) propose fine-grained token representations using 1-dimensional vectors with base-b values, allowing partial unmasking during denoising, which is most effective with many decoding steps (T >> L). In contrast, our SM approach improves decod- ing performance while maintaining constant input dimensionality. By superposing the [MASK] token and top-k predictions, SM introduces continuous feedback without increasing complexity.
Efficiency improvements of MDLMs dLLM-Cache (Liu et al., 2025) introduces caching to MDLMs, maintaining almost static cache for prompts and a dynamic cache for the responses. This yields a speedup of up to 9x for long prompts at iso-accuracy. Semiiautoregressive generation via block diffusion (Arriola al. [2024] Nie et al. 2025), optionally combined with caching (Wul iet al. (0226). offers speedups but compromises full bidirectionality. NFE efficiency and generation quality have also been improved through dynamic unmasking strategies (Jin al., 2025; Wei al.)
2025 Wang et al., 2025), which adapt the masking schedule during decoding. These techniques are complementary our SM approach and can be readily integrated, as we have already demonstrated
by integrating ReMDM (Wang et al. 2025) and Fast-dLLM (Wu et al. 2026) with SM.
Concurrent works Concurrently with the development of SM, several other works (Zhnng et al., 2026 Pynadath et al.) 2025; Jo et al.] 2026 Zhou et al.) 2025; Shariatian et al.) 2025] Ma et al, 2025 sought to address information loss in MDLMs. Appendix D proovides an elaborate discussion and comparison, demonstrating that SM achieves superior language modeling performance where such comparisons are feasible.
CONCLUSION
We introduced soft-masking (SM), a lightweight mechanism for incorporating continuous feedback into masked diffusion language models (MDLMs). By blending the [MASK] token with convex combination of top-k predictions during iterative decoding, SM enables more expressive and flexi- ble updates without increasing model complexity. Applied to both small and large-scale MDLMs, SM consistently improves performance across language modeling and coding tasks. These results demonstrate that continuous feedback can enhance the capabilities of discrete diffusion models.
Limitations and future works Even though the training with SM is parallelizable in the sequence length, it requires an additional forward pass, which increases the complexity. We see future work in incorporating reinforcement learning-baa based methods (Black et al. 2023 Zhao et al. 2025 Zhu et al. 2025a) to leverage the richer feedback.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, or sensitive data. All experiments were conducted using publicly available models and datasets. The proposed method is intended for research in code generation and was evaluated in controlled settings. Our focus on improving model performance in low-compute budget environments was central consideration to mitigate environmental impact. The authors declare known conflicts of interest.
REPRODUCIBILITY STATEMENT
This paper describes the proposed SM method, including the training Algorithm [ and inference Algorithm 2 The setup used for training and evaluating our model on language modeling tasks is described in Appendix B.I The setup used for training and evaluating the Dream models for code generation is available in Appendix B.2 The code for both language modeling and code generation provided in the supplementary materials, along with the required experimental environments.
ACKNOWLEDGMENTS
This work was partially supported by the European Union's Horizon Europe Research and Innova- tion Program under Grant Agreement No. 101223271, and by the Swiss State Secretariat for Edu- cation, Research and Innovation (SERI) under contract number 25. 25.00443. We thank Ronan Tanios for his contributions the experimental evaluation. Moreover, grateful to Zaira Nazario and Abu Sebastian for the managerial support.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlfček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM2: When smol goes big data-centric training of a small language model. arXiv preprint preppia arXiv:2502.027377 2025. URL. www.ts:ss/ttrt.teeeeeettp http://s//aaaii....rr///////////
org.org////.... haagg//b//// 2200.
02737
Marianne Amriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block Diffusion: Interpolating Between Autore- gressive and Diffusion Language Models. In The Thirteenth International Conference Learn- ing Representations (ICLR), October 2024. URL
net. net/
d=tyEyYT267x
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured Denoising Diffusion Models in Diserete State-Spaces. In Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 34, November 2021a. http:httts////eeeeeee net. [forum?id=h? XixPCAL
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint 108.0700..7732. 2021b. URL
org/
12108.07733
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training Diffusion Models with Reinforcement Learning. In The Twelfthh International Conference Learning Representations (ICLR), October 2023. URL WtLhhttp:: /ttp:::e///peereieew .ne/ wtt.://rrmmmmromme
WihGrJFD
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray,

---

Published as a conference paper at ICLR 2026
Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances Neural Information Processing Systems (NeurlPS), volume 33, 2020. URL https//pprreeeeeggg.....oo
5. G.C fi |5
a a-Paper.pofj
Andrew Campbell, Joe Benton, and Valentin De Bortoli. A Continuous Time Framework for Dis- crete Denoising Models. In Advances in Neural Information Processing Systems (NeurIPS), vol- ume 35, 2022. URL
Chen-Hao Chao, Wei-Fang Sun, Hanwen Liang, Chun- Yi Lee, and Rahul G. Krishnan. Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. doi: 10.48550/a IXiv.2505, 18495.
URL hetps: nete http:///sw.....tt.oe....oooaaaaa
net
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel HerberrtVoss,, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Mísra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code. arXiv preprint arXivv22107.03374 July 2021. doi: 11155500001..211000033333 URL http: //tpx/aaavvv////a//aa org abs www./////00. 03374
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog Bits:
Generating Discrete Data using Diffusion Models with Self-Conditioning. In The Eleventh International Conference on Learning Representations' (ICLR), 2023. URL
net.
Fw
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reñichiro Nakano. Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv preprint 141666, November 2021. doi: 10.48550/arXiv.211 2111011111.111111166. URL Ut..httpp://axxx.orrg////.
.14168
Codefuse and Ling Team. Every sample matters: Leveraging mixture-of-exppers and high-quality data for efficient and accurate code LLM. arXiv preprint 2xXiv22503.17793, 2025. URL https:
//arxiv, warp.w..rr//ab//5533.2.
DeepMind. Gemini Diffusion, 2025. URL [ttppe//deepmnidd.oooll//////a//
1 i-diffusion/
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusson for categorical data. arXiv preprint arXiv:2211.15088 December 2022. doi: 10.48550/arXiv.22111 21..:///XX..221.555333 URL http: laaxxiv.or//ab////
15089
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025. URL https:://penreviieww http:/iwwww..eeeeeeeeeeeeeeeeeee wtw.eeewwnee//ooomm1dd0d00-
o6Bwtq7h
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:://kky1oooo000...i...
//ppenWebTexttCorguu 2019.
12

---

Published as a conference paper at ICLR 2026
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. In The Eleventh International Conference on Learning Representations (ICLR), September 2022. URL
uum imm rLVXs]
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and Lingpeng Kong. DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, December 2023, doi: 10.18653/v 1/2023. findings-emnll.666 URL httphtts/cclaattollllo/oo wwgpo//3.....//00..//// wtt.o////...............eeeeeee. findings-e mnlp. 6607
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling Diffusion Language Models via Adaptation from Autoregressive Models. In The Thirteenth International Conference Learning Representations (ICLR), May 2025a. URL https://ppenreveew..t
um. un.2idjjittLVkk
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. arXiv preprint arXiv:2506. June 2025b. doi: (0448550arxxw.2206220333
URL htthht/p/aaxiiii wttp://wwrrg/bb///5......
Ishaan Guirajani and Tatsunori Hashimoto. Likelihood- Based Diffusion Language Models. In Ad- vances Neural Information Processing Systems (NeurIPS), volume 37, November 2023. URL
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, Xiaokang Zhang, Xingkai Yu. Yu Wu, Z. F. Wu. Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, et al. DeepSeek-R incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(808 :3:08 638, September 2025. ISSN 1476-4687. doi: 10. 10.1038/s41586-022 025-860222--922--2 URL ht https:tp.... /ooi://ooo.. 100gg/100
22-2
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training Large Language Models to Reason in a Continuous Latent Space. In Second Conference on Language Modeling, 2025. doi: 00.48500/rxiv.2411..000 //openzeview net wttt:/ffrmmmiomo wtt:iiiddiiiiiiiei 7S4Ip3
URL https:
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2020. URL https://pp:/o eedings neur ips, a4b P00-Paper.ppo pdf
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang. Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xu- ancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-Coder Technical Report. arXiv preprint arXiv:2409. 12/86, November 2024. doi: 200.550//aXi.2249..122
URL
http: org/ abcp///.s//22/// 2ttp:///ww..../////////0/..0.0 12186
Labs Inception, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mercury: Ultra-Fast Language Models Based on Diffusion. arXiv preprint arXiv:2506.1 17298, 2025. doi: 10.48550/arXiv 20..500/X.....56... 17298. URL 8/ 8/2506.172 17298
Xiangqi Jin, Yuxuan Wang, Yifeng Gao, Zichen Wen, Biqing Qi, Dongrui Liu, and Linfeng Zhang. Thinking Inside the Mask: In In-Place Prompting in Diffusion LLMs. arXiv preprint arXiv:2508. 10736, August 2025. doi: 10.48550/arxiv.2500.
URL http:://arx
wwogg//bs//5000 10736
13

---

Published as a conference paper at ICLR 2026
Mingyu Jo, Jaesik Yoon, Justin Deschenaux, Caglar Gulcehre, and Sungjin Ahn. Loopholing Dis- crete Diffusion: Deterministic Bypass of the Sampling Wall. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. doi: 10.48550/arXiv 25100/0190..110119900. URL https: http://///eeeeeeee.eeo net forum? hatpee///oommmmmmwm/mmmm 2dwM.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord. Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2rXiv2241..15122 2025. URL httphttt////xiiiii /ttpps://aaxxv..oo org/ htt//bb///221111 15124
LCM Team, Loic Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Be- len Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R. Costa-jussà, David Dale, Hady Elsahar, Kevin Hefferman, João Maria Janciro, Tuan Tran, Christophe Rop- ers, Eduardo Sánchez, Robin San Roman, Alexandre Mourachko, Safiyyah Salcem, and Holger Schwenk. Large Concept Models: Language Modeling in Sentence Representation Space. arXiv preprint arXiv:2412.08822 December 2024. http: wtt./aiiiiii........ wwwporgg/abb/////
doi: 10. 10.48550/arxiv.2412..o
URL
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM Improves Controllable Text Generation. In Advances in Neural Information Processing Sys- tems (NeurIPS), volume 36, October 2022. htttpps://penneevee..eeee.
d= d=3a9IrEs
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman. Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https::/ //ppene view. net oot.:/oooumi----VLLPEEEEEEE. v810pN6E01
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023. URL
htttps://openrevite wet.:iiwwwweeeeeee.m.m.. htt.:pawwwe.ee....eeeeoooonnaaaa
Shih- Yang Liu, Chien-Y- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decoopposed low-rank adaptation. arXiv preprint arXiv:2402.09933 09355,, 2024. URL.
09353
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang, dLLM-Cache: Accelerating Diffusson Large Language Models with Adaptive Caching. arXiv preprint 2506 06225, 06295, May 2025. doi: (D.4855WarXii.22566669999 URL
httt:://axii.. org. 2506 06295
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings the 41st International Conference Machine Learning (ICML), volume 235, July 2024. URL htttp:://openeeee/e/ http://wwwwweeeeoummm/ forum? www.roowwuwnnonooooooooororoooee id=Niic VIPA
net.
Yuxin Ma, Lun Du, Lanning Wei, Kun Chen, Qian Xu, Kangyu Wang, Guofeng Feng, Guoshan Lu, Lin Liu, Xiaojing Qi, Xinyuan Zhang, Zhen Tao, Haibo Feng, Ziyun Jiang, Ying Xu, Zenan Huang, Yihong Zhuang, Haokai Xu, Jiaqi Hu, Zhenzhong Lan, Junbo Zhao, Jianguo Li, and Da Zheng. dlnfer: An Efficient Inference Framework for Diffusion Language Models. arXiv preprint abtp///aa///aa wtwxxvvorgg/bb//250..6666
October 2025,
doi: 10.48550/arXiv. 2510. 0.08666. URL http:
org
Jinjie Ni, Qian Liu, Longxu Dou, Chao Du, Zili Wang, Hang Yan, Tianyu Pang, and Michael Qizhe Shieh. Diffusion Language Models Super Data Leamerr. arXiv preprint arXiv:2511.03277 November 2025. doi: 10.48550/axxv..551...0 URL (tthhtp:// //ar////.vvv.. org/ abs/2511
03276
14

---

Published as a conference paper at ICLR 2026
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), 2025. URL http:::/opperreeeeee.ntttt
um? um? ciliddingCCzz
OpenAl. Learning to reason with LLMs, September 2024. /ttpat/pps///geeeea.cc////// eex ex/learning iegaaninng-to reason weithhlllms http://noeeaiii.mmmmiddd////aame with-llms/.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL net. ftttt/oormmiddds/// 8Tamm
https://openrevaie
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pp. 4195-4205, 2023. thecvf.......c com/ wwwontent/Iccv00000 http3//hhtmm/eeeeeeee Sca abaalable_am _ille__iffusoooo Modelsoeeelllll with Watth_Transfooeeeee ICCV 2023 wwwpppappp....
URL http:ttt::///eacccccccaa
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
URL /tlep:://pperreview wet.iiew.ee..eee...em ftt//oorrmmrdr////// foopmmmiddTTgggnnnppPP
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffu- sion Beats Autoregressive in Data-Constrained Settings. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. URL (tLlhttp::/ / /ttp:////opeereve.....
St//oorum?id=SSHtO0S
Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang. CANDI: Hybrid
Diffusion Models. arXiv preprint 2222002510.22510, October 2025. doi: 1044550/r2xi.2510.25510
URL httptt:t/a/xvv..oo
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. arXiv preprint arXhv:2412.15115. January 2025. doi: 10.88500axxi..412..5511 URL http: wwtp.orraaaa///1.155202
Qwen Team. QwQ: Reflect Deeply on the Boundaries of the Unknown, November 2024. URL https: http///ennlm.. thu.. Lttp.ooo/bloog/ggg-2222--- https://qwnlmmmgiiiie http://www....ibbbo.oooooo////b/
previeew.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners, 2019. UR//nttps://storage http://tstoorr..rr..o..oo resea
rchhub. ccm/ oads
Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hof- mann. Generalized Interpolating Discrete Diffusion. Forty-second International Conference Machine Learning (ICML), 2025.
ZvV9DPV9
Subham Sekhar Sahoo, Marianne Arriola, and Yair Schiff. Simple and Effective Masked Diffusion Language Models. Advances in Neural Information Processing Systems (NeurIPS), volume 38, 2024.
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. The Diffusion Duality. In Forty-second International Conference Ma- chine Learning (ICML), 2025. httss::p::/ /tttts:://oenreevieer net.iwewwweeeeeeeeee.e..e..eee.. weett/forrmmidd-9P9p/0.
15

---

Published as a conference paper at ICLR 2026
Dario Shariatian, Alain Durmus, and Stefano Peluchetti. Latent Discrete Diffusion Models. arXiv preprint arXiv:2510.18114, October 2025. doi: 10448500a2x..2210..1111 URL http:
//axxiv. waxxiv..rg//bs//2511..0
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K KTitssis. Simplified and General- ized Masked Diffusion for Discrete Data. In Advances in Neural Information Processing Systems
httpr/eewwwiweweeeeeeeeeeeeeeeee www.eeeeetooriieeeeeeoeoe (NeurIPS), 2024. URL htt.a///t///////oooaa/eoa//./.aa
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibriun Thermodynamics. In Proceedings of the 32nd Inter-
national Conference on Machine Learning (ICML), volume 32, June 2015. URL https: //proceedings. mil.iiinnn..l.......mmmmm.mm...m prespssssss....ss.........s.s.ss varess/v3l 3/ sohl wwikkstiiini... ww111m.........
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations (ICLR), October 2020. URL https: //openzeview. wwt. wtteet/foormmmmmppceie..31
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, and Rémi Leblond. Self- conditioned Embedding Diffusion for Text Generation. arXiv preprint 1..iv:22110022666 Novem- ber 2022. doi: 10.48550/arXiv 2211 222222112444 1.04236.
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. LLM Pretraining with Continuous Concepts. In The Fourteenth International Conference Learming Representations (ICLR), 2026. URL net/ forum? www.:mmddwwwweead000000
htttps://openrettt
Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. HART: Efficient Visual Generation with Hybrid Autoregressive Transformer. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL httpt: htttps://openrevl.ee www.://ooooommldaalooooeeeeee//e
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural In- formation Processing Systems (NeurIPS), volume 30, 2017. httpsstss:///rooeee
heurips.com cappepppppppppepeeppeeeeeeeeeeee
4a845aa Paper. pdf
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking Discrete Diffusion Models with Inference-Time Scaling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. URL hattps://openrevieee wwwprrewwwww..eeeeeeeeeeeeeeeeee net/fott
um. umm?id-IJJn 1ridd-IrryQAO001
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-Thoughh Prompting Elicits Reasoning Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, October 2022. URL hottt::t/://// wwwnoeeeeeiieee.. net. forum? wwwoooommdddoveeeeeeeee ViQMeese________
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang. Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles. arXiv preprint arXiv:2506. 10848, June 2025. doi: 10.48550/arXiv2255 10848. WRRl/http://axxiv orrg/
abs 2t1/22566.00 106.108440 10848
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding, In The Fourteenth International Conference Learning Repre- sentations (ICLR), 2026. URL htth::tp:/// owwprevieneeeeeeeeeeeeeeeereeewr / wwtp:ee//fooot wwwp/eeetwiiddaaeeeeeee......mae 3231s 6hnno
Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel Continuous Chain-of- Thoouhh with Jacobi Iteration. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 926, 2025. doi: 10.18653/v 1/t01/ main.47. URL hntptst//aaaaac
nthol nthology.org/2033 2005 emn p-ma i 477
16

---

Published as a conference paper at ICLR 2026
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream-Coder 7B: An Open Dif- fusion Language Model for Code. arXiv preprint arX1v22509001142. September 2025. doi:
(..48550/rrXv.2500.0 142. htt.htttp//arxx.voooo wtw.://rr/g.b///0...0111...
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of Thought: Chain-of- Thouuhh Reasoning in Diffusion Language Models. Advances in Neural Information Processing Systems (NeurIPS), 37,
December 2024. URL https: /ttpttps:/ppcoceeeee ngs. new.enneuuiie citp:rissssppppppepearaaaeeeee
html
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7B: Diffusion Large Language Models. arXiv preprint arXiv:2508. 5447,, August 2025. doi: 10.48550/arXiv 2508 15487. UBL.nttp://arxiv..rrr//bb/// 15448
Zhen Zhang, Xuchai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang. Yelong Shen, and Xin Eric Wang. Soft Thinking: Unlocking the Reasoning Potential of LLMs Continuous Concept Space. In The Thirty-ninth Annual Conference on Neural Information Processing Sys- tems (NeurIPS), February URLhttps:: /ttp://///peeeeeeeeee/mmo net. forum? (orumm?.d=BBQQaPGKKg
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dl: Scaling Reasoning in Diffu- sion Large Language Models via Reinforcement Learning, In The Thirty Annual Conference on Neural Information Processing Systems (NeurlPS), 2025. URL htttps://openrevlews
et. et/ 1d= 1BUUE
Huangjie Zheng, Shansan Gong, Ruixiang Zhang, Tianrong Chen, Jiatao Gu. Mingyuan Zhou, Navdeep Jaitly, and Yizhe Zhang. Continuously Augmented Discrete Diffusion model for Cat- egorical Generative Modeling. In The Fourteenth International Conference on Learning Repre- sentations (ICLR), 2026. http:://p /ttp:////eeeeeeieeeeeeeee net httt:/ooummiddddoe
BBwt.
Cai Zhou, Chenxiao Yang, Yi Hu, Chenyu Wang, Chubin Zhang, Muhan Zhang, Lester Mackey, Tommi Jaakkola, Stephen Bates, and Dinghuai Zhang. Coevolutionary Continuous Dis- crete Diffusion: Make Your Diffusion Language Model Latent Reasoner. arXiv preprint arXiv:2510.03226, October 2025. doi: 10.48550/arxivv251000 w.org/abs/2510. 03206
URL http:://arai
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. LLaDA 1.5: Gtriance-Reduce Prefer- ence Optimization for Large Language Diffusion Models. arXiv preprint arXiv:2505. 19223, May abs /200 19223
2025a. doi: 2..4550//rXiv.25500 19223. UtLppttp://rxxxvorr
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought. In The Thirty- ninth Annual Conference Neural Information Processing Systems (NeurIPS), 2025b. URL
Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao. Mixture of Inputs: Text Generation Beyond Discrete Token Sampling. In The Thirty-ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2025. URL https: http:///ppneeeeeeeeeem
17

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Background: Masked Diffusion Language Models
2.1
MDLM Modeling
2.2
MDLMs in Practice
3
Soft-Masked Diffusion Language Models
3.1
Soft-Masking (SM)
3.2
Learning the SM Feedback
3.3
SM as an Interpolation Between Absorption Uniform Diffusion
Experiments
Language Modeling
Code Generation
5 Related Works
Conclusion
10
Inference with SM
19
Experimental setup
19
B.1 Language Modeling
19
B.2 Code Generation
20
More Results and Ablations
23
C.I Language Modeling: OWT Validation Perplexity
23
Language Modeling: Generative Perplexity and Entropy
Language Modeling: Training Share, Top-k, Softmax, and Gradient Updates
23
Language Modeling: Inference Speed
25
Language Modeling: SM Visualization
25
C.6
Mathematical Reasoning
26
Coding: Iso-Compute Models
26
C.8
Coding:
Top-k and Trainable Sofumax Temperature
26
C.9
Coding: Time-dependent Masking
27
C.9.1
Early Late Stage SM Impact
28
C.9
When SM Most Beneficial?
29
C.10 Coding: Leamed SM-feedhack Parameters
29
Concurrent Works
30
18

---

Published as a conference paper at ICLR 2026
Algorithm 2: Inference with soft-masking (SM)
Input: Backbone ge. SM function with parameters sm generation length L, number of denoising steps T.
Output: Generated sequence x)
X
<- m, m.... m
// Tritialize sequence with full masks
repeat
1:4 + ge (x)
// Backbone pass with SM 1/ Sampla backbone stribution le.g., nucleva) // Uanasking (e-g.r based on entropy) Computing feedback
x < sample p
8 unmask & 1:L P P 1,,,,,
1. Pi
x
< (x) P )
t t 1-
until 1=0;
A INFERENCE WITH SM
Algorithm describes the inference procedure with SM.
B EXPERIMENTAL SETUP
This appendix provides details on the experiments conducted for language modeling and code gen- eration. All experiments were 1-2 compute nodes with 1-8 NVIDIA A100 GPUs, each with 80 Gigabytes of VRAM.
B. LANGUAGE MODELING
Pretraining from scratch The pretraining experiments follow the setup by |Sahoo et (2024). We use bidirectional Transformer backbone with 12 layers, 12 attention heads, and 768 hidden dimensions. The model is tokenized using the GPT-2 tokenizer (Radford et al. 2019). Pretraining performed on the same Open WebText (OWT) (Gokaslan & Cohen] (2199) split, with the last 100k documents reserved for validation. We use an AdamW optimizer with a linear learning warm-up for the first 2500 steps, and then keeping constant at the =3e-4 and r/mm =le-2 for the backbone and the SM parameters, respectively. We train the model for 1M training steps using a batchsize of 512, which yields 262B tokens seen during training. The dropout rate set at O.I. Training was performed on 2 compute nodes, each with 8 A100 GPUs (80 Gigabytes of VRAM each), using batchsize of 32 per device and deploying gradient accumulation achieve an effective batchsize of 512.
Pretraining continuation Our starting checkpoint was pretrained on OWT for IM steps and was released by Sahoo et al. (2024). We train the model for 100k training steps using the same hyperpa- rameters as in pretraining from scratch. We train each model with 5 different seeds (1, 2. 3, 4, 5) to account for training variability. Training was performed on 4 A100 GPUs (80 Gigabytes of VRAM each) using batchsize of 32 per device and deploying gradient accumulation to achieve an effective batchsize of 512. Pretraining one model took approximately 64 hours and 139 hours for binary and SM, respectively.
Soft-masking parameterization For the SM feedback, we add a trainable module to the model. This module contains all SM logic and augments the input embeddings with SM before the main forward process. We initialize the three SM-feedback parameters with the parameters given in Ap- pendix B.2, We have few imposed constraints on these values: W, E [0, 1]. 2 0. and wb S 0. To account for these constraints, we apply simple re- parameterizations during training:
w, is passed through a sigmoid, ensuring it remains in [0, 1]-
Wa and Wh (negative version) are each passed through a softplus, guaranteeing non- negativity.
19

---

Published as a conference paper at ICLR 2026
Before inference, the learned parameters de- de-parameterizee:: take the direct output of the optimization, apply the inverse transforms, and for w/ additionally negate the result so that respects the W/ 0 constraint. All other SM parameters (i.e., k) specified in our added module. We simply perform a forward pass through this module to get the mixing weights for the new input embeddings.
Generative perplexity, entropy, and MAUVE in unconstrained generation Our unconstrained generation evaluation follows the experimental setup by Wang et al. (002)). We perform uncon- strained generation of 5000 samples (L = 1024) per model with a batchsize I using nucleus sam- pling with p 0.9. We use GPT-2 large for measuring the generative perplexity. Moreover, GPT-2 large as the embedding model for MAUVE score computation, where we set the MAUVE scaling hyperparameter to 5. Concerning ReMDM unmasking, use the max-capped schedule (Map = 0.04) for fast sampling (T < L) and the loop-strategy (len = 0.55, loft = 0.05, (((an) = 0.9, Mepp = 0.02) for inference-time scaling (T > L).
use
B.2 CODE GENERATION
This appendix describes the details of our finetuning experiments on Dream-7B and Dream-Coder 7B.
Backbone models We use the pretrained Dream-7B (Ye et al. 2025) and Dream-Coder-7B (Xie et al. 2025) backbones, respectively. Both of these 7B parameter models that are adapted from the Qwen2.5 family (Qwen et al., 2025 Hui et al. 2024). For both models, we use the instruction- tuned versions.
Tasks We primarily evaluate on four code synthesis benchmarks:
HumanEval Chen al. 2021 1 benchmark of 164 Python programming problems de-
signed to test a model's s ability to write correct and functional code from natural language
specifications.
MBPP Austin et al. (2021b): benchmark consisting of 974 "Mostly Basic Programming Problems," each specified in natural language and accompanied by input-output test cases, targeting introductory- samples the test subset. It is important to note that some works report higher scores for the Dream models on the MBPP task. This is result of using the "sanitized" subset of the data, instead of the standard Im-eval version that we used.
programming tasks. We only perform evaluation with the 500
EvalPlus: HumanEval+ and MBPP Liu et al. (203)): Extended henchmarks involving adding more unique test cases and correcting any inaccurate ground-truth solutions.
For our experiments, we use the standard instruct implementations of these benchmarks as implemented by im-evallation llnaiinnnnnn-i harness at the time of writing. For HumanEval++ we created custom instruct version of the task, using the same prompt as humaneval_ing ruct. All tasks were evaluated in zero-shot setting with temperature of 0.1, top-p value of 0.9, and the entropy-based unmasking algorithm. The HumanEval tasks were evaluated with a max generation length of 768, and the MBPP tasks evaluated with generation length of 512.
NFE Budget MDLMs typically have maximum generation length as a parameter during gener- ation. For our experiments, we use 768 for HumanFyal(+) and 512 for MBPP(+). These models also have a parameter that quantifies the number of diffusion steps that should be taken unveil all tokens. Typically, diffusion models set the same number of steps as the number of maximum tokens. In these cases, exactly one token is unmasked at each diffusion step.
Decreasing the number of diffusion steps leads to much more efficient computation-by unmask- ing more than one token each step. From a computational cost perspective, this is essentially a linear relationship. If four tokens are unmasked at each denoising step, the generation will hap- pen 4x faster than if only one is sampled each step. This leads us to define the NFE budget of a generation:
20

---

Published as a conference paper at ICLR 2026
Given fixed-lengte generation task with max number of tokens, the NFE budget of the generation will be:
NFE budget # of diffusion steps
(4)
max # of tokens
In our experiments, we discuss NFE budgets of 1/4, 1/2, and 1/1. the NFE budget is 1/n, means that, average, tokens are unmasked per step.
General setup and hyperparameters We aimed to keep the finetuning implementation as close to the original Dream-7B (Ye et al. 2025) SFT implementation possible. Due the fact that the exact SFT implementation code is unavailable, we use the DaffuLLaMa code (Gong et al. 2025a) and the Dream paper (Ye et al.) 2025) for reference. Algorithm [ displays overview of the algorithm that we use. 'The only change for scaling the algorithm for finetuning larger models is that, rather than updating the full weights, we update only the weights of light-weight parameter- efficient finetuning (PEFT) module. As our PEFT module, we use a DoRA adaptor with parameters: rank r = 16 and a = 16. We apply the module only to attention matrices: _"__proj", "k _proj", "v_pro j "o "o_ j_ When finetuning the SM versions, we only use the two pass approach and activate SM with psm = 0.5. We didn't It see as much of an effect as language modeling with varying this value. We use AdamW optimizer with cosine scheduling, a 0.03 warmup ratio and a max gradient norm of 7.0. The learning rates are capped at Thb =le-5 and 77mm = =le-2 for the DoRA and the SM parameters, respectively. The finetuning is performed with an effective batch size of 8 on one A100 40GB GPU and takes about 71 hours for 33.5k gradient steps.
Training corpus For the training corpus, we aimed to use the same datasets that were used by the Dream models (Ye et al. 2025
Xie et al. 2025) in the SFT phase of their Instruct model training.
Dream-7B The authors report instruction-tuning with ... 1.8M instruction-respont pairs from Tulu 3 Lambert et al. 2025 and SmolLM2 (Allal al. 2025 p... We use the same mix. For the Tulu data, we use al
wwimmmituuuue xture, consisting of 939,00 939.000 pairs in their training set. Since there is no validation set, we hold out random 1% of these pairs for validation. For the SmolLM2 data, we use HuggingF aeeinngaaceeee aaaee///////// amollaal kk, consisting of both a training set with 1.04M pairs and a test set with 54.9k pairs. We used the defined test set for validation. These datasets are the concatenated and shuffled to make up our training corpus.
Dream-Coder77B For the Coder model, Xie al. (2025) specify explicitly that they use SFT (Codefuse & Team (025) for their SFT training. We use the same dataset. This dataset consists of 4.48M pairs. We hold out a random 1% at the beginning to be used for validation.
i inclus i
Preprocessing The preprocessing of these question-answer pairs executed as follows:
1. Max train/validation size. We first sample 300,000 datapoints use for training and 500 to use for evaluation.
2. Context and response splitting. All datapoints contain a sequence of user and assistant messages. We consider the last assistant message be the response and all other previous messages to make up the context. We apply the respective models' chat template to the messages before tokenization.
3. Filtering the dataset. We conduct dataset filtering on three different attributes of the data: (1) inputs > 2048 tokens; (2) responses < 5 tokens; (3) any examples with tool-calling After filtering, less than 270,000 context-response pairs. While this varies with our sampling seed, the total number of tokens in our training is often just under 200 million. These 200 million tokens consist of a fifty-fifty split between prompt and response tokens.
(i.e., using <tool. call>> and </tool-call
typically left with just
4. Padding with <eos>> tokens. After finetuning, we want the diffusion model to retain its ability to decide when to end its generation process. In MDLMs, this is typically done by predicting <eos>> tokens for all the end positions that the model does not want to use. In order to ensure this is part of the training process add aed Uniform(0. 50) <eos>>
21

---

Published as a conference paper at ICLR 2026
tokens to the end of each training sample. The amount of padding that is added to the training set varies for each sample.
5. Partially masking the responses for training. We only apply masks to the response tokens during the training process. This is performed in the following way: First, as Algo- rithm [ describes, mask probability value Uniform bn) is sampled. This value / then used the masking probability: fraction t of the response tokens masked for the loss calculation. [Ariilaa et al. (2024) found that sampling extremely low/high t can lead to high variance in the gradient norms, making training quite difficult. For this reason, a clipped noise schedule (b), by) = (0.2, 0.8). The masks are sampled uniquely for each sample.
SM-feedback parameter initialization As mentioned in Section 3.1 we introduced mapping from the entropy to the amount of feedback:
X (P = ws oo. ( -() -ww))
(5)
These parameters: Way W/ and Wa are learnable during the training process. We ensure that the scale value, is initialized close to O at the start (slightly larger due to the sigmoid reparameterization discussed in Appendix [..] This ensures that the model only adds SM-feedbbck if it learns that this is optimal via the finetuning process. For the steepness and center of the sigmoid, we conduct small statistical analysis of the expected entropy distribution. Although the theoretical range of - -H(p is l)) in practice we observed that 95% of values were above LB ~ -1. For this reason, we initialize the center of the sigmoid at LB//2 and choose a 10/LBB. Effectively. this normalizes the negative entropy to be between [0, before applying sigmoid of a' 10, 0.5. We found that the learning rate for the hyperparameters needed to be set much higher than for the DoRA adaptor in order for them to be able to traverse the entire range of options. The rate was set to 7mm -2-2 for the training process.
Fast-dLLM Throughput Experiment Since Fast-dLLM (Wu 2026 performs blockwise decoding, there is noticeable correlation between the block lengths and the inference time. Longer blocks allow much more parallelism in decoding and lead to faster response, while shorter block lengths tend to drift the model towards more autoregressive generation. For our experiment, we used block lengths of 16. 32, 48, 96, 128, 192, 256, and total generation length of 768. These led to fractional lengths of 1/48, 1/24, 1/16, 1/8, 1/6, 1/4, 1/3. We then calculated the throughput as the total tokens generated (for all questions) divided by the total decoding time. The results are shown
in Figure 4
22

---

from the previous Published as a conference paper at ICLR 2026
C MORE RESULTS AND ABLATIONS
C.I LANGUAGE MODELING: OWT VALIDATION PERPLEXITY
In this appendix, benchmark the validation perplexity of SM against state-of-the-art methods. We note that, in contrast to standard baselines, SM necessitates two model passes for perplexity evaluation. demonstrated in Table g our method achieves superior performance compared to all other diffusion models in the iso-update regime, including a concurrent work (LDDM-M; Jo et al. 2026). Furthermore, even under the stricter iso-compute constraint, SM outperforms MDLM and SEDD and remains competitive with GIDD+
Table 4: Validation perplexity OWT. RResults for AR and SEDD taken from 2025
Sahoo al.
Gradient updates
Forward passes
Training tokens
PPL
ART
0.5M 1M
0.5M
262B
17.54
SEDD Lou et al. 2024
1M
262B
24.10
MDLM (Sahoo el al 2024
IM
IM
262B
S 23.21
GIDD+ (Rütte et al.
2025
1M
1M
262B
S. 222.9
LDDM-M [o et al. 2026
IM
2M
262B
S 21.90
Our MDLM+SM (is0- (io-compute)
0.5M
1M
131B
S. 22236
Our MDLM+SM (iso-update)
IM
2M
262B
21.47
C.2 LANGUAGE MODELING: GENERATIVE PERPLEXITY AND ENTROPY
This section analyzes the generative perplexity and the entropy observed during unconstrained gen- eration. Table 5 shows the generative perplexity and the entropy when training MDLM with SM from scratch. The reported generative perplexity repeated from Table to facilitate comparison. As shown, our SM maintains an entropy on par with the binary masking baseline. For ReMDM unmasking, SM shows its lowest entropy at the 1/2 NFE budget (5.234 and 5.217 for iso-compute and iso-update, respectively). While still being higher than binary masking entropy (5. (5.209), this dicates a highly low-diversity output, which likely explains the observed degradation in the MAUVE score, despite the simultaneously low generative perplexity (11.40 and 10.85). This suggests that the ReMDM unmasking process might require specific hyperparameter tuning to balance diversity with human-like text generation (MAUVE).
Table and Table respectveely show the generative perplexity and entropy for the continuation of pretraining. Here, too, SM consistently improves the generative perplexity while maintaining the entropy.
C.3 LANGUAGE MODELING: TRAINING SHARE, TOP-K, SOFTMAX, AND GRADIENT UPDATES
This appendix ablates the SM training probability (pem) and the top-k value in the language mod- cling experiments. Recall that k is the number of predicted step-that will be combined with the mask token. We start with default configuration of pam 0.8 and k = 3 and vary each parameter separately. As shown in Figure 5a increasing the SM training probability from 0.5 to 0.8 improves the performance. This is likely due to the model learning to leverage continuous feedback more effectively, soft-masking mechanism more frequently during training. Increasing the SM training probability to is detrimental, as the model loses its ability to handle binary masking, which is essential for the initial decoding steps.
higher probability means it is exposed to the
Figure 5b shows that the perplexity improves when increasing k from I to 5, with only marginal gain observed between 1 3 and SM mechanism using trainable softmax temperature. This method calculates probabilities using learned temperature applied across the entire vocabulary, V. letting the model scale the range of its own feedback. In Figure 5b. denote this ablation k |V] signify that it considers the entire
Beyond the top-k study, we investigated an alternative
23

---

Published as a conference paper at ICLR 2026
Table 5: Generative perplexity (4) and entropy (1) with pretraining from scratch. We perform un- constrained generation of L 1024 tokens using MDLM (Sahoo et al. (024) with binary masking or our SM. Evaluations are tabulated by varying NFE budgets. For unmasking, we use either the standard the more recent ReMDM (Wang et al. (025);; the highest scores are bolded. Gain shows the performance improvement between the SM and the baseline MDLM.
Gradient updates
Forward passes
Generative Perplexity L
Entropy
Unmasking
Foedback
1/8
1/8
Binary
IM
IM
60.02
51.95
52.36
50.16
5.508
5.482
5.461
5.150
Our SM (iso-compute)
0.5M
1M
41.08
31.97
27.36
24.63
5.496
5.448
5.409
5.374
Standard
Gain
18.93
22.98
-24.99
-25.83
Our SM (iso-update)
IM
2M
39.61
30.74
26.1
23.53
5.488
5.438
5.398
5.357
Goin
20.41
-24.21
-26.23
-26.93
Binary
IM
IM
42.53
31.05
21.75
28.62
5.424
5.336
5.209
5.368
Our SM (iso-com) coo-comppte)
0.5M
IM
29.90
18.08
11.40
17:29
5.424
5.334
5.234
5.349
RcMDM
Gain
-12.63
-12.97
-10.35
-11.33
Our SM (iso-update)
IM
2M
29 62
17,58
10. 85
16.72
5.116
5.323
5.217
5.338
Gain
-12.91
-13.48
-10.90
-11.90
AR (T 1021)
12.1
5.22
Data
14.8
5.44
Table 6: Generative perplexity (L) with pretraining continuation. We perform unconstrained gener- ation of L = 1024 tokens using MDLM (Sahoo al. (024) with binary masking or SM. For unmasking, we use either standard or ReMDM (Wang et al. 2025); the highest scores are bolded. Gain shows the performance improvement between the SM and the binary MDLM with pretraining continuation.
Number function evaluations (NFEs)
Pretraining steps
Unmasking
Feedback
1/8
1/4
1/1
Binary
IM
60.02
54.95
52.36
50.46
Binary
IM+100k
59.99
54.71 55)
52.15;10.89)
50.63
Standard
Our SM (iso-compute)
IM+50k
51.10
(0.25
39.44(20.77)
37.46;=0.80
Gain
-8.89
-11.46
-12.7]
-13.18
Our SM (iso-update)
IM+100k
50.99 (.0411
42. (.0.5))
(8.56(22..4)
35.81(#1.24)
Gain
-9.00
-11.97
-13.59
-14.82
Binary
IM
42.53
31.05
21.75
28.62
Binary
IM+100k
42.85
(1.07(22030)
21.74:
28 (00.33
ReMDM
Our SM (iso-compute)
IM+50k
39.61 (2.087)))
26.29 (00
17.65
22.47 (=0.38)
Gain
-3.24
-4.78
-4.08
-6.18
Our SM (iso-update)
IM+100%
39.52 0:0
25. +0.22)
17.23 (60.36)
22.10
Gain
-3.33
-5.14
-4.51
-6.54
Table 7: Entropy (1) with pretraining continuation. We perform unconstrained generation of L 1024 tokens using MDLM (Sahoo al., 2024) with binary masking or SM. For unmasking, we use either standard or ReMDM Wang et al. 2025).
Number of function evaluations (NF-Es
Pretraining
Unmasking
Feedback
steps
1/8
1/4
1/2
Binary
5.508
5.482
5.464
5.450
Binary
IM+100k
5.500))0.000)
(.1.7
5,458 000)
5.447
Standard
Our SM (iso-compute)
IM+50k
5.534(+0.004)
5.503(40.004)
5,480 uma)
5.167(=0.007)
Our SM (iso-update)
IM+100k
5.542)10.008)
5,509 0033
5,485 004)
5.453 036)
Binary
1M
5.424
5.336
5.209
5.368
Binary
IM+100k
5.422(200007)
3.333
3,200, :0
5.361 5.361(=0.000 0.05)
ReMDM
Our SM (iso-compute)
IM-50k
5.476(+0.006)
5.396 (0.008))
3.302
5.410
Our SM (iso-update)
IM+100k
5.482 (+00005)
5.404
5.31 2
5.4 5.416
vocabulary. As shown, this method can slightly improve the perplexity. Crucially, the softmax mechanism is differentiable (in contrast to the non-differentiable top-k selection), which allows for the backpropagation of gradients the first model pass. However, we did not observe any further
24

---

Published as a conference paper at ICLR 2026
improvements in perplexity when applying the gradients both forward passes (see the "k [ (two updates)").
Overall, decided to use the top-k 3 option the final configuration. The softmax approach, while theoretically interesting, comes with increased compute and memory demands (due to storing [V dimeenniional vectors) and did not show a conclusive benefit in downstream coding performance
(see Appendix C.8
Validation Perplexity (PPL)
Pum
validation Perplexity (PPL)
23.0
==3
1.0
22.75
22.5
(two updates)
22.50
22.0
22.00
21.75
1.00
1.01
1.02
1.03
1.04
1.05
1.00
1.01
1.02
1.03
1.04
1,05
Training Step
1e6
Training Step
(a) SM training share
(h) SM feedback and gradient updates
Figure 5: Ablation study language modeling on OWT. Default SM parameters are pem = 0.8 and k ==3 3.
C.4 LANGUAGE MODELING: INFERENCE SPEED
This appendix analyzes the inference time for both SM and binary masking when unconditionally generating samples using I, = 1024 diffusion steps on an NVIDIA A100 GPU. We use the inference script provided by ReMDM (Wang al. 2025). The setup uses ancillary sampling, where the back- bone's forward pass is not called if the predictive logits have not changed in the previous iteration (i.e., caching). SM configured with 3. We measure the time using Python's cProfile.
Figure shows that SM yields a small overhead of 12.5% (22.26s vs. 19.78 s). One can see that checking the activation change ("torchaalclloee) yields a major overhead in both configurations. Diving a bit deeper. one can notice that the MDLM with SM calls the backbone more often than binary masking (651 vs. 636 calls). This is likely because the more detailed input representation from SM causes the state to change more frequently, resulting in fewer cache hits. Besides, SM slightly increases the complexity on two fronts. First, computing the SM distribution requires total of 1.61 (0.00248s per call), which is dominated by the top-k computation. Moreover, the sparse embedding (a weighted sum of k tokens) is slightly more complex than the standard embedding (a single token lookup), increasing the per-call time for the backbone by 1.7% (from 0.01728 to 0.01757s s per call). summary, the 2.48 s total overhead from SM is primarily composed of the 1.61s for the SM calculation, with the remaining time due 1.7% increase in per-call backbone cost and a 2.4% increase in the total number of backbone calls. Despite the small overhead, the generation quality is significantly increased,
be in Table 2
C.5 LANGUAGE MODELING: SM VISUALIZATION
This appendix provides further insight into SM's decoding dynamics. As an illustrative example, we analyze unconstrained generation trajectory of L = 8 tokens over T = 8 steps. Note that we use standard sampling schedule without caching or confidence-based unmasking heuristics. As shown in Figure 7 SM's confidence (indicated by color intensity) generally increases over the denoising steps. However, high SM confidence does not strictly dictate the final outcome: the top-1 predicted token within the SM evolves over time, and the final sampled token differ from the highly confident SM prediction of the previous step (c.g., the transition from "search" " to "word" at t = for the second token).
25

---

Published as a conference paper at ICLR 2026
Totali 22 22.135 1:35 (899 calo)
Tunction Component
20
Overnedd (Cther)
cumru time (s)
76 120 calls
844 (1024 calla)
Softmmak ng
4.5
10.99s (636 call)
13.44x callo
2.245
SM
SW
Figure 6: Cumulative time for unconditionally generating 1.=1024 tokens on an NVIDIA A A100 GPU using standard unmasking.
1.0
the
the
the find the
t=
find find find find search to find out
0.8
t=
6
search search search search search to find their
SM confidence
t=5
search search search search search to find their
-0.6
[=4
search
search search search search to find their
t=3
search word search engine search to find thelr
0..4
t=2
search word search word search to find their
0.2
[=1
search word search word search to find their
search word search word search to find their t=0
0.0
Figure 7: Unconstrained generation trajectory of L 8 tokens over T steps using MDLM with Soft-Masking (trained from scratch, iso-compute. 500k steps). Green-shaded cells indicate masked tokens where SM is active; color intensity corresponds to the SM confidence, with darker green indicating higher certainty. The text inside these cells displays the current top-1 predicted token. Bold, unshaded text represents tokens that have been unmasked (sampled) and fixed.
C.6 MATHEMATICAL REASONING
We also perform experiments on mathematical reasoning tasks GSM8k (Cobbe al. 2021 and Math-500 (Lightman el al. 2024). These evaluations are shown in Table 8 Both evaluations were performed with 1m-eval eva menttiionhneeeee for GSM8k was 256, and for Math-500 it was set to 512. These values were the same as the default ones used by Dream-7B in their original evaluation scripts. Since the MATH tasks are more difficult, the model often needs more generation space to come to the final answer.
in a zero-shot setting. The max generation length
C.7 CODING: ISO-COMPUTE MODELS
Table SM's performance in the iso-compute training setup. Note that this more restricted setup, the model does only see half of the data, there is no data repetition in finetuning. While the variance across seeds increased slightly, we still observe consistent gains with SM, primarily low NFE budgets.
C.8 CODING: TOP-k AND TRAINABLE SOFTMAX TEMPERATURE
We also perform ablation tests on For these tests, we finetune four models, each with the exact same configuration, but with one key exception: train each of these models with different
26

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) on math tasks. Evaluations are displayed under varying computational NFE budgets. We finetune the models with seeds and report the mean accuracy (# standard deviation). SM is configured with k=3, Gain shows the comparison between the SM model and the finetuned baseline.
Dream-77 7B
NFE budget
FT steps
Feedback
GSM8K
Math-500
Binary
57.8
14.2
Binary
33.5k
(1.11
(7.2(
1/4
Our SM
33.5k
(2.3(=2.5)
19.8(42.1)
Gain
+2.8
+2.5
Binary
76.0
36.6
Binary
33.5k
(6.5
36.3(+0.8)
1/2
Our SM
33.5k
(9.4(=0.6)
38.8(41.8)
Gain
+2.9
+2.5
Binary
82.0
45.6
Binary
33.5k
(2.9(=10))
(2.7(+114)
I/1
Our SM
33.5k
84.0
41.((112)
Gain
+1.1
-1.3
Table 9: Accuracy (%) on coding tasks. SM has been finetuned the iso-compute training setting. Evaluations tabulated by varying NFE budgets. We finetune the models with 5 seeds and report the mean accuracy (= standard deviation). SM configured with k=1, Gain shows the comparison between the SM model and the finetuned baseline. The best performing model is marked in bold.
Dream Coden 7B (instruct)
Dream 7B (instruct)
budget
Feedbuck
steps
Humannival
HumanEEval
MBPP
MBPP
HumanEval
HumanEvale
MBPP
MBPP
Binary
25.0
25.0
27.4
29.4
18.9
17.1
26.6
30.2
1/4
Our SM
Binary
16.75k
33.5k
28 Sraa.:
30.4 30.4
29.0;11-3)
27.7:a1.)
27.8
(11.0)
28.8 28.8
21.77
19.0
24.8
22 22.471477
15.9
27 (2200
28. 28.2
36. (6..
29.2
+1.9
+1.3
+1.9
+5.8
+6.5
Binary
54.9
50.6
$1.6
51.3
31.1
29.3
42.8
45.8
1/2
Binary
33.5k
63-8 683-8
(9.4(=2.1)
49.3 49.3)=1.))
$4.3;=1.9)
19 (n
(+)0.s)
53 33-2(e11.0
(
33 33.0
(e2.(
33.9 33.9 +22.55)
43. Tres-4)
44.3 (+2.4)
39
53.6
Guin
+3.6
+3.0
0.0
+2./
+3.5
+4,44
+1.2
+..0
Binary
75.0
69.5
65.8
70.4
57,9
53.0
57.8
63.5
Rinary
7
68 9.+2.00
(9.5(e18:
53 63.0
58 (10011
62 (e
Our SM
16.75k
(5.9
68.5:20.7)
66. 66.6
68 6
58 58.0
50.7
57 ((.8
62.2 62.2
Giin
+0.2
-1.5
-0.6
k-value [1,3,5, 10). We use the same seed for all models to ensure the same training data and initial setup.
We also train fiftth model with trainable softmax temperature. This method uses probabilities calculated with a learned temperature instead of using the top-k predicted tokens, letting the model scale the range of it's own feedback. In the results table, call this ablation k [V]
The results given in Table 10 illustrate a degrading performance with higher values, with k and A 3 having the highest average performance. However, all ablations perform better than both of our baselines on average. This further illustrates the success of our proposed method.
C.9 CODING: TIME- DEPENDEENN MASKING
We explored three methods of time-dependent (TD) feedback. By time-dependence, we mean seal- ing the amount of SM feedback as a function of the point in the decoding process (i.c. t). The basic assumption here that, the model may benefit from having more less feedback at different steps in the diffusion process. For the following, let t T, ..... be our current denoising step, with T being the first step in the reverse process. Let (p) = Wa a waa H(pp-z) ws) be the default (non-time-dependden)) defined in equation
27

---

Published as a conference paper at ICLR 2026
Binary feedback
SM feedback with top-k
No FT
33.5 FT steps
33.5k FT steps
NFE budget
Task
h = 5
10
[
1/4
18.9
17.7
25.6
23.8
24.4
20.1
25.0
Humaneval
1/2
31.1
33.5
35.4
36.0
35,4
34.1
34.1
1/1
57.9
57.3
60.4
56.1
60.4
63.4
58.5
1/4
26.6
27.2
32.8
31.4
27.8
27.6
27.2
MBPP
1/2
42.8
43.6
49.6
48.0
45,4
45.6
44.4
1/1
57.8
58.4
56.2
57.6
57.2
56.8
57.8
1/4
57.8
60.0
63.3
62.9
63.3
60.9
62.3
GSM8k
1/2
76.0
78.7
80.3
79.8
80.9
81.7
78.9
1/1
82.0
83.5
82.9
84.6
84.2
83.6
84.0
1/4
14.2
18.6
21.8
22.0
20.6
19.2
20.2
Math-500
1/2
36.6
37.0
35.4
41.4
40.4
39.0
37.2
1/1
45.6
42.0
38.4
41.2
43.2
43.2
44.4
Avg.
All
45.6
46.5
48.5
48.7
48.6
47.9
47.8
Table 10: Comparison of different finetuned (FT) models, each trained with a different value. The A | ablation is trained and evaluated with learnable softmax temperature. Evaluations are performed on three computational budgets and four math and coding evaluation tasks. The binary baselines also included for comparison. We see that k 1,3 and perform best, with degrading performance at higher & values. The best performing model is highlighted in bold, and the second best is underlined.
1. No TD. In this adaptation, apply no time-dependent feedback modification. SM is applied exactly as described above.
2. Stepwise TD feedback function.
This time-dependent feedback defines threshold value 0 < t' < T. At the threshold, the model switches between SM and Binary as follows:
Binary-SM stepwise TD function: X X'(p/11) = g(p) 1(ge))
SM-+Binary stepwise TD function: X X((p g(p) 1(22))
3. Linear TD feedback function. For the last formulation, we add linear time-dependence to the feedback magnitude. Again, this entails model switch from SM to Binary, or vice versa. The switch happens with linear transition function:
Binary- SSM linear TD function: \l(p) (006:)=(11))) [s(p)]
SM SM-+Binary linear TD function: X \((p
[p)
T
Note that these expressions are written this way because, in the reverse process, the time index decreases from 7' + 1. Although we did not explicitly finetune the models to incorporate external time-dependence (TD) the feedback function, we perform ablation studies on the TD during inference.
C.9.1 EARLY VS. LATE STAGE SM IMPACT
We first look at simple comparison of whether SM is more beneficial at the early or late stages of denoising. To test this, SM-:Binary and the Binary-SSM stepwise TD functions. We define each with thresholds set such that 80% of denoising steps use SM and the remaining 20% use Binary:
the stepwise feedback function: Specifically, compare the
Binary- SM stepwise TD feedback with threshold t' 0.8: SM only during the last 80% of the denoising steps.
28

---

Published as a conference paper at ICLR 2026
2. SM-> MM--rr stepwise TD feedback with threshold t' 0.2: SM only during the first 80% of the denoising steps.
The results given in Table 11 evaluate both approaches, comparing them with both binary baselines. We can clearly see that ablation (2) performs significantly better than ablation (1), suggesting that SM is necessary during the early denoising steps.
Binary feedback
SM feedback TD
No FT
33.5 FT steps
33.5k FT steps
Task
Binary-
SM SM-+Binary
Humaneval
18.9
(11.7
17.4(10.9)
(4.8
MBPP
26.6
(1..6
25..
30.9 (10.5)
GSM8K
57.8
59.5(21.1)
59.3 (9..(
61.6
Math-500
14.2
(7.3(
18.2
20.2 (0.2(111.))
Avg.
29.4
31.4
30.0
34.4
Table 11: Comparison of whether SM is more beneficial at the early or later stages of the denoising process. Both ablations Binary-+SM and SM--Binayy involve a denoising process in which 80% of the steps are with SM and the remaining 20% are with binary masking. We find that placing the SM steps at the beginning of denoising results in much greater performance boost. These ablations are compared with our fully binary baselines. The two ablation columns are mean performance of the finetuned SM models with k=3 and a stepwise time-dependence. For each task, the best performing model is highlighted in bold. These evaluations are performed at an NFE budget of 1/4.
C.9.2 WHEN IS SM MOST BENEFICIAL?
After confirming that SM is necessary during the early denoising steps, we can start looking at the exact steps of the denoising where SM is more beneficial. To further understand this time- dependence, we compare five forms of our SM-Binary TD feedback: (1) no time-dependence (TD): (2) linear SM SM-Binary TD; (3)-(5) SM SM-+Binary stepwise TD feedback with thresholds of 0.2, t' = 0.5, and 0.8.
The results given in Table 12, We tabulate with the of the finetuned binary models show that all given TD ablations of SM still perform better than the binary feedback.
Binary feedback
SM feedback with top-k
No FT
33.5 FT steps
33.5k FT steps
Task
No TD
No TD
No TD
Linear
1' = 0.2
1' = 0.5
I' = 0.8
Humaneval
18.9
(9.0(1
24.8
24.8 22.4(11.2)
24.8 24.8(11.1)
MBPP
26.6
27.0(+1.6)
30.8(+0.5)
28.4(+1.0)
30.9(40.5)
30.7
30.6(40.0)
GSM8K
57.8
59.5(41.1)
62.3
61.6(20.8)
61.6(
61.8(422.)
59.8(+119)
Math-500
14.2
17.3(112)
19.8(42.1)
20.4(11.4
20.2
19.8(11.4)
18.2(11.8)
Avg. 29.4
31.4
34.4
33.2
34.4
34.3
33.3
Comparison of varying forms of TD. All TD functions transition from SM SM-Binnry with different processes. These ablations are compared with our binary baselines. All SM abla- tion columns are the mean performance of the finetuned SM models with k=3 and the associated TD. For each task, the best performing model is highlighted bold.
Table 12:
1. CODING: LEARNED SM-FEEDBACK PARAMETERS
add the SM-feedback parameters: SM-scaling Wat SM- offset = wp. and SM-steepness = WP to the computation graph, allowing the optimizer to train these parameters. We also add the softmax temperature for our k |V] ablation (Section C.8 The
As discussed in Section 3.1
29

---

Published as a conference paper at ICLR 2026
progression of these parameters throughout the full training of our SM-FT models is illustrated in Figure | We find that the learned scale is much lower than in the language modeling. This is likely due to the absence of time conditioning. Since the model has no other cues telling it at what stage at in the decoding and/or which tokens are actually masked, it relies on the existence of the mask tokens for this information.
0.30
Transparency Scale uk
0.25
Transparency Offset Glp
-
-3
0.15
0.10
-5
0.05
0.00
Number of Training Steps
Number of Training Steps
1.0
10
0.9
Transparency Steepness
Softmax Temperature
0.7
Mnw
0.6
0.5
0.4
0.3
0.2
5000 10000 15000 20000 25000
30000
35000
5000
15000 20000 25000 30000 35000
Number of Training Steps
Number of Training Steps
k=1
k=3
k=5
k=10
tunable temperature (k [V])
Figure 8: Plots of all four tunable hyperparameters over all of my SM-FT runs and ablations. We show the evolution of the SM-scale (top-left), the SM-offset (top-right), the SM-steepness (bottom- left), and the tunable softmax temperature (bottom-right) (for the k = V ablation described in Section C.8
D CONCURRENT WORKS
Concurrently with the development of our SM, the following works also aim address binary MDLM's weakness in discarding valuable information during the denoising iterations.
Continuously Augmented Discrete Diffusion (CADD; Zheng et al. 2026) models the forward dif- fusion transition function as a product of a discrete (masking) and a continuous (Gaussian) diffu- sion process. This formulation permits closed-form calculation of the forward process marginals, thereby enabling training with single forward pass. In CADD, the input to the denoising model superposition of the [MASK] embedding. the embedding of the current top-1 token estimate, and a noise vector drawn from normal distribution. The authors also propose a "soft" variant that computes convex combination over the entire vocabulary (our k = |V]). However, they do not show any downstream performance on language modeling and coding due to the high computational costs. contrast, our SM covers the entire spectrum between hard (top-1) and soft through top-k superposition, making beneficial trade-off between generation quality and computational cost (see Appendix C.5) Moreover, while CADD employs a deterministic schedule to define the weighting of the estimated mean vector, SM utilizes a learnalle, confidence-bert based weighting mechanism. Table 13 compares the natural language modeling performance of our SM against CADD. Our SM achieves notably higher MAUVE scores in both iso-compute and iso-update settings. A similar trend
30

---

Published as a conference paper at ICLR 2026
MDLM
CANDI
MDLM repr. (Tm1, top-pm0.9. 0.9)
(iso-update)
CADD
NFE Budget 1/16
NFE Budget 1/8
10
10a
Generative Perplexity
10:
10
10a
10a
10a
Sample Entropy
Sample Entropy
Figure 9: Comparison to CANDI (Pynadath et 2025) and CADD Zheng al. 2026) on uncon- strained text generation. MDLM and CANDI results were visually extracted from (Pynadath et al. 2025). Our SM yields the best trade-off between generative perplexity and entropy.
observed for the generative perplexity. Finally, differences evaluation setups (qwencoder-eval vs. eeevlluation-hareeee and base models (DiffuCoder vs. DreamCoder) render a direct compari- son on coding benchmarks difficult.
Table 13:
Comparison to concurrent CADD Zheng et al. 2026 All methods perform uncon- strained generation with sequence length L 1024. Evaluations are tabulated by varying NFE budgets. Standard unmasking is used; the highest scores are bolded.
MAUVE
Generative Perplexity
Batch size
Gradient updates
Loss tokens
Feedback
tokens
1/8
1/4
1/2
1/1
1/8
1/4
1/2
1/1
MDLM
524B
0,036
50.5
CADD Zhonn et al. 2026
256
2M
5248
2628
0.017
0.080
0.140
0.210
55.3
19.6
46.1
11.6
Our co0--ppp))
512
0.5M
524B
1318
0.143
0.417
0.498
0.596
41.1
32.0
27.4
24.6
Our SM (iso update)
512
IM
1049B
262B
0.155
0.383
0.535
0.602
39.6
30.7
26.1
23.5
Continuous and Discrete Diffusion (CANDI: Pynadath et al. 2025)
analyzes the limitations of con-
tinuous diffusion models on discrete text through framework based on token identifiability Similar to CADD, CANDI merges discrete and continuous diffusion via the product of the respective transi- tion functions. The score function is approximated via a single-step Monte-Card estimation, which uses the sampled token embedding. The denoising model is conditioned current estimate, Gaussian noise, and a mask token, weighted by a fixed hyperparameter (A). While allowing for closed-form marginal computation and thus single-pass training, their formulation does not involve any confidence- based weighting. Experimentally, the authors find that the unconstrained text generation is highly dependent on their sampling temperature. They show that CANDI achieves a favorable trade-off between generative perplexity and entropy when varying the temperature. Our sampling does not vary the temperature, relying solely nucleus sampling with top-p=0.9, follow- ing the setup of (Wang et al. 2025). With this setup, our SM yields superior perplexity-entree trade-off, shown in Figure
superposition of the
Jo et al. (2226) describe the information loss in binary MDLMs as sampling wall, and propose to break this wall with their Loopholing method. Loopholing directly feeds the denoising model's output latent representation (prior to readout with embedding table). The denoising model is then conditioned on the superposition of the current tokens (masked and decoded ones). Similar to SM, their training relies on self-conditioning with a conditioning probability p. However, Loopholing has not been successful in pretraining continuation, requiring training from scratch. We hypothesize this limitation stems from feeding back the final latents directly (causing embedding space mismatch) without any scaling. In contrast, our learnable feedback operates in a known embedding space, enabling pretraining continuation and fine-tuning. Consequently, Loopholing results are limited to
31

---

Published as a conference paper at ICLR 2026
small-scale language modeling and reasoning tasks. On OWT, they achieve notable test perplexity improvement (21.9), which is nonetheless slightly worse than our SM (21.47) (see Table 4). Their results unconditional text generation not directly comparable to ours (showing overall much higher generative perplexity), which is likely due to their removal of time conditioning.
Zhou al. 2025 theoretically show that continuous diffusion models more expressive than masked ones, arguing that the continuous model's achievable state trajectories are a superset of those of masked model. In order to improve MDLMs, which are more popular in practice, Coevolu- tionary Continuous Discrete Diffusion (CCDD) proposes mixing discrete and continuous diffusion a product of the two forward distributions. The denoising model takes input the concatenation of the discrete and continuous representations (along the sequence dimension), predicting both the continuous and discrete representations. This doubles the input sequence length, thereby increasing compute and memory demands during training and inference. In contrast, our SM merges con- tinuous and discrete representations superposition with minimal computational overhead (see Appendix C.5).
Similarly, Latent Discrete Diffusion Models (LDDMs) (Shariatian et al. 2025) enrich MDLMs with latents. The discrete and latent representation can either evolve simultaneously (FUlly JoInt De- noising; FUJI) or sequentially with the latent first (SEQuential denoising; SEQ). The two channels (discrete and latent) interconnected through multi-modal model or additional MLPs, which increase the computational complexity.
Finally, dInfer (Ma al. 2025) introduces iteration smoothing, which adds convex combination of softmax probability to the masked tokens. The weighting of the feedback increased over the decoding iterations based fixed schedule. While elaborate ablations are missing, authors report an effective decrease of 30-40% of decoding iterations thanks to iteration smoothing.
32


Reviews:
[{"review_id": "kMEUkZpJO9", "reviewer": "Reviewer_wkhr", "summary": "This paper introduces soft-masking, a method for improving masked diffusion LLMs by blending the mask state with a soft prediction from previous step before feeding into the denoiser network. It shows strong empirical improvements over models without soft-masking in continued pretraining.", "questions": "Please see above weaknesses (i will consider increasing the score if they are sufficiently addressed)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The motivation is strong. The work addresses a key weakness of masked diffusion LLMs - where the state is a binary decision and cannot represent superpositions of different tokens. \n\n* The proposed fix is sensible and do not change the formulation of the probabilistic model, i.e., the model can still be trained with standard ELBO objective, with additional modification in the input of the neural network. \n\n* The empirical improvements over pure mask models in continued pretraining is very strong (PPL 23.14 -> 21.63 vs. 23.14 -> 22.88). When applying this method to finetuning Dream coding models, The performance gain is also significant, especially in the high-throughput settings which is the main regime people prefer diffusion over AR LLMs.", "weaknesses": "* The proposed method requires two evaluations of the denoiser network in each training iteration. This makes the comparison to pure masked models unfair (the latter only need one network evaluation) given the same batch size. I would expect to see a comparison that matches the training flops. \n\n* There is a very closely-related prior method \"self-conditioning\" that the authors failed to prominently highlight. Although it is briefly mentioned in the related work, given the smilarity of the two approaches (in fact the soft-masking can be seen as self-conditioning in discrete state spaces), in my opinion they should be discussed in more detail (the authors should point out the similarity in the method section, e.g., the two-forward-passes training).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MUJx3s0O3Y", "reviewer": "Reviewer_rBVW", "summary": "Masked diffusion models have been recently introduced to text modeling, where the forward process is defined by transforming the valid tokens into [mask] tokens.\nDespite their great potential, the mask tokens may result in information loss during the denoising process.\nThe authors propose a framework, called soft-masking, to assign non-trivial embeddings for the mask tokens, which can retain the semantic information of the partially decoded sequence.\nThe authors also give a practical training pipeline, which finetunes a pre-trained masked diffusion model.\nThe experiments on unconditional text generation and code generation demonstrate the superiority of SM compared to baselines.", "questions": "- In the training pipeline, the first model pass (Line 6, Alg 1) does not propagate the gradient. Why? How will the result change if we turn on the backpropagation in the first model pass?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The presentation is clear and easy to follow.\n- Although the information blank issue has been observed by concurrent works, I find the idea of using the mixture of top-k token indexes as mask token embedding is novel.\n- The authors validate the effectiveness of the proposed SM method through text and code generation experiments, which supports their claim.", "weaknesses": "- The training methodology in Section 3.2 lacks a detailed derivation. Since the authors introduce an embedding for masked tokens, both the forward process and backward process have been changed. The authors then propose to use the two-pass method to train the the new SM model. However, this training method is rather heuristic and the authors do not provide any theoretical analysis to explain what we exactly do in the training process (e.g., maximizing the likelihood?)\n- The formulation in Line 202-206 only retains the top-k probabilities on the mask position of the current prediction step. This way, it seems that SM works by memorizing the predictions that have been made previously. To exclude the memorization effect, I think an important baseline should be fixing the MDLM pre-trained model and manually tuning the parameter $\\lambda$.\n- As a continuation of the above point, why not use a scheme to memorize all the historical predictions, rather than only the current prediction step?\n- The authors claim that the DLMs beat AR  in terms of accelerated sampling and controllable generation. Can the authors provide more evidence to support this claim?\n- In line 159, what do the authors mean by $f_\\theta=h\\odot g_\\theta$?\n- As the authors do not provide a likelihood estimate formula for SM, they should clarify how they compute the perplexity score for OpenWebText.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TzEeHi2sDH", "reviewer": "Reviewer_GihD", "summary": "This worked presents soft-masked diffusion models. The idea is quite simple, during the de-noising/decoding process, have the mask embeddings at each position retain some information about the top-k possible tokens from the previous de-noising step. This method is generally applicable to most diffusion models, the authors show that you can take an off-the-shelf diffusion LM and and either continue pre-training or fine-tune with their soft-masked method to achieve performance gains with a relatively small amount of compute.", "questions": "1. How much cost does the does adding your SM module add to each forward pass of the model? It seems like it'd be marginal but I didn't see an exact description in this work.\n2. Do the authors have any experiments on pre-training from scratch using this method? Given the results form Fig 3 (the validation PPL decreasing very fast w.r.t MDLM), it seems like MDLM+SM might lead to effeciency gains during pre-training too, do the authors have any evidence or intution about this? (Follow up, when you say \"our MDLM+ SM\", is the only addition the SM module or is there some other differences on MLDM loss compared to Sahoo's original work?) \n3. Related to the above, how many updates steps are needed to update an MDLM model to use MDLM+SM, I see 100k steps of additional pre-training for the 169M paramedics model, how was this number chosen and generally how do the performance gains of adding +SM look for different compute budgets. This is partially touched on in the ablations, but can the authors comment a bit more on this please?\n4. I'm very curious about some of the ablations that were conducted, particularly on the choice of K. Appendix B is a good start, but I'm still a bit confused on a few things. Namely (1) Why does K=1 seem to have way worse Val PPL and other K choices in Fig 5 but in Table 4 K=1,3, and 5 are very similar?  Have you noticed in your training that the choice of K is quite dependent on the task/dataset? Furthermore, in your ablations on the time dependence of SM I see that SM is most useful in the early steps (which makes intuitive sense in my opinion, SM is most useful when there are more masked tokens), but isn't the time dependence also a function of the dataset and the choice of K? I see that using a learned K wasn't particularly useful in this work, but I'd appreciate some comments on this from the perspective of a user trying to choose a reasonable K for a given task.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I'm very positive on this paper! The idea is very simple, but the authors conducted a fairly comprehensive amount of evaluations under various reasonable configurations and showed tangible performance gains. Overall I think this is a good paper and thus lean towards acceptance, although there are a still a few unanswered questions I would like to see answered.", "weaknesses": "Overall I think the paper is well executed. There are a few things that I'd like to also see to get a better idea of the limitations and benefits of this method, I'll defer those to the questions section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jx0XISusDL", "reviewer": "Reviewer_E5oT", "summary": "This paper introduces a soft-masking algorithm for masked diffusion models, replacing static one-hot encodings in inputs with dynamic, context-dependent representations for mask tokens. The algorithm blends continuous feedback from an auxiliary neural network with one-hot encodings, using a confidence-based weighting strategy based on the denoising network's probability outputs at masked dimensions.\n\nThe authors assessed their methodology from two perspectives:\n1. Language Modeling: They investigated if their method could improve the performance of pre-trained MDLMs through additional training steps with SM.\n2. Coding Task Performance: They evaluated the method's effectiveness on coding tasks, specifically when constrained by a limited computational budget after fine-tuning.\n\nAcross both benchmarks, the proposed method demonstrated good enhancements in sample efficiency. Furthermore, the authors emphasized the consistent improvement in performance, particularly when faced with a constrained computational budget during inference.\n\nAlthough the authors claim that this method is suitable for pre-training diffusion language models, no evidence about the effectiveness has been provided.", "questions": "1. Please can you also provide more detail about inference with SM?\n\n2. What do the two sub-columns under Binary-Feedback in Table 4 and 5 correspond to?\n\n3. To make the presentation consistent, it’ll be better to change the absolute NFE numbers in Table 1 to their corresponding ratio, same as in other Tables (3) and (4).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is sound, yielding strong improvements in the fine-tuning tasks. Moreover, the evaluation is well designed.", "weaknesses": "While the idea and novelty are good, the analysis of the underlying mechanism of SM is insufficient. In particular, table 5 shows mixed results for SM feedback with finetuning compared to binary baselines at high compute budgets. Similarly, in Table 2, SM with finetuning underperforms Dream-7B at NFE=1/1.\n\nMy intuition is that soft-masking's \"activation\" of top-k tokens creates a biased learning environment, which could aid constrained inference but hinder generalization. Binary feedback, conversely, offers unbiased exploration for better generalization, albeit with higher computational cost. Analyzing soft-mask probability distribution during inference could clarify the algorithm's mechanism.\n\nThis method, if its low-compute gains stem from greedy top-k token exploitation, might be better suited for finetuning or post-training MDLMs, rather than pre-training from scratch as claimed.\n\nOther weaknesses:\n\n1. In Table 1, in most cases, binary feedback with additional 100k step training performs worse than the baselines with only 1M pre-training steps on MAUVE. Do the authors have any explanation for these results?\n\n2. Following 1, the baseline results on MAUVE with binary feedback seem inconsistent with those reported in Table 1 in [1]. Please can you elaborate?\n\n3. In addition to MAUVE, it’s worth reporting validation perplexity as well, which is a standard metric of language modeling. If the perplexity was biased by the confidence-based weighting, which is a greedy strategy, it’ll be useful to know how much bias this strategy can bring in.\n\n4. Lacking a comprehensive comparison against a wider range of models in language modeling, including AR, other MDLMs with different model sizes.\n\n\n[1] https://arxiv.org/abs/2503.00307", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)