DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
Authors: Zirui Wu
Lin Zheng
Zhihui Xie
Jiacheng Ye
Jiahui Gao
Shansan Gong
Yansong Feng
Zhenguo Li
Wei Bi
Guorui Zhou
Lingpeng Kong
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a simple yet effective method for diffusion language models to perform variable-length generation.

Abstract: Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a critical limitation: the requirement of a fixed-length masked sequence for generation. This constraint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. 
To address this, we propose DreamOn, a novel diffusion framework that enables dynamic, variable-length generation. DreamOn augments the diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into existing DLMs with minimal modifications to the training objective and no architectural changes. 
Built upon Dream-Coder-7B and DiffuCoder-7B, DreamOn achieves infilling performance on par with state-of-the-art autoregressive models on HumanEval-Infilling and SantaCoder-FIM and matches oracle performance achieved with ground-truth length.
Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flexibility and applicability for variable-length generation. 
Our code is available at https://github.com/DreamLM/DreamOn.

Published as a conference paper at ICLR 2026
DREAMON: DIFFUSION LANGUAGE MODELS FOR CODE INFILLING BEYOND SIZE CANVAS
Zirui Wu Lin Zheng's " Zhihui Xie .Jacheng Ye', JJihuu Gao', Shansan Gong',
Yansong Feng'. Zhenguo Li Wei Bi², Guorui Zhou², Lingpeng Kong'
The University of Hong Kong Kuaishhu Technology "Peking University z kiruiwu@pku. edw. en, lzheng20cs. wat.k2eecs..kuhh...
"Huawei Noah Ark Lab
hku.hk, hk, 1pkecs. hhu.hh
ABSTRACT
Diffusion Language Models (DLMs) present compelling alternative autore- gressive models, offering flexible, any-order infilling without specialized prompt- ing design. However, their practical utility is blocked by critical limitation: the requirement of a fixed-length masked sequence for generation. This con- straint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. To address this, we propose DREA- MÖN, a novel diffusion framework that enables dynamic, variable-length gen- eration. DREAMON augments the diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into ex- isting DLMs with minimal modifications to the training objective and no ar- chitectural changes. Built upon Dream-Coder-72 Coder-7B and DiffuCoder-7B, DREA- MON achieves infilling performance on par with state-of-the-ang art autoregressive models
and SantaCoder-FIM and matches oracle per- formance achieved with ground-truth length. Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flex- ibility and applicability for variable-length generation. Our code is available at https: //gittgtttt wigggthuuuhgg. com wwt.e/ow///eee///.......aa Dremonn
INTRODUCTION
In recent years, autoregressive language models have achieved remarkable progress (Comanici al., 2025 AP 2025 Guo et al. 2025 Qwen et al. 2025). They model language as generating text sequentially in a fixed left-to-right manner. While dominant, this paradigm is now being comple- mented by Diffusion Language Models (DLMs) (Hoogeboom et al. Austin Austin et al. 202 Zheng
jtt al. 2023 Singh et al. 2023 Lou et al. 2024 Sahoo et al.) 2024 Shi et al. [0224] Nie et al. 2025
Ye et al. 2025 20255DDepMMin 2025 Labs al. 2025 which have emerged promising alternative
and are gaining significant attention.
DLMs operate through multi-step denoising process, progressively refining masked sequence enable flexible, any-order generation (Austin et al. 2021 Hoogeboom et al. 2021). This property makes them inherently suited for infilling tasks generating content to fill between a given prefix and suffix (Bavarian et al.) 2022; Fried et al., 2023: Allal et al. 2023). In contrast, autoregressive models must resort cumbersome workarounds for infilling, such permuting the target span to the end of the sequence (Fried et al. 2023 Guo et al. 2024a Hui et al. 20224) Seed et al. (025). Such methods not only disrupt the natural contextual structure but also necessitate specialized prompting during training and infenence.
Despite the theoretical advantage, the practical application of DLMS is hindered by critical bot- tleneck: the reliance on pre-specified, fixed-length mask. Current DLMs (Ye et al.) 2025 Nie
ct al. 2025 Xie et al., 2025 Gong et al. 2025b) require the input and output sequences to have
"Equal contribution. "ooresponning author.

---

Published as a conference paper at ICLR 2026
Prefix
Prefix
def correct bracketing(braccett
ann brackets string "e" and ">",
def close elements(mutbers, threshold):
mas closer ues
has corresponding closing bracket. ans
True every bracket
given any threshold.
depth brackets:
idk, enunerate(numbers):
*c*;
Infilling with 4 [MASK]
Infilling with 16 [MASK]
depth closing
Ground
Ground Truth
for 1dx2, elen2 in (
Plff
Suffix
Suffix
depth if depth
abs(elen distance threshold: return
return False
return depth
Figure I: Example of DreamCoder-7B failing at code infilling due to the length mismatch between masked input and ground truth. Incorrect and correct code is marked in red and green. With too few masked tokens, diffusion models lack sufficient room for meaningful code infilling. Too many masks cause over-generation of unnecessary code snippet (e.g., depth that is incorrect).
identical lengths, which prevents them from dynamically determining the length of the output. This limitation is especially damaging for code infilling, where solution lengths can vary significantly across examples. As shown in Figure Bream-Coder-2 (Xie et al.. 2025) produces incomplete or over-generated code when the mask length does not align with the ground truth. More critically, we observe average performance drop of 38% the predefined mask length does not align with the ground truth length (Table 2), highlighting the extreme sensitivity of current DLMs this hyperparameter.
HlumanEval-nfiillinn (Bavarian al. 2022) when
To address this bottleneck, we propose DREAMON, a discrete diffusion language modeling frame- work equipped with adaptive length adjustment ($..I) DREAMON introduces dynamic length adap- tation through two dedicated special tokens, [expand] and [delete], requiring no architectural modifications. We augment the standard diffusion training process with auxiliary length-conte states, allowing DREAMON to be trained with minimal deviation from conventional DLM objec- tives ($3.2). During inference, the model adaptively expands and contract the masked sequence solely its predictions without external guidance ($3.)). Based
(Xie et al.) 2025) and DiffuCoder-7B (Gong et al., (2025b), DREAMON achieves competitive infilling perfor- mance with state-of-the-aa -he- autoregressive models HumanEVal-Infillinn (Bavarian al. 2022) and SantaCoder-FIM (Allal et al. 2023) ($4), and approaches oracle-level performance achieved
with ground truth length ($5.1).
We alleviate the fixed-length bottleneck of diffusion language models (DLMs) by introducing DREAMON, which allows the model to dynamically expand or contract sequences during gener- ation without any architectural changes.
Our method achieves variable-length generation with two special states [expand] and [delete], and supports scalable end-to-end learning of length adaptation through simple aug- mentation techniques with minimal deviation from standard diffusion objectives.
On multiple infilling benchmarks, DREAMON delivers an average absolute performance boost of 26.4% over diffusion baselines, matches the performance achieved with oracle length, and brings diffusion models close to or on par with state-of-the-an autoregressive models.
2 PRELIMINARY
Let X = [x|].... ...
.i.....,ppp x be sequence of N discrete tokens sampled from the data distribution 9xx).
Each token takes values from a vocabulary of size V + 1, consisting of regular symbols plus an additional absorbing state [mask]. We represent each token Xg, well as the absorbing
2

---

Published as a conference paper at ICLR 2026
Forward Augmenlation and Noising
Zo
Dillusize
despunnll
Infiling
despandl
<dbletee
Zi
<maski>>
<imasip-
Infilling
<imakki>
<insk''s
bt
<maskj
<tmaskj
Jeloski
<imsk''s
<mask
<mask'ss
Backward Denoising
Infilling
Cenvzs
Diffustee
sespend]>
Infilling
Canrzs
<imask'>
<ideltte
41-1
Diflustoe
<imaskk'
Infilling
Canvas
<imaskj'
Figure Overview of the augmented diffusion process. Top: the forward augmentation-and noising procedure maps the input sequence Xn to an augmented latent 2o containing [expand] and [delete] states, and then applies a standard masked diffusion process over Zo to obtain z, and eventually 2r. Bottom: single denoising step where [mask] positions in Z, can be predicted as either regular tokens or special states; [expand] deterministically expands into two [mask] tokens, while [delete] will remove the corresponding position, yielding sequence 2 with a different length from Ze.
state [mask], as one-hot vectors in {0, 13 Typical discrete-time masked diffusion models are defined as class of latent variable models over such sequences with forward and backward transition process. In the forward process each token is preserved with a certain probability or replaced by [mask] otherwise, giving q(x, xn) = ax + (1 ar)) [mask] with a prede- fined schedule og, As / increases, the schedule is designed such that the sequence converges to full mask tokens. The generative model reverses this process, starting from xT and applying parame- terized transitions pa X;-1 xx) that approximate the true posterior ((x,11 Xo) This yields po(Xo:r) p(xr) [[= Po (x x().
This class of generative models can be generalized to continuous-time parameterization by consid- ering [0, 1]. which avoids the bias introduced by predefined discretization over time steps. We adopt the frameworks in Kingma et al. (202T); Campbell et al. (2022); [Shhoo et (2024) y; [Shi et al. (2024); Ou al. (2025) and train po with a weighted cross-entropy loss objective,
((0) -E xx(=g(x))
w(t)
Σ 1 |x [mask]] log log xx)
(-.1(
==1
where the indicator 1 [pxy= (mesk:] implies the loss is only evaluated on masked positions, and w(t) e (0, is a time-dependent weighting term derived from the noise schedule or (Shi et al. 2024 Gong et al., 2025a). This objective provides tractable variational upper bound the negative log- lieelikeldoood and serves as an effective training target for large-scale diffusion language models.
METHOD
In this section, present formulation for extending masked diffusion models beyond fixed- length generation. We begin with an overview of our framework in $3.1 followed by training and inference procedures in sections 3.2 and 3.3 and practical implementation details in $3.4
3

---

Published as a conference paper at ICLR 2026
Algorithm DREAMON Training
Require:
Model parameters Ps, merge rate scheduler S;
1: repeat
2: Sample original data Xo ~ xxx) and time step ( ~ (0, 1);
3: Construct augmented sequence Zo from Xy with [expand] and [delete] with S;
4: Sample masked sequence Z, (z [z)]
5: Compute weighted loss L(0) via eq. (2
6:
Update parameters 0 via VoL;
7:
until convergence
3.1 MASKED DIFFUSION WITH AUGMENTED STATES FOR LENGTHI CONTROL
The key ingredient of DREAMON is to introduce two new special states [expand] and [delete] during the diffusion process. When a token transitions to the state [expand], we will expand into two [mask] tokens at the same position of the sequence; and whenever [delete] state is yielded, the token is removed from the sequence. With proper predictions of these special states, the model acquires native length control.
Simulating Special Transitions via Data Augmentation. To train the model to predict [expand] and [delete], we introduce an auxiliary augmented sequence zo constructed from the original input Xo. The augmentation merges random token spans into [expand] and inserts [delete] into the sequence. For merging, we first sample time step t ~ Uniform(O, 1) and compute set of mask indices M₁ according to the schedule oy. Rather than masking tokens, use M, to gate merging such that only spans of consecutive mask indices in M, will be replaced with [expand], under merging probability controlled by rate schedulers ($3.4). This pseudo- masking process provides finer control over the ratio of special to regular tokens, producing 2g with variable length and balanced mix of regular and special states.
We impose the masked diffusion process Po on For these special states [expand] and [delete: the forward diffusion process always maps them to [mask], ensuring that all such tokens are masked and contribute to the learning signal. By construction, the prediction targets in the masked diffusion loss naturally include [expand] and [delete]. Consequently, the model is now trained to denoise not only regular tokens but also special sentinels, thereby learning length control behavior and enabling variable-length generation without any architecture changes.
Masked Diffusion over Augmented
3.2 TRAINING
Similarly to the masking state [mask] and any regular tokens, we treat [expand] and [delete] as sentinel tokens in the vocabulary, and the model is trained predict them using the objective in Eq. ( During training, however, we observe an imbalance: many [mask] positions correspond to [delete] targets, while fewer correspond to [expand] since each [delete] is trans- formed into single [mask], whereas multiple [mask] tokens are merged into one [expand]. As a result, [delete] tokens contribute disproportionately to the loss. To calibrate this, we in- troduce loss weighting scheme that downscales the contribution of [delete] predictions that their total weight is equivalent to that of a single [mask] prediction. The weighted training loss is then given by
C(0)= -E xxx)(()
w(t) Σ
[pp [mask]] Wn log po zz))
(0.1
zu 22 (xa)
z;nug (2, (z))
with the per-token weight wn, defined as
Nmask
1,
if [delete],
x
(3)
Nmask Nactete +1
if 2 [delete],
Naelete
4

---

Return z. Published as a conference paper at ICLR 2026
where Mask and Ndelete denote the number of [mask] and [delete] tokens in the sequence, respectively. The normalization factor ensures that the expected loss magnitude remains consistent across sequences with varying numbers of deletions.
3.3 INFERENCE
Our inference procedure, outlined Algorithm builds upon the standard masked diffusion denois- ing framework with key modifications support variable-length generation. At each diffusion step, we simultancously predict all masked positions and then selectively re-mask tokens based on predic- tion entropy, following Ye et al. (2025). However, the prior work employs fixed masking schedulers to determine how many tokens to unmask per step, which are ill-suited for dynamic-lenge since they assume pre-specified output length. Instead, we directly control the denoising trajectory by specifying n, the number of mask tokens to denoise at each step. enabling adaptive sequence length modeling. During denoising, predicted [expand] tokens are immediately expanded into two [mask] tokens, while generated [delete
modeling
tokens are removed from the sequence. To en- sure stability and prevent unbounded growth, we enforce a maximum output sequence length Lmaa- The generation process terminates once all [mask] positions have been resolved.
Algorithm 2 Variable-Lengte Generation with DREAMON
Require: Trained model parameters 0, initial sequence length I., maximum length Immax unmask- ing budget n per iteration, and sampling temperature T;
1: for 2, ......... do
2: Initialize z' + [mask];
3: end for
4: while [mask] in do
5: Compute token probabilities p +- po(: 2);
6: if [z] Lmax then
7: Set the probability of [expand] to and renormalize; end if
8:
9:
Select up to n masked positions with highest confidence;
10: for each selected position do
11:
Draw z Categorical(p' /T):
if z [expand] then Replace z|i] with [[mask], [mask]];
13:
14:
else if z [dele [te] then
15:
Remove z [i] from the sequence;
else
17:
Set z|i] + ; end if
19:
Update position indices if length has changed;
20: end for
21:
end while
22:
3.4 PRACTICAL IMPLEMENTATIONS
Span Merging Schedulers. We design empirical mask merging schedulers. (1) Static sched- uler: merges adjacent [mask] tokens with a fixed probability Pmerge and (2) Dynamic inverse scheduler: sets the merging probability inversely proportional to the number of [mask] tokens in the sequence. This scheduler merges less with more [mask] tokens to avoid merging too many tokens that might potentially influence the original performance of the base model. We find that mixture of two schedulers during training yield the best performance as detailed in $52
In practice, observe that performance degrades slightly when there is large discrepancy between the initial masked span and the true target length. This introduces inefficiency during inference, the model must expend numerous forward passes
Broadcasting Deletion Length Predictor.
5

---

Published as a conference paper at ICLR 2026
to adjust the sequence length via incremental expansions contractions. To mitigate this, we in- troduce a training-free free inference-time adaptation method that accelerates convergence. Specifically, whenever the model predicts [delete], eliminate all subsequent tokens to its right if they all [mask] tokens. This mechanism significantly reduces unnecessary computation and improves inference efficiency without sacrificing generation quality.
EXPERIMENTS
4.1
SETUP
We fine-tune DB (Ye et al. 2025). DiffuCoder-7B (Gong et al. 2025b), and DreamCoder- 7B (Xie al. 2025) on the eduucationinnttrueiii subset of OpenCoder SFT data (Huang et al.) 2024). which contains about 110K Python instruction-solution pairs synthesized from high-quality educational data. Our experiments focus on code infilling, where the goal to generate missing surrounding prefix and suffix contexts. During training, randomly split each solution into three segments: prefix, middle, and suffix. The instruction, prefix, and suffix are fixed as context, while diffusion is applied only the middle segment.
spans conditioned
For sequence contraction, we find it sufficient to append a random number of [delete] tokens (from to 64) to the end of the middle segment during training. For sequence expansion, [expand] tokens are constructed with merging probability Pmerge as 0.5, using a 1:1 mix of static and dynamic inverse schedulers. Models are trained for 10 epochs with batch size 128, maximum context length 1024, and learning rate le-5 under a cosine decay schedule with 10% warmup steps. It takes ap- proximately hours to train with 8 H800 GPUs. The compute of DreamOn is only 0.15% compared with the compute for pretraining base model (Ye et al. 2025). During inference we set tempera- ture as 0.2 and top p as 0.9. To prevent excessive growth, we cap mask expansion in DREAMON at Limaz 128. We also disable mask expansion in inference after expanding Imaz times.
4.2 EVALUATION
Baselines. We compare against state-of-the-art autoregressive models pretrained with infilling 7B (Guo et al. 2024b). Qwen2.5-Coder-7B (Hui et al., 2024) and Seed-Coder-8B (Seed al. 2025). For open-source diffission language model baselines of similar scale, we evaluate LL (Ne et al. 2025), Dream-7B (Ye al. 2025), DiffuCoder- Gong et al. 2025b), and DreamCoder-78 (Xie et al. 2025).
objectives, specifically Deepseek-Codeed -orcckk--oder-66775
7B
Benchmarks. We evaluate models on HumanEval-IIfillln (Bavarian et al. 2022) benchmarks. including single-line and multi-line subsets, and the Python subset of Santacoder-FIM (Allal al. 2023). We use the official evaluation scripts to report pass @ for HumanEval Infiiiiiing and exact match for Santacoder-Fin filling templates used during pretraining. For all diffusion models, we set the mask length to 64 by default. DREAMON variants dynamically adjust this length as detailed in $3.4
We evaluate autoregressive language models using their respective
4.3 RESULTS
Table T shows that baseline diffusion models struggle with code infilling due to their fixed-length generation, lagging significantly behind autoregressive models. DREAMON effectively resolves this limitation, yielding an average absolute improvement of 26.4% over diffusion baselines and highlighting its effectiveness model-agnostic enhancement.
Notably, with DREAMON, DiffuCoder-7B and DreamCoder-77 not only match the performance of leading autoregressive models like Qwen2.5-Coder-778 but also surpasses them in the more chal- lenging multi-line infilling benchmark. This demonstrates that equipping diffusion models with our length-adaptive mechanism makes them highly competitive for infilling tasks.

---

Published as a conference paper at ICLR 2026
Table I: Pass@1 on HumanEval-Infillnng and exact match on Santacoder-FIM, comparing open- source auto-regressive and diffusion model baselines. The best results across diffusion models are shown in bold, and the second best are underlined.
HumanEval- Infilling (Pass@
Models
SantaCoder (EM)
Single-line
Multi-line
AR Models
Peepseek-Coder-6.77
73.0
45.7
76.3
Seed-Coder-8B
89.7
59.3
77.2
Qwen2.5-Coder-7B
92.6
58.7
79.8
Diffusion Models
LLaDA-8B
48.3
21.1
35.1
Dream-7B
48.2
21.9
60.3
+ DREAMON
88.6440.4
53.3
73.8
DiffuCoder-7B
53.7
45.0
58.0
DREAMON
92.2
63.1
77.4
DreamCoder-7B
55.5
43.2
59.3
DREAMON
92.1 +36.6
63.8
79.0
Table 2: Infilling performance across different designs for diffusion language models. Oracle: per- formance with the oracle target length for reference. f: We use an AST parser to compute exact match to normalize huge syntactic differences between the model output and the ground truth.
Initial Mask Length
Models
Avg.
Oracle
4
16
32
64
Single-Line (Pass @
Dream Seaam-Coder-7B
24.9
61.2
72.6
62.4
55.5
55.3
93.3
+ DREAMON
88.7
90.6
91.0
91.6
92.1
90.8
91.6
w/o Delete
87.8
77.9
71.2
62.3
37.8
67.4
93.3
w/o Expand
25.1
71.6
88.0
90.9
91.5
73.4
92.5
Multi-line (Pass @ 1)
Dream-Coder-7B
5.5
14.7
27.1
39.4
43.2
26.0
69.0
+ DREAMON
50.2
53.8
56.9
60.9
63.8
57.1
66.6
w/o Delete
44.6
45.3
46.1
46.7
44.7
45.5
67.9
w/o Expand
5.5
16.5
30.7
48.2
61.3
32.4
63.2
SantaCoder- FIM (EM)
Dream-Coders
20.0
26.6
43.5
50.8
59.3
40.0
76.3
+ DREAMON
75.0
76.8
78.4
78.0
79.0
77.4
82.0
w/o Delete
74.2
44.3
40.2
50.0
56.2
53.0
84.2
w/o Expand
22.5
55.0
74.7
77.8
78.0
61.6
78.6
5 ANALYSIS
In this section, we conduct ablation studies to evaluate the effectiveness of DREAMON. All vari- ants are fine-tuned from DreamCoder- 7B and evaluated with initial mask lengths ranging from to 64. We additionally evaluate the infilling results under oracle setting, where the initial mask length matches the ground-trun truth solution length, providing an approximate upper bound for infilling performance of diffusion language models.

---

Published as a conference paper at ICLR 2026
5.1 PERFORMANCE WITH DIFFERENT MASK LENGTHS
Performance Breakdown. As shown in Table DreanCoder-7B without finetuning suffers sig- nificant performance degradation when using fixed mask lengths compared to the oracle-length per- formance, highlighting the strong dependence of infilling quality accurate mask length. By trast, DREAMON achieves near oracle-level performance across a wide range of initial mask lengths. Importantly, the performance gains stem from the combined use of both mask expansion and traction mechanisms. DREAMON maintains stable performance on both single-line infilling and SantaCoder-FIM tasks regardless of the initial mask length. We provide two denoising trajectory examples in Appendix D
Ablation on Length Control. To isolate the contributions of expansion and deletion mechanisms,
we evaluate ablated variants of DREAMON: (1) w/o Expand, disabling mask expansion; and (2) w/o Delete, disabling mask deletion. Removing deletion leads to sharp performance drop longer mask lengths, as the model tends to over-generate and fill all given [mask] ] tokens. On the other hand, removing expansion severely harms performance short lengths, the model cannot dynamically extend mask sequences to accommodate more complex or longer completions. We also observe slight performance decline on long masks without expansion, suggesting that even for longer masked inputs, expansion remains beneficial by allowing fine-grained length adjustments.
5.2 EXPANSION MECHANISM DESIGN
Mask merging strongly affects the number of [expand] tokens and the gap between initial and target sequence lengths. We study this in-depth and evaluate two merge rate schedulers: static scheduler with fixed merge probability and a dy- namic inverse scheduler with merge probability in- versely proportional to the number of [mask] to-
Single-Line Infilling
92
90-
Performance
kens. Using only the static scheduler enables ef- fective expansion, achieving an 88.9% pass rate for
length-4 masks. However, its performance is lim- ited on longer masks.
86
DreamOn
Dynamic Inverse Scheduler
The dynamic inverse scheduler merges less when more masks are present. It achieves higher per- formance on longer masks but struggles with large expansions, dropping to 82.5% on length-4 masks. We find a 1:1 mixture achieves the best overall re- sults, offering favorable balance across various mask lengths (Appendix C).
Static Scheduler
8
32
Length
Figure 3: Ablation on merging rate scheduler design choices.
5.3 DELETION MECHANISM DESIGN
We ablate our design choices for handling [delete] tokens with the following experiments: (1) w/o Loss Balancing: train the model without down weighing the loss on [delete] tokens, treat- ing them equally with other tokens in the loss computation; (2) w/o In-place Deletion: Instead of removing [delete] tokens, keep them in the sequence, similar to generating padding placeholder tokens in standard diffusion language (Nie et al., 2025; Ye al. 2025). To implement this, we ran- domly mask or preserve [del lete] during training; and (3) w/o Deletion Broadcasting: disable the inference-time mechanism described $3.4
As shown in Table removing loss balancing leads to substantial performance drop to 84.6% average pass@1 rate, confirming that down-weighing [delete] loss is essential to prevent the model from overfitting to deletion signals. Keeping persistent [delete] tokens also performs poorly (average 85.3%), indicating that placeholder-lik deletion tokens in the sequence disrupt positional coherence and degrade training. Disabling deletion broadcasting reduces performance by 0.6% on average, especially when the given mask length is much longer than the expected solution. The deletion broadcasting mechanism also accelerates generation by 2.1
8

---

Published as a conference paper at ICLR 2026
Table Ablation study for mask deletion mechanism implementations.
Initial Mask Length
Models
Avg.
Oracle
4
16
32
64
HumanFval-Inmn Single-I (Pass @ 1)
DREAMON
88.7
90.6
91.0
91.6
92.1
90.8
91.6
w/o Loss Balaneing
75.8
82.5
87.0
87.2
90.4
84.6
88.6
w/o In-Place Deletion
85.9
85.7
88.5
84.8
78.0
84.6
93.1
w/o Deletion Broadcasting
88.7
90.5
90.0
90.2
91.4
90.2
91.6
5.4
EFFICIENCY ANALYSIS OF DELETION BROADCASTING
Number of Mask Expansion
Number of Mask Deletion
Total Diffusi on Steps
80
brooddasttngg
brsaadcasting
32
broadcasting
breadcasting
30
28
26
80
24
70-
22
brondrasting w/o troadcasting
60
20
50-
10
30
40
50
60
10
20
30
40
50
60
10
20
30
40
50
60
Mask Length
Mask Length
Mask Length
Figure 4: Average generation steps of DreamCoder DREAMON multi-line infilling subset.
The introduction of broadcasting dramatically enhances inference efficiency, primarily by transform- ing the deletion process from ooken-by-token operation into a length prediction action. Without broadcasting, deletion steps scale almost linearly with the initial mask length because the model must iteratively predict and remove each excess mask token individually. In contrast, DREAMON with deletion broadcasting mechanism keep the number of mask deletion nearly constant around with roughly only step on average. This optimization eliminates the computational bottleneck caused by large discrepancies between the initial masked span and the true target length, reduc- ing total inference steps from as high as 122.8 (w/o broadcasting) to just 52.4 (w/ broadcasting) mask length 64. Consequently, broadcasting not only slashes unnecessary forward passes but also stabilizes overall inference cost, making the generation process both faster and more robust to initialization variance without any impact on output quality.
RELATED WORK
Code Infilling with Autoregressive Models. Code infilling requires generating missing code seg- ments conditioned on bidirectional context, a task inherently challenging for standard left-to-right autoregressive models. To address this, several approaches adapt architectures to better capture bidi-
rectional dependencies Yang et al. (2019); Stern et al. (2019): Gu et al.] (2019a); Chan et al. (201
Welleck et al.) (2019)) Shen et al. (2020) Alon et al. (2020) Nguyen et al. (2023): Shen
2023
A widely adopted alternative preserves the standard left-to-right autoregressive architecture by re- locating the target infill segment to the end of the input sequence, enabling the model to generate the missing code autoregressively (Raffel et al. 22200) Tay et al. [2033] [202]|Anil et al.) 2023 Sun ct al. 2024). This approach is compatible with decoder-only architectures (Bavarian et al. 2022 and
has become the standard in modern code language models, including Codex (OpenAí et al., 2022
INCODER (Fried et al., (023). SANTACODER (Allal et al. (023). StarCoder (Li et al. 2023) and
StarCoder 2 (Lozhkov et al. 2024), CODEGEN 2/2.5 NNijkamp al. 2023), Code-Llama (Roziere (Guo et al. 2024a), CodeGemma (CodeGemma Team 2024). Qwen-
jet al. 2023). DeepSeek-Code
Coder (Bai al. 2023 Hui al. 2024), and Seed-Code (Seed et al. 2025).

---

Published as a conference paper at ICLR 2026
Discrete Diffusson Language Models. Discrete diffusion models have recently emerged as com-
pelling alternative to autoregressive models. Foundational work by Austin et al. (2021 1) Hoogeboom et al. (202T) introduced discrete diffusion processes for text data, enabling probabilistic modeling of token sequences through iterative and bidirectional denoising, Subsequent research has refined these approaches with continuous- time relaxations (Camphell et al., 2022), improved training objectives (Zheng et al., [2223] Lou et al., 2024), and generalized masked diffusion frameworks (Sahoo et al.) 2024 Shi al. 20244 Ou al. 2025). Scaling efforts have produced powerful models such as Plaid Gulrajani&& & Hashimoto| 2023) and LLaDA (Nie et al. 2025). Adaptation techniques leverag-
ing pretrained models, such as Laf (Gong et 20.55) and Dream (Ye al. (2025). have narrowed the performance gap with state-of-the-an autoregressive language models.
Non-autoregressiv Models with Length Control. Generating variable-length sequences re- mains significant challenge for aon-eutoregresssive models. Prior work has explored diverse strate- gies to address this, including learning separate length predictors (Gu et al. 2018: Lee et al.] 2018 Ghazvininejad al., [019] Zheng et al.
contract 2023), marginalizing over latent alignments
sequence length (Chan et al. (020)), incorporating edit operations (Gu et al.
2019a b) Stern et al.,
2019
Johnson
2021 Reid et al.
2023 Campbell al., [0023] Patel et al. 2025 Havasi al.
2025
and performing diffusion over sequence positions (Zhang et al. 2025
Kim et al. (025).
Recent concurrent works also address this challenge. Edit flows (Havasi al. 2025) present discrete flow matching with edit operations over extended spaces for tractable and effective train- ing: DDOT (Zhang et al. 2025) proposes jointly denoise token states and positions for dy- namic segment-length adjustment; FlexMDD (Kim et al., 2025) learns insertion and unmasking rates through a joint interpolant framework over both token states and positions, thereby enabling variable-length generation; and DAEDAL (Li et al., 2025) provides a training-free approach using inference-time prediction confidence scores to dynamically determine the response length.
In contrast, our method implements native length control in masked diffusion models with minimal additional training and no architectural modifications, directly adapting pretrained diffusion lan- guage models. This design preserves the simplicity of the original model, avoids complicated multi- stage inference pipelines, and yields substantial gains in flexibility and performance for variable- length generation.
CONCLUSIONS
In this work, we introduce DREAMON, simple yet effective framework that enables dynamic length control through two special tokens ([delete] and [expand]) without architectural changes. By augmenting the diffusion process with auxiliary length-control states, DREAMON learns to expand or contract sequences based solely on model confidence. Our results show that DREAMON approaches oracle-length performance and achieves competitive results with state-of-the-as autoregressive models. We hope our work can pave the way for more practical and flexible DLMs beyond fixed-size canvas.
Limitations. Currently, we limit our evaluation to focus on code infilling tasks that require strong variable-length generation capabilities. Future work will extend the scope broader applications to assess the generalizability of DREAMON. In addition, the training and inference procedures in DREAMON rely on heuristies enable variable-length generation developing more principled formulation for flexible inference in masked diffusion models is an important direction for future research. Finally, our current design uses single expansion state [expand] that deterministically expands into two [mask] tokens. This choice keeps the output space small and training stable, but requires multiple expansion steps when the target completion is much longer than the initial mask span. A promising direction for future work is to introduce richer vocabulary with multiple expansion factors or to couple expansion with an explicit length- -reett-predittion head. They could reduce the number of denoising iterations, but would also enlarge the decision space and require careful rebalancing of the training objective to maintain well-behaved length ad- justment dynamics. We leave the exploration of these richer expansion schemes to future work.
simple yet effective manner;
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We acknowledge the open-source community for providing high-quality datasets and evaluation frameworks. This research was supported in part by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N. HKU714/21.
ETHICS STATEMENT
Our research focuses on developing a diffusion-basee bassd language modeling method capable of variable-length text generation. We did not collect any data involving human subjects, private infor- mation. And our study does not include any human evaluation. All datasets used in our experiments publicly available benchmarks, and we strictly adhere to their respective usage licenses. Further- more, our method does not present any foreseeable risks of misuse or societal harm.
REPRODUCIBILITY STATEMENT
We have taken deliberate steps to ensure the reproducibility of our work. Detailed descriptions of the experimental setups and hyperparameter configurations are provided $4
REFERENCES
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, al. SantaCoder: Don't reach for the stars! arXiv preprint arXiv:2301 .03988, 2023.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Inter- national conference on machine learning, pp. 245-256. PMLR, 2020.
Rohan Anil. Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin. Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. PaI.M Technical Report. arXiv preprint arXiv:2305. 2005/ 2023.
Jacob Austin, Daniel D. Johnson, Jonathan Ho. Daniel Tarlow, and Rianne den Berg. Struc- tured denoising diffusion models in discrete state-spaces, In Advances in Neural Information Processing Systems, 2021. URL httpt::t////ooo Ttttpsse/openeeeveeeeetttt
id=h7 -
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255. 2022.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-28221 28279, 2022.
Andrew Campbell, William Harvey, Christian Dietrich Weilbach, Valentin De Bortoli, Tom Rain- forth, and Arnad Doucet. drans-dimensional generative modeling via jump diffusion mod- els. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URI. /tttps://opereev ew net forum? wtw.rrrmmmm....773737.33..
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. Kermit: Generative insertion- based modeling for sequences. arXiv preprint arXiv: 0.06601600 01404, 2019.
William Chan, Chitwan Saharia, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. Im- puter: Sequence modelling via imputation and dynamic programming. In International Confer- on Machine Learning, 2020.
11

---

Published as a conference paper at ICLR 2026
Google LLC CodeGemma Team. Codegemma: Open code models based gemma, 2024. URL bttps: http://ttrrgg..oool.oco com/ / codegemna report.pdt
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
DeepMind. Gemini diffusion.
2025.
URL httts:://eepmmldd.ggog/// htw.:m/mmdd..goollmmmmmmeellll
gemini -idiniiidiff
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zeutleooee,, and Mike Lewis. InCoder: generative model for code infilling and synthesis. In The Eleventh International Conference Learning Representations, 2023. URL httpps::/ net how-::id-hhww---aanee
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), 2019. URL httttta://aaaaaahllllee
Gxq/D19-1633/
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi. Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. International Conference on Learning Repre- sentations, 2025a.
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code gen- eration. arXiv preprint arXiv:2506. 206399 2025b.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Nor-autoregressive neural machine translation. International Conference on Learning Representations, 2018. URL n wwrumooriiio.oo forummiddBi1istt..... hattps://cpenne wttp://wwwwweeeeeeeeoeeeeeeeeee.
Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based decoding with automatically inferred
generation order. Transactions of the Association for Computational Linguistics, 2019a. URL
Q19 1019-117 1042/
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in Neural Infor-
mation Processing Systems, 2019b.
Ishaan Guirajani and Tatsunori B Hashimoto. Likelihood based diffusion language models. Ad- vances in Neural Information Processing Systems, 36: 16791-16715,,. 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseck-coder, When the large language model meets programming-the the rise of code intelligence. arXiv preprint 1rXiv:2401.11416. 2024a.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. 2024b.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501. 12944 2025.
Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint 200022506.090/8. 2025.
12

---

Published as a conference paper at ICLR 2026
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Information Processing Systems, 2021. URL
6nbpPqUCT17)
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:241 I.04905, 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Daniel D. Johnson, Jacob Austin, Rianne van den Berg, and Daniel Tarlow. Beyond in-place corruption: Insertion and deletion in denoising probabilistic models. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021. URL
Jaeyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du, Sham Kakade, Timothy Ngo- tiaoco, Sitan Chen, and Michael Albergo. Any-order flexible length masked diffusion. arXiv preprint arXiv:2509.01025, 2025.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506. 17298, 2025.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, 2018. URL https: awl.atlllanntyy..... org/
D18-1149)
Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. Beyond fixed: Training- free variable-length denoising for diffusion large language models. arXiv preprint arXiv:2508. 008/9, 2025.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.00 06161, 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024. URL
https://cpeerevia
net. Forum? A--C.IIIIII
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint A.Ki:22402.11777 2024.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vander- wende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, 2016.
Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. Meet in the middle: A pre-training paradigm. arXiv preprint arXiv:2303.07295, 2023.
Shen Nie, Fenggi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen. and Chongxuan Li. Large language diffusion models. arXiv preprint 22Xi::2502.00992 2025.
13

---

Published as a conference paper at ICLR 2026
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou Code- Gen2: Lessons for training LLMs on programming and natural languages. arXiv preprint arXiv:2305.02339. 2023.
OpenAl.
Openai o3 and 04-mini system card, 2025.
URL https://
cdn ctt..ppena...co///dd222222///3/ 02dc 4789- 800b 1ollb-7758855 77758f372cc1
03 3-
System Card.
OpenAI, Mohammad Bavarian, Angela Jiang, Heewoo Jun, and Henrique Pondé. New GPT-3
Capabilities: Edit and Insert. OpenAl blog, 2022. URL
gpt eedt---etitineerr/
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. International Conference Learning Representations, 2025.
Dhruvesh Patel, Aishwarya Sahoo, Avinash Amballa, Tahira Naseem, Tim GJ Rudner, and Andrew McCallum. Insertion language models: Sequence generation with arbitrary-positee insertions.
arXiv preprint arXiv: 2505:::50550555. 2025.
Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: wtwpaaaavvvv....//vvv//aaaaaaaav org/ abs /21224122222 15115
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:140:1114: 1-140:67, 2020.
Machel Reid, Vincent Josua Hellendoom, and Graham Neubig. DiffusER: Diffusion via edit-based reconstruction. The Eleventh International Conference on Learning Representations, 2023.
URL httpppsss//ppeeeeeeeew..... net www.en/ee/oormnnddddn//9192222
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 22900, 2023.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Thirty-eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. URL Ctl.https:.
ByteDance Seed, Yuyu Zhang, Jing Su. Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, et al. Seed-coder: Let the code model curate data for itself. arXiv preprint ar0iv:2506. 2444022000. 2025.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. Blank language models. arXiv preprint 2rXXi:20022.30079 2020.
Tianxiao Shen, Hao Peng, Ruoqi Shen, Yao Fu, Zaid Harchaoui, and Yejin Choi. Film: Fill-in language models for any-order generation. arXiv preprint arXiv:2310.099300 2023.
Jiaxin Shi, Kehang Han. Zhe Wang, Amaud Doucet, and Michalis Titsias. Simplified and gen- eralized masked diffusion for discrete data. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. htlphtts:: http:///pp///eeeeeeee eww www.nee//eee fert//frrm?tt xcqgofft.4g
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreamu, and Gust Verbruggen. CodeFusion: A pre-trained diffusion model for code generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:://
acl www.annoll...rgg//aa.. emn 1 ma n. 716/
14

---

Published as a conference paper at ICLR 2026
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning, pp. 59.6-995-585 PMLR, 2019.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma. Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint 2rX1v:2403.14734 2024.
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0. % extra compute. arXiv preprint arXiv:2210.11999, 2022.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Met- zler. UL2: Unifying language learning paradigms. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httph::://///..t..ttt http::ss///penzeiew.... netpleeww.eeetteee www.ffooo/fmitttoooooeeo..mm
6rLVLB727MC
Sean Welleck, Kianté Brantley, Hal Daumé lii, and Kyunghyun Cho. Non-monotonic sequential text generation. In International Conference on Machine Learning, pp. 6726. PMLR, 2019.
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao, Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. Dream-coder 7b: An open diffusion language model for code. arXiv preprint arXiv:2509.00 142, 2025.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances Neural Information Processing Systems, volume 32, 2019.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025.
Andrew Zhang, Anushka Sivakumar, Chiawei Tang, and Chris Thomas. Flexible-length text infilling for discrete diffusion models. arXiv preprint arXiv:2506. /3579, 2025.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXtv:2302.057337.022
A THE USE OF LARGE LANGUAGE MODELS
We employ large language models primarily for polishing written text-for example, to correct grammar and improve clarity. However, L.MSs do play significant role in the research tivities, including idea generation, experimental design, or the substantive writing of the manuscript.
B GENERALIZABILITY BEYOND CODE INFILLING
To demonstrate that DREAMON is not limited to code infilling, we further evaluate the model's capability of commonsense narrative understanding on the ROCStories corpus Mostalazadeh et al. (211)). We fine-tune Dream-78 7 (Ye et al., 2025) on the ROCStories training split in two variants: with DREAMON, and an SFT baseline trained following the recipe Ye et al. (022). We consider two setups: (1) Narrative Infilling, where the model generates a missing sentence within the middle of a story, requiring bidirectional context understanding: and (2) Prefix-guided Generation, where the model is provided with the preceding story context and generates the final concluding sentence, serving as a proxy for general completion tasks.
Table 4 presents Rouge-L scores across varying initial mask lengths. In both infilling and prefix- guided settings, the baseline performance degrades significantly when there is a mismatch between the initialized mask length and the natural length of the missing content (e.g., at lengths 4 and 32). In contrast, DREAMON utilizes its madtth-adppiie logic- mediated by [expand] and [eeltte] -to achieve superior performance.
15

---

Published as a conference paper at ICLR 2026
Crucially, method exhibits high stability: the generation quality remains consistent regardless of the initial mask length. This confirms that the proposed eentt-aadptaate mechanism successfully decouples generation quality from the initial mask length, demonstrating robust generalizability to creative, variable-length natural language tasks.
Table 4: Rouge-L scores on the ROCStories corpus across variable initial mask lengths.
Initial Mask Length
Method
4
16
32
Narrative Infilling
Dream + SFT
19.2
29.8
26.5
18.9
DREAMON
31.6
31.4
31.3
30.6
Story Ending Generation
Dream + SFT
16.3
25.1
22.4
16.7
DREAMON
24.5
24.6
24.4
24.1
C ABLATION FOR HYPERPARAMETERS
In this section, we provide the results on DreamCoder-7B with different training hyperparameters. As shown in Figure [5a| the @ass@ score for single-line infilling reaches its peak approxpapxii 90.9%-when -when employing balanced 1:1 mixture of static and dynamic inverse schedulers. This result highlights the substantial performance gain achieved through this synergistic combination. Similarly, the right panel reveals that the model attains its highest Pass score of roughly 90.5% at a merge probability of 0.5. Guided by these findings, we adopt a static/dynamic scheduler mix ratio and merge probability of 0.5 in DREAMON configuration to maximize performance.
Single-Line Infilling
Single-Line Infilling
92
90.0
87.5
82.5
80.0
89
77.5
72.5
87
70.0
Static/Dynamic Inverse Scheduler Ratio
Merge Probability
(a) Result with different scheduler merging ratio. (b) Result with different merging probability
Pmerge
Figure 5: Performance on single-line subset of HumanEvallnfillngg FIM with different hyperparam- eters during training. The performance is computed as average pass@1 with mask length 4, 8, 16, 32 and 64.
D INFILLING EXAMPLES
key advantage of DREAMON lies in its adaptive handling of sequence length variations during inference. This is achieved through two complementary states [expand] and [delete]. First, as depicted in Figure 6. DREAMON possesses the capability to expand mask sequences. This dy- namic expansion allows the model to generate outputs longer than its initial input mask, effectively preventing truncation and enabling the generation of comprehensive sequences. Second. Figure showcases the efficacy of the deletion broadcasting mechanism. This mechanism plays crucial role
16

---

Published as a conference paper at ICLR 2026
def has close
c elements threshold):
Check given are any two numbers closer to each other than given threshold.
for idx, elem in (atteerat((umbbsss)
Prefix
|nask]>
>>>>
< | [
< |aaaa//a5a5....aaa/////aaaaaaaaa
maskk |> | hrtpahayhha.aaaaaaaaaaaaaaaaaaaa
< mask >< [aak|><]expand]]]] ||a_a|1|||5aagaggggaaaaaaaaaaa23
< mask
>< | |1aaaaas55.aaaaaaaaaaaaaaaaaaaa3
||
for idx2< |x for idx2< (ack22/mmssk>>> ><
| |
mask mask
for idx2 elem< < mask >>
mask|>
for idx2 clem<|mask expand


for for idx2 <
| |
for idx2 elem2 [<<]
for idx2 elem2 an</maak|]> [
expand
for idx2 elem2 in< |
for
for idx2 elem2 enumerate (nuunerrtt((mbbrs<<//an
for idx2 elem2 in enumerate (numbers).
Infilling
1= idx2:
distance abs elem2) if distance threshold: return True
return False
Suffix
Figure 6: DREAMON adds mask tokens as needed.
in promoting rapid convergence to the optimal predicted sequence length by selectively removing redundant mask tokens, thereby streamlining the generation process and improving efficiency.
17

---

Published as a conference paper at ICLR 2026
def correct correct_bracket
str): brackets is a string of "c" and ">"
every opening bracket has corresponding closing bracket.
depth =
for b in brackets: "<":
depth
Prefix
< [mask|><
mask >< mask
mask >< mask
>< mask
mask >< mask
< |mask >< mask ><
mask
mask >< mask
>< mask >< mask 1 ><| mask
< mask|><<
mask >< mask ><
mask ><
mask
><| |
< |makk ><
maskk|>
<
< http://hheettinnn.a...///n////nr
< |2tkk//.221ffffaao//////aaa21111 |
||gp|//||||||||//oooaaaaaaa/aaoa m=
mask | | || | | mask
| ">

elif
 == "></mask|>
elif<|mask/> == ">":
Infilling
elif b == ">":
if idx I= idx2:
distance abs (elem elem2)
distance threshold:
return True
return False
Suffix
Figure 7: DREAMON deletes excess mask tokens with the deletion broadcasting mechanism.
18


Reviews:
[{"review_id": "iCev1lieXF", "reviewer": "Reviewer_4aWj", "summary": "Diffusion language models have been a promising direction, especially in code generation because of flexible code in-filling capabilities. However, traditional diffusion language models use a fixed size canvas and diffuse language tokens within those. This paper introduces a novel way to remove the constraints of a fixed size canvas with two special control tokens, [expand] and [delete], allowing the model to autonomously adjust the output length during generation without architectural changes. Experiments show this method removes the fixed-length bottleneck, allowing DLMs to achieve performance on par with state-of-the-art autoregressive models and nearly matching oracle performance where the correct length is known in advance.", "questions": "- For deletion broadcasting, were there any experiments done to make the model learn the start and ends of spans for deletions instead of the heuristic of deleting all mask tokens?\n\n- I'm curious how your method would behave on non-structured language modeling tasks, would it be fair to say that using the expansions and deletions would be a more significant challenge when it's not code?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper  addresses the fixed-length mask, a major practical limitation for diffusion language models with `[expand]` and `[delete]` token solution is elegant, effective, and requires no architectural changes.\n\n1. The paper has very nice presentation, with illustrative examples of the control tokens, and their effects. \n\n1. The paper clearly validates the necessity of both the expansion and deletion mechanisms through detailed experiments and ablations.\n\n1. The proposed method allows diffusion language models to match or exceed the performance of state-of-the-art autoregressive models in code infilling of the same compute level.", "weaknesses": "- The method is presented as a general solution for Diffusion Language Models, but it is only evaluated on Python code infilling. It is unclear if the `[expand]` and `[delete]` logic would translate well to more fluid and creative natural language tasks.\n\n- The deletion broadcasting is an example of a \"training-free\" efficiency, but it seems like it would be better suited if this were learned in a data-driven manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUeoMbELWf", "reviewer": "Reviewer_Pntm", "summary": "The paper proposes DreamOn, a simple extension to diffusion language models that enables variable-length generation by introducing [expand] and [delete] control tokens. It eliminates the need for fixed mask lengths, significantly improving infilling performance on code benchmarks and matching oracle-length results.", "questions": "- Could the authors clarify the discrepancy between Table 1 and Table 2 regarding the mask length = 64 setting?\n- Is there any safeguard (e.g., termination condition, token-level constraints, or training regularization) to avoid infinite loops or erratic length changes caused by conflicting [expand]/[delete] predictions?\n- The method is evaluated only on infilling tasks. Can DreamOn be extended to general unconditional or prefix-guided generation tasks, where target length is also unknown a priori?\n- Would it be possible to report the actual FLOPs or computational cost in Table 1 to better assess the efficiency trade-offs of dynamic length adaptation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Addresses a critical and practical limitation of diffusion language models—fixed-length generation—with a minimal, plug-and-play solution.\n- Achieves strong empirical results, closing the performance gap with autoregressive models and matching oracle-length performance on standard infilling benchmarks. Table 1's performance is impressive.\n- Requires no architectural changes and integrates seamlessly into existing DLMs with only minor modifications to the training objective.", "weaknesses": "- Clarity of Figure 2: The current illustration of the diffusion process in Figure 2 is hard to follow. It would be significantly clearer if the forward and backward processes were presented separately, with explicit depiction of how [expand] and [delete] tokens interact with the sequence during denoising.\n- Inconsistent or Missing Baselines: The paper mentions evaluating LLaDA (line 305) but does not include it in any experimental results (e.g., Table 1 or 2), making it difficult to assess comparative performance. Additionally, there appears to be an inconsistency between Table 1 and Table 2: Table 1 claims all experiments use an initial mask length of 64, yet the performance reported for mask length = 64 in Table 2 does not align with the corresponding numbers in Table 1.\n- Lack of Analysis on Length Control Stability: While [expand] and [delete] tokens enable dynamic length adjustment, the paper does not discuss mechanisms to prevent unstable behaviors, such as oscillation (e.g., expanding a segment only to delete it in the next step) or unbounded expansion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUWCBhjtxw", "reviewer": "Reviewer_bMyK", "summary": "This paper identifies the fixed-length mask limitation of diffusion language models for code infilling and proposes DREAMON, a lightweight augmentation that enables dynamic, variable-length generation. DREAMON trains diffusion models to predict two special sentinel states, expand and delete, so the model can autonomously lengthen or shorten masked spans at inference without architectural changes. The authors demonstrate large gains on code-infilling benchmarks, showing near-oracle accuracy and much faster, more robust generation.", "questions": "I didn’t have time to read all the details carefully.\n1. In a single inference run, is the length that `[expand]` produces a fixed value? If so, what happens if fewer tokens are actually needed at that point, and how sensitive are results to this parameter?\n\n2. Is it possible for `[expand]` to expand into `[delete]` (i.e., can the expansion produce delete sentinels)?\n\n3. Have the 7B DLMs already been SFT-trained on OpenCoder, and are the training settings the same when you add DREAMON? The single-line improvement looks unusually large — how should that be explained?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a clear mechanism (expand/delete sentinel states) that gives masked diffusion models native variable-length control without changing model architecture. \n\n2. Empirical results show strong improvements on code-infilling benchmarks, achieving near-oracle accuracy and robust generation compared to prior diffusion and autoregressive baselines.", "weaknesses": "1. I feel the baselines are weak — simply plugging this into a DLM will almost certainly yield improvements. You should compare against other methods applied to DLMs to see whether DREAMON still has an advantage. Or are you the first to implement dynamic-length generation on diffusion language models?\n\n2. Claiming in Contribution 1 and 2 that you’ve solved the fixed-length bottleneck of DLMs feels overstated, because the validation is only on code infilling. If you want to make that claim, you need to validate on a more diverse set of tasks.\n\n3. As you note in lines 466–469, your advantages over concurrent work are low training cost and no architectural changes; however, the DAEDAL method you cite in lines 464–465 is training-free. I know that is concurrent work, but the advantage is not accurately stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h6Z1ZdUzJs", "reviewer": "Reviewer_SUPh", "summary": "This paper solves the fixed-length mask limitation of code infilling diffusion models by introducing [expand] and [delete] tokens. The training process is adapted to first (synthetically) introduce these tokens in the sequence before embedding and predicting each token. Experiments show that diffusion models with these tokens perform significantly better than the original models, and even perform almost on par with auto-regressive models with infilling objective (pass @ 1).", "questions": "- How does the performance evolve in function of the length of the final generation? I'm supposing that longer masks, which require more [insert], will perform worse?\n- If the above holds, would extending [insert] into [insert1], [insert2], ..., [insertX] make the process converge in fewer iterations? In table 2, the multi-line performance has a big jump to the oracle performance, indicating that requiring many insertions is a bottleneck for longer code snippets.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Controlling the size of the generation is a fundamental challenge in discrete diffusion models for text. Approaches that do not require (large) architectural changes can leverage pre-trained models more effectively.\n- The ablations focus on relevant design decisions, such as not using either of the tokens.\n- The paper is properly structured, well-written and clear.", "weaknesses": "- The paper mentions but does not compare against other approaches. FlexMDM and DDOT are both applicable to infilling and can be directly compared with, and their code is available.\n- The paper considers only pass@1. CodeFusion [1—missing reference] shows that code diffusion models excel at higher pass@k rates, because of the model's increased diversity: compiler and execution feedback can be used to select candidates.\n\n[1] Singh, M., Cambronero, J., Gulwani, S., Le, V., Negreanu, C. S., & Verbruggen, G. (2023, November). Codefusion: A pre-trained diffusion model for code generation. In The 2023 Conference on Empirical Methods in Natural Language Processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)