Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
Authors: Fangyu Ding
Ding Ding
Sijin Chen
Kaibo Wang
Peng Xu
Zijin Feng
Haoli Bai
Kai Han
Youliang Yan
Binhang Yuan
Jiacheng Sun
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion language model
deletion-insertion process
denoising score entropy
Primary area: generative models

Abstract: While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) $\texttt{\<MASK\>}$ tokens inherent to its paradigm, and 2) $\texttt{\<PAD\>}$ tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.

Published as a conference paper ICLR 2026
BEYOND MASKS: EFFICIENT, FLEXIBLE DIFFUSION LAN- GUAGE MODELS VIA CLATTION-INSERION PROCESSES
Fangyu Ding 1.2,
Ding Ding!.2.
Sijin Chen
Kaibo Wang Peng Xu²
Zijin Feng"
Haoli Bai²
Kai Han?
Youliang Yan?
Binhang Yuan Jiacheng Sun
*HKUST Huawei Foundation Model Dept 'CUHK
ABSTRACT
While Masked Diffusion Language Models (MDLMs) relying token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Innertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead MDLMs: the computations non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-lengte settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which efficiently solve via parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.
INTRODUCTION
Diffusion language models (DLMs) (Austin et al., 2021 Campbell ct al. 2022 Lou et al. 2024) have rapidly emerged powerful paradigm for language modeling, offering compelling alternative the dominant autoregressive (AR) approach. They offer distinct advantages, including bidirectional context modeling and the potential for parallel decoding. Within this domain, a body of work Masked Diffusion Language
Models (MDLMs) (Nie et al.) 2024 2025 Ou et al. 2025 Sahoo et al. 2024 Shi et al. 2024) is the most widely studied. These models operate through a forward process that progressively corrupts each token into an absorbing state <MASK> and a backward process that reconstructs the original sequence by iteratively unmasking tokens from a fully masked sequence, with a fixed sequence length during the diffusion process.
Despite their success, MDLMs are fundamentally limited by their fixed sequence length. The first issue lies in their restricted generation flexibility, which leads to challenges in modeling variable lengths and performing self-correction: once a token is unmasked, its content and position become fixed, thereby risking error accumulation a similar sense to autoregressive models. The second issue their substantial computational
"Equal contribution.
I "Correspondence to (biyuan hkk sunjiachengl @huawei.com).

---

Published as a conference paper ICLR 2026
Sun
Ext
HiLh
0000
Scm
VET1
FSr
Nalk
Borrarg
Hilh
Fre
wall
This
seertig
/NGR
(a) MDLMs, sequences padded to length 10.
(b) DID.
Figure 1: Conceptual diagram of MDLMs compared to Deletion-Insertion Diffusion language models (DID).
inefficiency, as the model must repeatedly process full-length sequences. Under the typical log-linear noise schedule, about half of the FLOPs are allocated to the non-informative <MASK> tokens during both training and inference. Further, if MDLMs are applied to variable-length sequences, their fixed-length nature demands
padding to the same length (Nie et al.] [024]] 2025 Wu et al. 20256 Gong et (024) (Fig. [a allocating
extra FLOPs to the non-informative <PAD> tokens. This means generating a shorter sequence is not faster.
To address these issues, we propose Deletion-Insertion Diffusion language models (DID), a novel discrete diffusion paradigm that fundamentally differs from MSLMs. DID replaces the in MDLMs with delction-insenne the forward process until the sequence is empty; in the backward process, generation starts from an empty sequence and iteratively inserts tokens until a complete sequence is reconstructed. DID eliminates the <MASK> and <PAD> tokens used in MDLMs, saving FLOPs and improving computational efficiency. Regarding generation flexibility,, DID natively supports variable-length data, and an insertion- based language model, features an intrinsic self-correction mechanism that dynamically adjusts token positions during generation.
processes processes (Fig. [b). Specifically, tokens are progressively deleted in
We implement DID by addressing several non-trivial design and training challenges. First, rigorously formulate the deletion and insertion processes within the discrete diffusion framework, and develop a score- based approach built upon the Denoising Score Entropy (DSE) (Lou al., 2024) objective to train DID. Concretely, we define an insertion score that models the probability of inserting any token at any position of a sequence a given time interval, and derive a corresponding Denoising Insertion Score Entropy (DISE) training objective. The DISE objective is based on a ratio of subsequence counts in the clean data after and before an insertion, which serves as the training target for the insertion score. To efficiently compute the ratio, we develop parallelized dynamic programming algorithm that exploits data parallelism. Moreover, we demonstrate that under the fixed-length setting of MDLMs, the DISE objective can be further simplified to a form resembling cross-entropy, further improving parameterization and learning of the insertion score.
Comprehensive experiments demonstrate the effectiveness of DID in enhancing efficiency and flexibility, In fixed-length language modeling benchmarks, DID achieves superior performance compared to strong MDLM baselines (e.g., RADD (Ou et al. (025)) when aligned by computational budget (FLOPs) (Tab. D. Compared to MDLM baselines, DID could accelerate training by up to 1.99x and 3.42x (Tab.B] [ and inference by up to 1.58x and 3.79 x (Tab. 2 D. for models trained on fixed-length and variable-length datasets, respectively. Moreover, DID shows strong performance in variable-length settings, outperforming MDLMs and existing insertion-based LMs in sampling quality and consistency with data length distribution (Tab. Fig
In summary, our main contributions are as follows:
We propose DID, a novel diffusion LM based on deletion-insertion processes that eliminates the use of <MASK> and <PAD> tokens, improving the computational efficiency and generation flexibility of DLMs.
We develop DISE, a score- score-based training objective, and an efficient parallel dynamic programming imple- mentation that enables effective learning of DID's insertion process for language generation.
Our experiments demonstrate the superior efficiency and flexibility of DID over baselines of MDLMs and other cntertion-basedd LMs on language/length modeling, generation quality, and training/inference speed.
2

---

Published as a conference paper ICLR 2026
2 PRELIMINARIES AND RELATED WORKS
2.1 CONTINUOUS- TIME DISCRETE DIFFUSION
continuous-time discrete diffusion model consists of a forward noising process and a backward denoising process, both continuous- time Markov chains. In the forward process, the training samples are progressively corrupted into pure noise. The model aims to learn its backward process that inverts this corruption, and generate samples by sampling through the backward process from noise.
Continuous time Markov chain. We consider discrete state space X. continuous-time Markov chain (CTMC) is process 2; on X' with E [0, 1/, starting from an initial data distribution po (z%). A CTMC is characterized by time-dependent transition rate matrix Q E x| For distinct states æt. y X, Qe y) 2 0 defines the instantaneous transition rate from 2, to y. This means, at an infinitesimal time interval (,, f At), the transition probability is given by:
Po+Ar (y) (xt)) 5 y) + (r y)
where 5 is the Kronecker delta. In other words, the evolution of the marginal distribution Pt E A|z1 follows the Kolmogorov forward equation de
dp, PtQ Note that the diagonal entries of Q, should satisfy
Qe( x.) = - Ea (z y) to ensure the weights of pt add up to 1.
Determining the forward process. In the forward process, a common parameterization (Campbell et al. 2022) of the transition rate matrix is Q (t)),, where a (t) is scalar noise schedule and Q constant rate matrix determined by the model design. Taking this to the Kolmogorov forward equation, one can analytically solve the marginal distributions by pt ps Pt|s, where Pils exp((o(t) (s)) is the
transition probability matrix from time to time t. and ot) = So a(T)dT.
Learning the backward process. is known that the time reversal of this process is also a CTMC, with its infinitesimal transition probability similar to [q[
Pt-Atji xx)) 8(x,, y) + Qr At.
The reverse transition rate matrix Q is associated to its forward counterpart Qe by the identity Q (a,,, y) Qr(y, æe (()) By for 3; * and Q.(x,, æx)) = - Qr( y) (Lou et al. 2024 Here, s(x;, f)y (x))))(((x)) is the concrete score, which is generally intractable and commonly approxi- mated by parameterized network s0 t)y trained with the Denoising Score Entropy (DSE) objective (Lou
et al. 2024). negative evidence lower bound (ELBO) for discrete diffusion models (details in Appendix D.I
Pro (z) log so (xi, (ar,,)) t)2 + K
Pejo Pejo(y) (r)) (=)o
]
CDSB (x))=
E
Σ Qe(y,x))) y/
so s0(2i,t),
(3)
1,00
P=)((z)200)
where the expectation is taken over I. Unif((o, 1]) and m, ~ p.o((a) and K K(a) = a(log a 1).
2.2 INSERTION- BASED LANGUAGE MODELS
Classical -aseetion-based models, such the Insertion Transformer (Stern et al. 2019), Levenshtein Trans- former (Gu et al. 2019). and InsNet (Lu et al. 2022), pioneered sequence generation via iterative insertion or edit operations. These models demonstrate the potential of flexible decoding orders and parallel decoding. However, their training objectives are defined at the level of local edit policies, rather than arising from a probabilistic modeling of the global data distribution.
Recently, insertion operations have been integrated into discrete diffusion and flow matching. Flexible Masked Diffusion Models (FlexMDMs) (Kim et al. 2025) augment MDMs with learned insertion expectation
3

---

They also suffer from several practical Published as a conference paper ICLR 2026
insert additional <MASK> tokens during generation; this enables variable-length generation but still relies on the unmaasinng paradigm. Edit Flows (Havasi et al. 2025) instead define CTMC directly over variable-length sequences via insertion, deletion, and substitution edits. To make sequence- flow matching tractable, Edit Flows augment the state space with auxiliary edit-path variables and estimate the training objective by Monte Carlo sampling of these paths. While this preserves theoretical correctness, adds implementation complexity and an additional source of stochasticity beyond the randomness of the CTMC forward process. Insertion Language Models (ILMs) (Patel et al.] 2025) draw inspiration from diffusion approaches and formulate CTMC forward process that deletes tokens learn backward insertion process. However, ILMs are not diffusion models and cannot learn the true backward insertion process since their training objective is more heuristic rather than limitations; for instance, they can only insert one token per step and require another network to determine when to stop the generation.
DID bridges the gap between flexible insertion-based generation and rigorous discrete diffusion. Unlike classical insertion models and ILMs, DID is well-grounded in continuous-time diffusion framework: deletion and insertion form the forward and backward CTMCs over the full variable-length sequence space, and the DISE objective inherited from the DSE objective to enable Eikeihhood-bounde training. Unlike Edit Flows, however, DID does not introduce auxiliary edit-path variables. Thanks to the independence structure of the deletion process, the forward transition probabilities admit closed-form expression terms of subsequence counts, and the subsequence count ratios that appear in DISE can be computed exactly via parallel dynamic programming. Therefore, DID avoids the additional variance and engineering overhead associated with sampling edit paths, while maintaining principled, saeeihoodd-bunnder training objective and completely eliminating the use of computationally wasteful <MASK> and <PAD> tokens.
3 DID: DELETION-INSERTIO DIFFUSION LANGUAGE MODELS
We propose DID to improve the efficiency and flexibility of diffusion language models. Instead of masking and unmasking in MDLMs, DID reconstructs the diffusion processes with deletion and insertion. In this section, we rigorously formulate the forward deletion process in Sec. 3.1 backward insertion process and sampling algorithm in Sec. 3.2 develop a score-based approach to train DID in Sec. Sec.B.3] discuss efficient and analyze the additional
implementation supporting parallelism for DID training objectives in Sec. 3.4
optimization for the fixed-length data setting considered by MDLMs in Sec. 3.5
3.1 FORWARD PROCESS: DELETION
The forward process of DID is CTMC on the state space U Vd that gradually shortens the sequence length by deleting tokens, thus equipping the model with variable-length ability. Similar MDLMs, we define this forward process through independent token-level deletions with rate (t). Specifically. at the token level, token V can be deleted (denoted by transition to an empty state 2) with an infinitesimal rate (t),, or remain unchanged otherwise. Thus, the transition probability within infinitesimal time Al is:
(t), d' d'= Ø Peosej(co(lu) = (1-a
(
(4)
V' d'=00 V.
Based on this independent token-level process, the sequence-level transition probability between timesteps s and t with 0- < s <t < can be derived as:
(x) (1 e (a(()) (+))))) |ao1 - Imlle (1))))))))))) x.).
(5)
4

---

Published as a conference paper ICLR 2026
Here, [æ] denotes the length of the sequence æ, (t) = So ( is the integrated noise rate, and N (æ xx)) is the number occurrences æt as distinct subsequences in æx. of all the possible independent deletion paths from a; to æy; see Appendix D.2 for the proof of Eq. 5
[ This number accounts for the multiplicity
The infinitesimal transition of the forward process captured by the sequence-level transition rate matrix Qt- Due to token-wise independence, at most one deletion can occur within an infinitesimal time interval with non-negligible (S(At)) probability. Thus, the rate Qi (y, (t) is non-zero only when y = æ, or y > 2/. i.c. x is the result of deleting exactly one token from y, and the rate for y >1 Dy is (details in Appendix D.3
Pc+ Pe+Ane(EEll)) ( y).
Qr xz))
lim A1-0
(6)
Al
Note that, in our implementation, we prepend undeletable special token <BOS> at the beginning of each sequence, so the above derivations should exclude <BOS>. Therefore, the fully noised sequence single <BO5>, which also serves as the initial input token at the first generation step to represent an empty sequence.
3.2
BACKWARD PROCESS: INSERTION
As in Sec. 2.1 aim to learn the time reversal of the forward process, CTMC with rate matrix (1,, y) Qy(y. xz (sDe, f)y- where s the concrete score. Since Q: involves single-token deletions, the backward process Qi only considers single-token insertions (i.e., >1 x..)
Directly applying the concrete score learning approach used in MDLMs is impractical here. The reason that given Zy, the number of possible resulting states is variable, because different insertions can lead to the same result Consequently, targeting the concrete score requires calculating the number of possible y values and enabling the model to produce variable-shaped output representing the concrete score for each y. These requirements collectively introduce significant complexity and implementation challenges.
As an alternative, we target the insertion score 8. We learn score for every insertion, regardless of whether the resulting y are identical, We define an insertion action v) as inserting token v after the i-th position of we, resulting in Ins( i, v) (xgi U, xxx). The definition of insertion score is:
s(x, L) v] def Eego (1 (0) |z0| (Ins(an:a i, v), m0)
V(i, v) E
[0, 10, X2
(7)
Ex, [1 (()))
| N (z,, æu)]
whose shape is | [V), tractable for transformer-basee models.
Since the CTMC dynamics are based on the concrete score, to sample with the insertion score, we show that the concrete score is scaled average of insertion scores, and the reverse transition rate is summation of the rate of insertion actions (details in Appendix D.4p
(t) (t)
s(F 1 (01) ((1)
t(,,, v( y)],
(8)
N( y)
(E/(...))
(a(t)- a(t)
(.(2).9)= =
e o(t) s(x,, 1) v (o:: y)]
(9)
iET(m..y)
Rate action (1,((2.))))
For example, N(<BOS> bag, <BOS> babgbag) 5. The distinct subsequences (highlighted in bold); <BOS> b h b <BOS> b babgbag <BOS> babgbage <BOS> b g b g. <BOS> b babgba bagag FFor example, inserting 'a' after the 1st 2nd index of *<BOS> b a g' both yield '<BOS> b baag'. 'ee use a prepended, non-deletable <BOS> token (index 0) for insertions the start.
5

---

Published as a conference paper ICLR 2026
where (3 y) the set of viable insertion positions and v(x,,)) y) is the inserted token from Φ; to y
Based on Eq. 2 we can transform the concrete score-based sampling in Eq. 2 into an equivalent insertion score-based sampling without computing (x y). [ y), or v( y) (details in Appendix D.5):
a(rie I-e- 50 ( 1) ما 11-Ewy Duya Pi- Delc ( ((i, w) (e)),
V. + 23,
p ((i, v)(an))
(10)
t = Ø,
where 2 indicates no insertion, and Tau-leaping (Gillespie 2001), very popular approximate simulation method, could be adopted to sample all insertions simultancously for parallel decoding.
3.3 TRAINING OBJECTIVE: DENOISING INSERTION SCORE ENTROPY
We aim train the insertion score So using the DSE objective (Eq. 3 with the derived transition rate Q Zt
(Eq. (6). conditional distribution Pria (z) (t.)) (Eq. 5 and parameterized concrete score 80 f)y (Eq. g)
for DID. Here, directly substituting se (Eq. (8) into DSE is challenging. Since sq involves a summation of insertion scores 8g, it results in an intractable log-sum structure within the DSE objective (Appendix (D6)). we
apply Jensen's s inequality to derive tractable variational upper bound, the Denoising Insertion Score Entropy
(DISE).
Proposition 1 (Denoising Insertion Score Entropy (DISE)). The DSE objective for the
process is upper bounded by the DISE objective, LBSE (x)) I CDISF (xz),
(xx)) which is defined
C (x))= = E 1,00
(()))==) a(t)
Σ
5o 5o(2,, t)li,
N (In((ar, i, v), ao)
1og5o( t) uj to].
Stlliuot]]]] (I))
e (0) ((1)
N (1 vo)
where C² [(
(Ins (1n) N(2; .80)
a f-free constant, f (ni 1)), and x ~ Pr Pro (z))
The proof (Appendix D.6 utilizes Jensen's 's inequality and a summation identity (Lemma T to transform the objective from state-based (y) to action-based ((i, v)).
3.4 EFFICIENT PARALLEL DYNAMIC PROGRAMMING FOR SUBSEQUENCE COUNTING PROBLEMS
A fundamental challenge for the DISE objective (Eq. 11 is to efficiently solve the ratios of N ((In((x: (zt, i, v), an)) and N (xx,, xD) for all possible insertions of (i, v) on x4- Suppose the lengths of zo and H, are m and n, solving single subsequence counting problem of N (æx, 2o) has a well-known time complexity of ((m)) through dynamic programming, performing naively for n x times would be prohibitive. Here, we show that N (Ins(2;; (x; i, u), œo) for all (i, v) pairs could be efficiently solved based on the intermediate results of solving N N(x;, zo) just twice (via a prefix DP in Eq. 13 and a suffix DP in Eq. 14 reducing the time complexity compute all ratios from O(mn2 V) O(mr), and making the training of DID practical.
Counting N(x,, Zo). Here we briefly introduce the classic prefix DP and suffix DP solutions to this problem. Using Python's 's slicing syntax, the base cases for empty sequences initialized as 1 (other entries 0):
Na 0). aol: :i) = N N< moli :)) = 1, Vj E m..
(12)
The prefix DP iteratively computes N(x,, [: 1), ao[: .1) from the solved subproblems N(a mo j 1)) and N(x; [: 1], [ j 1)) to get the final result (xx n), mo ml). and the state transition is:
N(x; |: 1), æol: 51) = N(x [: d) Dol: j - 1)) + 6(x æoll 1]) (2x - zol: j 1]). (13)
*For example, if æ. <BOS> ba and y <BOS> ba g, then Ins(æ,, 1) = Ins(x:, 2, a) y. v(x,, y) I( 3) (1,2), and ((., 3) [1 y) 2.
6

---

Published as conference paper ICLR 2026
The suffix DP iteratively computes N (x :], æooj :I) from the solved subproblems (x :] æoli :]) and (( i + 1 :I. zo j + :)) to get the final result N (=; :]. zo =I), and the state transition is:
N [ [i :]. mا =I) NN (z) :], Foli :)) ( [i]. xo (i) -N (ær =ll :]).
(14)
The time complexities are (mn) for both the prefix and suffix DPs. Notably, they could be batched and parallelized along the i-dimension, thus supporting vectorization and parallel execution, and it only needs to sequentially loop over the j-dimension for m times.
Counting i, v), Z.). It could be efficiently solved based on the results of prefix and suffix DP:
( an)
((2o())) v)
(ax i), 2o -1)) N (ax (i mn =))
(15)
prefix DP result
suffix, DP resalt
index addition
due to the form of Eq. 15 results for all (i, v) pairs can be solved in parallel with elementwise multiplication of the prefix and suffix DP result matrices, followed by an index addition that could efficiently implemented with sparse tensor coalescence, Algorithm details and a Py PyTorch implementation are in Appendix E
3.5 SIMPLIFIED MODEL FOR FIXED-LENGTH SETTING
To facilitate a fair comparison with MDLMs on the videly-adopted fixed-length language modeling bench- marks (Tab. [T). and clearly isolate the superior FLOPs efficiency of DID, we develop a set of optimizations enhance DID in the fixed-length setting. We show that when aal is a constant, 1) the insertion score becomes
time-independent as the time-dependent terms of (1 (-) in Eq. 7 could be canceled out, which leads
2) a sequence- normalization property (details in Appendix D.7
Σ 8( 11, oj
(16)
This benefits the parameterization and training of DID from two aspects. First, the time-independence means that the network does not require time input in the fixed-length setting, thus the parameterization reduces to šp (xx,), saves the parameters for time embedding, and enables cache mechanism similar to (Ou et al. (025) the sequence not changed between steps. Second, the output of the insertion score network could be explicitly normalized with summation of | |ac| as in Eq. 16 Therefore, the summation term of outputs from the insertion score network in the DISE objective (Eq. II turns into a constant, giving rise to a simplified Denoising Insertion Cross Entropy (DICE) objective (details in Appendix D.8):
(1) ((()) (()
i,),
log 5o (xx))) + c c.]
C DIIE (mo) = 1 e (t)
(17)
t,
N(x,,
1,0
where C = log ml))
is O-free constant. DICE can be interpreted as a weighted cross-entropy loss between the predicted insertion scores and the ground truth of subsequence count ratios, hence the name.
N(w: N(w.,
4 EXPERIMENTS
We evaluate language modeling performance, sampling quality, and training/inference efficiency of DID the fixed-length setting in Sec.[4.1]and the variable-length setting in Sec. 4.2 For more details, results, discussions, generation examples, and intermediate generation process, please reter Appendix F
7

---

Published as conference paper ICLR 2026
Table 1: Zero-shot language modeling perplexity. Results for diffusion models are perplexity upper bounds.
Size
Method
WikiText
Lambada
Pubmed
AG News
LMIB
Arxiv
PTB
Small
RADD
38.27
51.82
56.99
73.18
72.99
85.95
108.79
DID-S
38.72
49.10
55.02
76.02
74.04
82.41
115.37
DID-F
36.91
48.00
52.89
71.48
72.04
78.38
111.60
Medium
RADD
28.44
44.10
41.06
48.96
60.32
66.28
81.05
DID-S
29.19
41.94
40.84
52.53
59.88
63.95
91.87
DID-F
28.35
41.00
38.71
48.84
58.05
61.77
87 87.09
Table 2: Generative perplexity (PPL, evaluated by GPT2 Large), unigram entropy, inference time (in seconds), speedup, and average generation length for fixed-length models under different total denoising steps.
Method
Steps
16
32
64
128
256
512
1024
RADD
PPL
284.78
155.01
1 111.56
95.10
87.56
84.00
84.05
Entropy
8.35
8.26
8.20
8.15
8.11
8.10
8.09
Time (s)
0.220
0.317
0.499
0.879
1.644
2.882
4.512
DID
PPL
158.93
110.06
97.32
91.25
86.98
86.04
85.35
Entropy
8.15
8.13
8.13
8.12
8.09
8.08
8.09
Time (s)
0.169
0.246
0.353
0.573
1.047
1.826
3.006
Speedup 1.30
1.29
1.41
1.53x
1.57 X
1.58x
1.50x
Length 1023.29
1024.01
1024.18
1024.07
1023.91
1024.03
1024.10
4.1 DID FOR FIXED-LENGTH LANGUAGE MODELING
Settings. Following RADD (Ou al. 2025), which serves as our baseline method of MDLM, we train DID of both small and medium sizes the Open WebText (OWT) dataset (Gokaslan & Cohen 2019) with the DICE objective (Eq, [7). We adopt the GPT2 tokenizer (Radford et 2019), concatenate all sequences, and split them into fixed-length chunks of 1024 tokens, and the training batch size is 512. RADD-small and -eeiium are reproduced with their open-sourced model checkpoints (trained for 400K steps on OWT).
Zero-shot language modeling perplexity. We evaluate the zero-shot modeling perplexity seven datasets in Tab. [ Since DID eliminates the computational FLOPs for <MASK> and hence reduces FLOPs by approximately half compared RADD for the same training steps, compare DID under training configurations: DID-S (Steps-aligned), trained for the same steps as RADD (400K) and DID-F (FLOPs- aligned), trained for double the steps (800K) match the total computational budget. We observe that when aligned by training steps (DID-S), our model achieves performance comparable to RADD, despite utilizing only about half the computational FLOPs. When aligned by the total computational budget (DID-F), DID consistently outperforms RADD across the majority of datasets for both small and medium sizes. This demonstrates that the computational savings achieved by eliminating <MASK> tokens are effectively turned into improved modeling performance. We also provide an ablation study in Appendix G.3 for DID trained with DISE objective (Eq. [T). i.e., without the additional optimizations in DICE for fixed-length data introduced This results in a reduced performance than DICE-trained DID in Tab. [ yet comparable to RADD.
Sec. 3.5
Generative performance. We report generative perplexity (PPL), unigram entropy (diversity), and inference speed of direct sampling across different numbers of total denoising steps in Tab. Compared with RADD, DID achieves significantly better generation quality (lower PPL) with fewer denoising steps. When more total denoising steps used, RADD slightly outperforms DID, which reasonable since RADD is naturally
8

---

Published as a conference paper ICLR 2026
designed for the fixed-length setting. Nonetheless, DID could consistently outperform RADD in the variable- length setting, which is deferred to Tab, Furthermore, DID consistently provides ~ 1.5x inference speedup. This improvement stems from the fact that the average sequence length during the iterative insertion process is shorter than the fixed-length process of RADD. We also provide nucleus sampling results in Appendix G.6 where DID achieves lower PPL and entropy compared with RADD, demonstrating a stronger annealing effect.
Training speed. We report the training speeds for different model sizes in Tab. to verify the efficiency gains of DID. DID demonstrates substantial speedups, increasing from 1.89 to 1.99 x as we scale up the model. Empirically, removing the computations for <MASK> tokens significantly boosts training efficiency,
While the reduction in FLOPs suggests a theoretical 2x speedup, the actual gains in Tab. are more modest. This discrepancy, as discussed in (Zheng et al. 2025), arises because inference is not purely FLOPs-bound task. For training. the observed gains in Tab. Bare also slightly lower due to additional overheads such the DP algorithm for loss implementation (Sec. (344) which is a constant overhead independent of the model size, thus the speedup could be scaled up with model
Table 3: Average training time (in seconds)
per 50 steps (i.e. batches) Open WebText.
Small
Medium
Large
RADD
26,46
53.17
92.90
DID
14.03
27.77
46.60
Speedup
1.89
1.91
1.99x
size. Besides, a less developed system-level support for variable-length data also constrains the speedups.
4.2 DID FOR VARIABLE-LENCIE LANGUAGE MODELING
Settings. Following ILM Patel et al. 2025). an insertion-based approach, we train DID small on the Stories dataset (Eldan & Li 2023; Mostafazadeh ct al. 2016) for 60K steps with batch size 512, uti- lizing its variable-length sequences (average length 213.43 under the Bert-base-uncased tokenizer (De- Vlin et al.] 20188) truncated to a maximum length of 1024 and without padding. We also train RADD- small on Stories for the same steps (> 2x FLOPs of DID). details in Appendix[F] Since RADD requires fixed-length inputs, we pad all sequences to length of 1024, which is in line with the setting for MDL.M in ILM experiments (Patel et al., (2025). As result, RADD generates fixed-length outputs containing <PAD> tokens, which we subsequently remove to obtain the final variable-length outputs. ILM is re- produced with its open-sourced checkpoints.
Table 4: Generative PPL, unigram entropy, inference time (in seconds), and average generation length for variable-length models under different denoising steps.
*; as outliers significantly affect PPL, only samples with PPL < 300 are counted, i speedup over RADD.
Method Steps
64
128
256
512
ILM
PPL
161.80
137.64
42.29
31.14
Entropy
5.20
5.65
5.97
6.01
Time (s)
0.016
0.034
0.087
0.271
Length
63.34
120.77
206.44
234.44
RADD
PPL*
81.92
50.89
34.47
26.78
Entropy
5.22
5.58
5.79
5.85
Time (s)
0.246
0.441
0.827
1.461
Length 110.66
200.73
349.54
353.47
DID
PPL 22.78
21.07
21.90
23.88
Entropy
5.90
5.94
5.94
5,94
Generative performance. We report the genera- tion quality and speed of direct sampling for variable-
Time (s)
0.090
0.132
0.218
0.388
length models Tab. 4 Compared with both base-
Speedup 2.73x
3.34x
3.79x
3.76x
lines, DID maintains significantly lower generative
Length 182.31
193.77
202.97
204.96
PPL (evaluated by GPT2 Large) with relatively high diversity and is more stable across different numbers of steps. This is important because strong sensitivity and non non-convergence to the manually predefined number of generation steps are undesirable.
Regarding inference speed, DID achieves speedup of up to 3.79 compared RADD, due to the savings of <MASK> and <PAD> tokens for both model calling and distribution sampling. On the other hand, ILM
LLrgge models not fully trained; only their training speeds measured.
9

---

Published as a conference paper ICLR 2026
Steps 64
Steps 128
Steps 256
Steps 512
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0
0.4
RADD
RADD
RADD
RADD
DID (ours)
DID (ours)
DID (ours)
DID (ours)
0.0
Longth
Length
Longth
Length
Figure Cumulative distribution functions (CDFs) of generation length under different total denoising steps.
achieves the fastest speed despite its lowest quality. We credit this its over-simplified generation mechanism, which samples token at exactly one designated position per step. In contrast, DID and RADD sample from all possible token positions to enable parallel decoding. ILM benefits considerably from this simplification, since categorical sampling is a major bottleneck in small models (Zheng et al. (202)). Moreover, ILM is limited to generating text shorter than the total steps (see Tab. 4 which also contributes to its faster sampling.
We also provide nucleus sampling results for variable-length models in Appendix G.6 and ablation studies of different padding lengths for RADD in Appendix GG. addressing potential concerns that the 1024 padding length for RADD might be too long unfair. When trained at length of 512, RADD exhibits observable degradation and remains ~ 1.59x x slower than DID, further confirming the original setting of ILM.
Length modeling. Besides, DID demonstrates superior length modeling capabilities, exhibiting consistency between the gen- eration length distribution and the training data length distri- bution. This is demonstrated in Fig. [ where the CDF of the length distribution of DID is closely aligned with the dataset compared to the baselines. DID's average generation length reported in Tab, iss also stable and approximating the ground truth distribution.
Table 5: Average training time (in seconds) per 50 steps on Stories.
Small
Medium
Large
RADD DID
19.93
37.87
67.75
7.71
12.30
19.83
Speedup
2.58X
3.08
3.42x
Training speed. We also compare the training speed on the Stories dataset (Tab. 5). The efficiency gains of DID are even more pronounced in the variable-length setting, reaching up to 3.42 speedup for large models. This because RADD suffers significant overhead from processing <PAD> tokens (since the average length 213.43 is much shorter than the padded length 1024), while DID benefits from the shorter length.
Regarding more efficiency analyses, ablation studies, inference strategies, and performance analyses, please see Appendix G
5 CONCLUSION
In this paper, we introduce DID to improve the computational efficiency and generation flexibility of diffusion language models by eliminating the use of <MASK> and <PAD> tokens in our paradigm. Theoretically, we formulate the diffusion processes for deletion and insertion, define an insertion score, derive and implement the corresponding training objectives and sampling algorithm for DID. We evaluated DID on modeling performance, generation quality, and training/inference speed, demonstrating the superiority of DID over the baselines of MDLM and existing insertion-based LMs in both fixed-length and variable-length settings. discussion of the limitations and future works of this paper is provided in Appendix A
Medium and large models fully trained; only their training speeds measured.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXhv:2503.09573, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981-17993 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 15:282666-22799 2022.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson.
One billion word benchmark for measuring progress in statistical language modeling, 2014. URL[https
7/arxiv. otg./i.org//ooo abs/
http:oogg//aooooooooooo..oaaaeat 1122.331.11 3005
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05455. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 2r 2021.
Arman Cohan, Franck Dernnncour, Doo Soon Kim, Trung Bui, Scokhwan Kim, Walter Chang, and Nazli Goharian. A discourse- aware attention model for abstractive summarization of long documents. Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018. doi: 10. 10.18653/1/111-2209 8- -
URL http:/ //tp doi
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 10-awareness. In Advances in Neural Information Processing Systems, 2022.
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/ Abb//880, 2018. URL http::://aaii...o///
abs/1810 04805
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023, URL pettps://arxiv..
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL

---

Published as a conference paper ICLR 2026
Daniel T Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. The Journal of chemical physics, 11(4))1716-1733, 2001.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus.
http:://Skylion007..gg
io/
@pperiebTextCorpuss 2019.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng. Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410.1789) 2024.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In Advances in Neural Information
Processing Systems, volume 32, pp. 11179-11189, 2019.
Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint 0rXiv:2506.09078 2025.
Jaeyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du Sham Kakade, Timothy Ngotiaoco, Sitan Chen, and Michael Albergo. Any-order flexible length masked diffusion, 2025. URL https
abs/ 2509. 01025
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading compre- hension dataset from examinations. arXiv preprint arXiv:100.004883 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. httt:p:s//peeeeriieenneee/e/eees
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Difffusion Modeling by Estimating the Ratios of the Data Distribution. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pp. 32819-32848. PMLR, 2024.
Sidi Lu, Tao Meng, and Nanyun Peng. Insnet: An efficient, flexible, and performant insertion based text generation model. In Advances in Neural Information Processing Systems, volume 35, 2022.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):13-330, 1993, URL http:::////et /ww
aclweb.org htt.:/e....r//aahhoooo////////// http://ahhhhhooyyooo
2004
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 0tff:::80000077770 2018.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839-849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 18.18653/v1/NI 116553/11/N16.. 1098. URL https: a/acl//cccc/ anthe 1 ogy. org N1 6- 1098/
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXi::2502.09992, 2025.
12

---

Published as a conference paper ICLR 2026
Manfred Opper and Guido Sanguinetti. Variational inference for markov jump processes. In Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. URL https://proceedings wtwpeeeeeiiinns.....e.eeeeeeeeee neurips. cc
00w.a8800060050000aPee00 pdf
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference on Learning Representations, 2025. URL BtLhhtts::. /otp:ee///peeeeweeww net/
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http: http:::/www.a..eecec www.://wwwwwww...coo.o//..... org/ anthology/ P P16-1144
Dhruvesh Patel, Aishwarya Sahoo, Avinash Amballa, Tahira Naseem, Tim G. J. Rudner, and Andrew McCallum. Insertion language models: Sequence generation with arbitrary-positicn insertions, 2025. URL
bttps://arxiv.co org/ abs/ 250..
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in 2024.
Neural Information Processing Systems,
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models, 2025. URL
https://arxiv:: org. abs 2506. 01928
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL htttp:://rxiv.. wttpg///////./..//.////
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP)) pp. 4463-4473. 2019.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances neural information processing systems, 37:103131-103167, 2024.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hes- SlimPajama:
tness, and Nolan Dey.
627B token cleaned and dedu-
plicated version of RedPajama. Learning, volume 97, pp. httts////gnnntteeeaaeeee///eaaea
https://ww.ccrrt
www.:eeeerbras..ee/bbboo. net/blog
s slimpa jama
2023. URL
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In Proceedings of the 36th International Conference on Machine
PMLR, 2019.
13

---

Published as conference paper ICLR 2026
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
09864
Guanghan Wang, Yair Schiff, Subham Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXivv2503.00307 2025.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding, 2025a. URL htt.:pps//aaaiii..rrraaaa//////s 22681
Zirui Wu, Lin Zheng. Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng. Zhenguo Li, Victoria W., Guorui Zhou, and Lingpeng Kong. Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025b. wtt./httt://hknnnt
gi thub. o/ b 1 og 2025/d
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 190X.v:905.07000 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances neural information processing systems, 28, 2015.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yt Yu Liu, Jun Zhu, and Qinsheng Zhang, Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling, 2024. URL
https://arxiie http:///aawwvvv..vaaaeee ab////bs/24//0 2409. 02908
Kaiwen Zheng. Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In The Thirteenth International Conference on Learning Representations, 2025. URL pecctps:/oppnrreet Det//oorum2GG=CICCCCI. rNr
14

---

Published as conference paper ICLR 2026
CONTENTS
Introduction
2
Preliminaries and Related Works
Continuous-Time Discrete Diftusion
3
2.2
Insettio-Baase Language Models
3
DID: Deletion-Insertion Diffusion Language Models
Forward Process: Deletion
4
3.2
Backward Process: Insertion
5
3.3
Training Objective: Denoising Insertion Score Entropy
6
3.4
Ellicient Parallel Dynamic Programming for Subsequence Counting Problems
6
3.5
Simplified Model for Fixed Length Setting
Experiments
4.1
DID for Fixed-Length Language Modeling
8
4.2
DID for Variable-Length Language Modeling
5
Conclusion
10
Limitations and Future Works
17
The Use of Large Language Models (LLMs)
17
Notation Summary
17
Detailed Proofs and Derivations
18
D.I Derivation of the Denoising Score Entropy Loss (Eq
18
D.2 Derivation of the Sequence- Level Transition Probability (Eq
19
D.3
Derivation of the Transition Rate (Eq(6)
20
D.4
Derivation of the Relationship between Concrete Score and Insertion Score (Eg.
8
20
D.5
Derivation of the Sampling Probability (0)
21
D.6
Derivation of the DISE Objective (Eq
22
D.7
Derivation of the Sequence-LLeve Normalization Property (Fqq.
16)
25
D.8
Derivation of the DICE Objective for Fixed-Length Data (Eq.) 17]
26
15

---

Published as conference paper ICLR 2026
Algorithm Details
27
Dynamic Programming for Subsequence Counting
27
E.2
DID Training Algorithm
29
E.3
DID Inference Algorithm
29
E.4
PyTorch Implementation of the Dynamic Programming Algorithms
29
F
Experimental Details
31
Fixed- Length Training Details
31
Varithle-Length Training Details
32
Sampling Details
33
Evaluation Metries
33
More Experimental Results
33
G.1
Efficiency Analysis of the D3P Algorithms in DID Training
33
G.2
Efficiency Analysis of DID Inference
34
G.3
Ablation Study of Sequence- Normalization for Fixed Fixed-Length Models
35
G.4
Language Modeling Performance for Variable-Length Models
36
G.5
More Interence Strategies: Cosine Timestep Schedule
36
G.6
More Interence Strategies: Nucleus Sampling
................................
37
G.7
Conditional Generation
38
G.8
Comparisons with RADD of Different Padding Lengths
................................
39
G.9
Evaluation with More Checkpoints
40
G.10 Scalability
40
More Discussions
41
Generation Examples
42
Samples Generated by the Fixed-Length Model Trained on Open Teb
42
IW Unconditional Generation
42
.1.2 Conditional Generation)
49
Samples Generated by the Variable-I.ength Model Trained on Stories
52
Demonstration of the Intermediate Generation Process
57
16

---

Published as conference paper ICLR 2026
A LIMITATIONS AND FUTURE WORKS
This work presents the core framework of DID and, unlike the more established MDLMs, has yet integrated many optimizations, such as advanced inference algorithms [Wu et al. 2025a Wang et al. 2025). hybrid models combining autoregressive approaches (Arriola al., 2025 Sahoo et al. (025). Since these optimizations are not inherently tied to a specific diffusion process, adapting them to DID represents a promising future direction. Second, although we have demonstrated the effectiveness, efficiency, and flexibility of DID, our models were trained at relatively small scale due to resource constraints. As result, their performance on larger and more complex tasks remains unexplored, and we leave scaling up DID future work.
B THE USE OF LARGE LANGUAGE MODELS (LLMS)
Following ICLR guidelines, wish clarify of Large Language Models (LLMs) during the preparation of this work.
The research ideas, methodology, experimental design, and analysis presented in this paper were developed entirely by the human authors. LLMs were not involved in the ideation process.
We utilized LLMs as tools for editing and polishing the text, helping improve the clarity and phrasing in various sections of the main paper and the appendix. The authors have reviewed the manuscript thoroughly and take full responsibility for content.
NOTATION SUMMARY
Table 6: Summary of Key Notations
Notation
Description
V
Vocabulary.
X'
U Pd
Sequence state space (variable length).
Ep, Ft, y
Clean sequence; sequence at time another sequence state.
[0]
Length of æ.
<BOS>, <MASK>, <PAD>
aegin-of-sequence (non-deletable), absorbing mask, padding.
Ø
Null token representing deletion or no insertion (not in V).
Qc, Qi
Forward and reverse sequence-level CTMC rate matrices.
(-I)((+))
Forward transition probability from s to Marginal distribution at time t.
Pr(-)
a(), a()
Noise rate and its integral Jn ((T) dr. Concrete score (y))pP
s(x,,))
N(a,y)
The number of occurrences of x as a distinct subsequence of y. is obtained by inserting exactly one token into æ.
v(x, v)
The unique inserted token when y > æ.
Ins(æ, v)
The result of inserting token E V after position i of æ.
I(x, 8)
Valid insertion indices s.t. Ins x. i, (xx, y)) = y. Insertion score for insertion operation (i, time t.
5(xr,
[i,,)
Time-independent insertion score (fixed-length setting). Convex function a(log a
K K(a)
17

---

Published as conference paper ICLR 2026
D DETAILED PROOFS AND DERIVATIONS
D.I DERIVATION OF THE DENOISING SCORE ENTROPY Loss (Eo
The training objective for discrete diffusion models is derived by minimizing a variational upper bound on the negative log-likelihood (NLL) of the data, log (z.)
Let Pe, denote the path measure (the probability distribution over entire trajectories) of the true posterior reverse process conditioned on the data 2p. Let Tpo denote the path measure of the learned reverse process parameterized by By the data processing inequality, we have:
log (xr) (r)) = DKL (6zo lI:) (FxoFxxl111
(18)
We define the training objective as this variational upper bound: [(0) DKL (Pa, IP") assuming both processes share the same prior distribution at =
Both processes are Continuous-Time Markov Chains (CTMCs), Let (æ y) denote the true conditional reverse transition rate (for Fx.) and Q (z,, y) denote the parameterized reverse transition rate (for Po).
The KL divergence between path measures can be decomposed using the chain rule. If we consider a discrete-time approximation with infinitesimal step At, the total KL divergence is the sum of the expected KL divergences at each step. We analyze the KL divergence between the infinitesimal transition probabilities (y) = 5(x,,y) y) + Q! (æ, a) + o(Dt) and y² ax²) = 5(2,, y) + Q (xt, y) + o(Dt). The instantaneous KL divergence (Opper Sanguinetti 2007):
(x), log (2 v) (1 y) (x 8) + o(At).
= At Σ yof
(xr,
(19)
Summing these contributions and taking the continuous limit (At + 0) yields the integral form for the path KL divergence:
((0) = [
@ (tt v) @) y) log Of y)
+ (2,, v) ( 8)
Σ prstiae
dt.
(20)
Pe
We now substitute the specific definitions of these transition rates. The conditional reverse rate Q! related to the forward rate Qy a.)) by
Pyju(1y wo) Pejo (xx (zo)
(2) v) = Qu x.)
(21)
The parameterized reverse rate Q9 is defined using the score network so (z t)y:
2 (ar,, y) (g (x))sp t)
(22)
We substitute these rates (Eg 21 and Eq into the expression inside the summation in 2g 20 The expression becomes:
(Qilw,az) 21) Pejo(ylaro) (n))
Qe xx)
(2 (eo)
Pejo sea)
Pe ar)
log
+ +1 2:)s0((1, t)y Qu(y,
(23)
Peo (x (x0)
Qi (x, (5) y)
(2) (xo)
18

---

Published as a conference paper ICLR 2026
We simplify by canceling Qe 2t) inside the logarithm and factoring it out from the entire expression:
Pejo (1o()a))) log (e (ma)
(2))
(1)0
Qe xx))
+ (0 t)
(24)
Ps (0 (mp)
(t:; 50 (0 1y
Pejo (20)
We expand the logarithm (Aog(A/B) = log A log B) and rearrange the terms:
(y 2o))
Prjo (n))
Pejo (2)0
xx;
log
log so s0(2 t) + se Be( t)g
(25)
(aju( (2o)
Pejo (20)
Pejo (ogo(axs(22)) (x0)
(y) (o) Pojo (2 (mr))
Prjo (oo) log so (are, 1) Pejo ar))
Pejo ( x2))
Qr 20x) 80 (0(
loog
(26)
Prjo (mt (m0)
Letting ((a) = ={log a - 1). Recognizing that the time integral Sol dl combined with the expectation over Ht ~ Pajo is equivalent to the expectation over uniform time ~~ (0, 1) and the corresponding conditional
It. the objective [(0) is exactly the DSE loss 3Eq
(D))
E
Σ (x))
s)(
(y 20)) Pijo (1)
log sa t)y K P (
(2 (zo) (
) (27)
1, 2
D.2 DERIVATION OF THE SEQUENCE- LEVEL TRANSITION PROBABILITY (EQ
We derive the sequence-level transition probability Puje (ar, based the definition of the DID forward process as an independent token-level deletion process. We begin by analyzing the dynamics of a single token.
The token-level process (Eq 4 is a Continuous- Time Markov Chain (CTMC) on the state space {u, Ø) where v E V denotes the presence of a token and Ø denotes the deleted state. The transition rate matrix Qtuk at time t, indexed by (v, Ø), is defined by the deletion rate o (1):
(()) -
Q - (-0(())
(t) 0
(28)
Let (T( be the probability that the token is in state at time t), given it started in state time (s.(s) = 1). The evolution of this probability follows the Kolmogorov forward equation:
d..(r) dr P. (v, v) + (a( (2, v) - (T))))
(29)
Solving this first-order ordinary differential equation by integrating from to t:
[. dTP P.(T) - ff- -o(()dr = ln()((t) - In(P = (ō (s)).
(30)
Thus, the probability that a single token survives during the interval [s, f) is Po(t) ( Conversely, the probability of deletion is 1 - ((s)))
We now consider the sequence-level transition from æ, x;. Let Ao (t) (()). A transition occurs if the tokens forming survive and the remaining |z.|| - lx, tokens are deleted. Since token deletions are independent, the probability of a specific path (a specific occurrence of x; in x.) is (e-Ar) ) (1 e
The total transition probability Pas (x,(xx) is the sum over all distinct paths, counted by the subsequence count N(x, æ.). Therefore, obtain:
Paja (,),(2 (z.) N(x,, 21,(1 e- (((()) ((((())-())) it |m l ((ot))-
(31)
19

---

Published as a conference paper ICLR 2026
D.3 DERIVATION OF THE TRANSITION RATE (Eo
We derive the transition rate (y m.) from sequence sequence 21, where æ is obtained from by deleting a single token (denoted as y 11 xy). This implies lyl = [x;] +
Pt+ Pt+ P++Aae((a)) (xell)
Qu xz)
lim At-0
(32)
At
(0144 - N(x, (æx y)
lim
(33)
At-0
At (1- (1- 1 (o(())(tt( (a v)
lim
(34)
At (1) dAl)) N(x,,))
(o(1)AI o(AI)
lim
(35)
At-0
At
(()A + dll) N( v)
lim
(36)
At-3
At
= (a),
(37)
D.4 DERIVATION OF THE RELATIONSIUP BETWEEN CONCRETE SCORE AND INSERTION SCORE (EQ.8)
We aim to prove the identity stated in Eq. 8 This identity concerns the backward process where transitions occur such that y >- I; Note that this condition implies |y/ | + 1.
First, we derive the explicit form of the concrete score s(x,, f)y = (x.))/((xx, by expanding the marginal distributions using the forward transition probability (Fq. 5)
Po(
Ex, Prjo (z.)
s(x,, t)y
((() Em [Prjo (we [xo)]
Emr 1(1- e0(() |00l | (1)) N N(y, [p))
Exo [1 - (1-e-8(()) e-(()) )(1())/a) ( [0 (xt, x00)
( (6--0)))
Exz (1 ( azul N (g. xo)
c-a(0)221(1 - e e-a(t))- me Ego (1 -- e- N (x 2v)
(t)) Exs [1 c (0(t))lam N zu)
v|=|=:|+1
(38)
1 1 -01)
=(1) Exp
(1 (zm) N(x,, 2n)
Next, we examine the right-hand side (RHS) of Eq. and substitute the definition of the insertion score s (Eq. 7
RHS-
Σ
s(x,, (i, o(x,, v)
(39)
--0( N( v)
((o(.)))))
a(1)
Exzo [ -e- ( N (Ins (2,, i, o(w,, y)), æ6o]
(40)
1 (0) N( y) iE (ET(M))))
Eu, [(1 e- e(a(t)))aaa) Jauul N(x;, mo))
By definition of the index set I (( y), for any E [( y), we have Ins(x i, o(x,, y)) y. Therefore: (10)) / N xo)]
e ((t)
Exo [1 e (1)))
RHS 1- a01) N(x 4) (
(41)
Exa --)( woal N (axe, mo))
20

---

Published as conference paper ICLR 2026
The term inside the summation is independent of the index i. We factor and utilize the property that Eie((r) 1 [T y)1 = N (x1. y):
Egg (1 e (()) mo N2y mo0]] Eze [ e ( mo (x,, 20)]
RHS
(ax y)
(42)
1 e (()) N( y)
(()) Esoo [11 - e = [ N an)
(43)
a(t) Ezz - e (1) æel N N( vo)
This matches the derived concrete score in Eq. 38 completing the proof.
D.5 DERIVATION OF THE SAMPLING PROBABILITY (EQ 10
We aim to derive the probability of executing a specific insertion operation inserting token e after position 1-within an infinitesimal time interval At, during the backward process.
The backward process is CTMC characterized by the parameterized reverse transition rate matrix Qf. The rate of transition from state æ, to a different state y is defined as:
Q (xx,, y) Qr at (e0 (mte D)v-
(44)
In the DID framework, backward transitions occur only when y > 2s. We substitute the forward transition rate ( Q, x,) ( y). We also substitute the parameterized concrete score sg. which
derived by parameterizing the relationship between the concrete score and the insertion score (Eq 8):
(()
s0 D2y = -- 6(0) N( v)
Σ jEI(...))
50(2, v(rr..))
(45)
Substituting these expressions into the definition of Q (2,, y):
(e) y) (c(() 9))
5 o(t)
1
Σ
80(2,, y)]
(46)
1-6- N(x y)
jEI(xE..))
(t)
( - e a(t)
Σ
s0(a,, t)
(47)
(
This result demonstrates that the total transition rate from Hy to is decomposed into summation of individual components. Each term in the summation, o I- " Bo ()βi, va y)]. corresponds the instantaneous rate of the specific insertion operation (j, v( y)) that transforms æ; into y.
By the definition of CTMC, the probability of a specific operation (i, v) occurring within the infinitesimal interval A/ is given by its corresponding rate multiplied by At. For + Ø. we identify this probability directly from the decomposition above:
p Pl- w))=))) = ((()) 6 (((())
1- -0 5o (a t)li, At o(At)
(48)
o (t)e o())
1-e-o 3e (x,, Li wJ + o(At).
(49)
This confirms the first case of Eq 10 The probability of no insertion occurring position i (i.e., v 2) is determined by the normalization constraint, ensuring the sum of probabilities for all possible events at that position equals 1.
21

---

Published as conference paper ICLR 2026
To confirm the equivalence between this action-based sampling and the required state-based transition, we verify that the action probabilities correctly recover the state transition probability Pi- (3) (z)=2)). A transition y occurs if any of the actions indexed by (aree y) occurs. In the infinitesimal limit Al 0, these actions are mutually exclusive events:
Σ
pi- Re-sqe(Le o(2,, 2)) + o(St)
(50)
(1)) E -
50(2,, t) u(r,, U)] At o(()).
(51)
(()(t)
jE
By Eq. 47 the term in the parenthesis is exactly Q% (a,, y). Thus, Pp_ At- (y|2,) = (a y) + ((At),
confirming that the action-basse sampling correctly implements the dynamics of the backward CTMC.
D.6 DERIVATION OF THE DISE OBJECTIVE (EQ
We derive the Denoising Insertion Score Entropy (DISE) objective (Eq 11 starting from the general DSE objective (Eq 3 demonstrating that DISE is a variational upper bound on DSE, CDiSE 2 CD (xo).
We begin with the DSE objective:
(y ato) Pejo (x 200)
) Pejo (xt (x0)
(ejo(ty (20)
LB (ar)
E. Σ Qr(y, mi) so(22, 1),
log (2 f) K
(52)
1, 2
where (aa)= = a(log 1). In the deletion insertonnnn process, the transition rate Qr xt) is non-zero only when y >1 Zt.
We now substitute the specific definitions for the DID process. The transition rate is Q, ( y). The conditional probability ratio is:
Pt o mo)
e a(1) N(y, ag) 1- --(( a()) 20)
(53)
(= mo)
The parameterized concrete score (from parameterized Eq is:
6 (t)
so (0 t)y
50( v(z,, v)l.
(54)
1 1- a(t) N( y) (EI(..)))
We examine the expression inside the bracket of the DSE objective by substituting these definitions. Let o( y) for brevity in the following block:
Py æo)
Pojo(y ao)) Dijo(ar, (oo)
50 b) (ajo xp))
log (x t)2 K
((1))
Σ iE((x..))
80(2c, t)
1 (t)(() N(x,,)) v)
(55)
a(t)
2x0) log
Σ
5a (at, 1)66 o0]
6 ((t) N(x,, mo)
e (((( N(x,, y)
N To) 1- a(()(( N N(x,, mo))
e a(() a(t)
+ K
22

---

Published as conference paper ICLR 2026
We expand the (+) term using (a) allog 1). By expanding the logarithms (log(AB) log A log B), we observe that the terms involving the time factor log( I-e euy cancel out exactly.
The expression inside the bracket simplifies significantly by factoring out the time prefactor:
a())
5o (t,
(())
N( y)
a
50 t) u)] ))
Nly zo)
log
1
Σ
(56)
(xx
N(xx, y)
(El(2..))
Nyy,a)) wo)
N (o)) N(x,, w7)
-
log
N(
Substituting this simplified bracket back into the main objective equation and multiplying by Q, xz) (t) y). The DSE objective can be decomposed into three terms (T1, T2, T3):
COSE (z)) = TI + T2 + T3.
(57)
We define these terms based on the components derived above:
(t)e-o(11)
E T1
Σ
N( 8)
Σ
so(2,, u)
(58)
01))
N Nox, y)
at)e--on x(t)
(t()
Nly vo) log
T2 E a())
Σ N(w,,) 3)
Σ
50(xr, t) u|
(59)
N (1
N( v)
1~11
ie (x)
atte -0()
(yy (wx, xo)
T3
E 1,
Σ
Na 4)
(60)
1 5
1/112t
We now apply Jensen's inequality to T2. The term inside the logarithm is average of insertion scores. Because the logarithm function is concave, log( + Eai) Z Σ log a, Since the logarithm negated, this leads to upper bound T2:
log Na y)
Σ
Co(
log 50 (a,, f(t,
(61)
N (a y)
ie (().))
We define T2Bound as the upper bound for T2:
E (1)))
N( y) N(y, vo) N( 2o)
T2 < T2
Σ
Σ
log 50 (x, (xx,, v)
t, 1 a(t)
N( v)
(E/(
(62)
We define the DISE objective the upper bound obtained by replacing T2 with T2 ensuring C CISE (xx)) 2
CDSE (xo):
DISE (xr)
T1 T2 Bound T3.
(63)
23

---

Published as conference paper ICLR 2026
To simplify the nested summations and transform the objective from state-level (y) operation-level ((i, v)), we rely on the following identity.
Lemma (Summation Change of Variables). Let w(re; y) denote the unique inserted token and (( y) the set of valid insertion positions that yield from æt. For any function G,
Σ
Σ
G(i, v(z,, v), v) - E G(i v, Ins( (, u))).
(64)
y~ (ET(..))))
Proof. Define the index sets
A ((y,)) 4 Bt, i ((x y)} B= ((i,)) ie {o...., [o.].., u V}.
(65)
The map g - A defined by g(i, v) = (Ins( i, v), i) is bijection, with its inverse f + defined by fzy i) (i, v( 3)). Changing variables over this bijection gives the claimed identity.
We apply Lemma to simplify TI, Bound and T3. For TI, the NN.,, y) terms cancel before the summation.
T1 = E (1)) (,
Σ
(a (), o( y)
(66)
c (()())
1> aim
iE (x..)))
(t)e (t)
(t())
E
sa t) u]. (Using Lemma
(67)
(, e o(t)
For T2 cancellation of N(x,, v) yields:
E (t)) (1)) (()
N zo)
B2
Σ
log (0 t) u( 4)]
(68)
f.m e o(t)
N (2,, xo)
1 (
Using Lemmal noting that y Ins( i, v):
σ (()))
(Ins(( 200) log so (wt, 1)(1, v).
Bound
E f, (())
(69)
N(x wo)
For T3, first utilize the fact that N(2,,, 8) EieT(me.))) 1.
T3 (t)e-
((),
Σ
Σ
t. - a(t)
N(I
(70)
1-1 L (E.(..)))
Applying Lemma to T3:
T3 E a(t)e-o(1)
(N(Ins(az,,, u,)) vo) (x)
Σ K
(71)
1 o())
We combine T1, T2 Bounds and T3 to obtain the final DISE objective. Let denote the constant term T3's summand.
CRISE
(xo)
TI T2 B2nundd T3
(72)
o (t)e a(t)
N ú,, log (a,, l) o| + +c] (73)
E
Σ
50( 1)(ú,
[
e -(t) o(t)
N (axe, mo)
24

---

Published as a conference paper ICLR 2026
D.7 DERIVATION OF THE SEQUENCE-LEVEL NORMALIZATION PROPERTY (EQ. T6)
In the fixed-length setting (Sec. 3.5 we assume [zpl K (a constant) for all data samples. Under this assumption, the insertion score becomes iieeinddependent. We aim to prove the normalization property (Eq. 16), restated here for the fixed-length context:
Σ ((x))]i, = K- -K-lan |xzl--
(74)
The proof relies on fundamental combinatorial identity regarding subsequence counts.
Lemma 2 (Subsequence Count Identity). For any sequences 2, and zo such that æ, is a subsequence of zo. the following identity holds:
(Inns( 20)) = N(x,,, mo) (arel)-
(75)
Proof of Lemma 2 We prove the statement Ei,- (Ins(a,, i), xo) N Nax, 20) |a,l) via a bi-
jective proof, by constructing two sets equal cardinality. Let S(x, z) denote the set of index tuples corresponding to all occurrences of a subsequence x in such that N Nxx, 2) = [S(x,
First, consider the set A, defined as the set of pairs (I,j), where / the index tuple of an occurrence of ay To, and is index in zy that is not part of that occurrence:
A = ((1.5) I S(æ,, mn),5 E (1....,/<<o)) V-- The cardinality of A is |A| (x,,,, 20) - |=x))-
(76)
Second, consider the set B, defined as the set of pairs ((i, v), J), where (i, v) is an insertion operation on æt. and J is the index tuple of an occurrence of the resulting sequence, Ins( i, v), in æo:
B = {((i,)) J) J E ((Ins(x i, v), oo)}. The cardinality of B is | = E., N(Ins (z,, i, m))
(77)
We now establish a bijection between A and B. For any element (1,j) E A, we define a mapping to an element B as follows: let the inserted token be v Z.) and let the insertion position relative 2, be == [k E I k < jH- The new index tuple is J = I {i}. sorted. The subsequence zo[-] is precisely Ins( i, v) by construction. This defines unique mapping f A
Conversely, for any element v(), E B, we can define an inverse mapping. The index of the inserted token in 2o is the + 1)-th element of the sorted tuple J; let this be j. Removing this index yields the tuple I= J \ { which corresponds to an occurrence of æ,, This defines unique mapping B - A.
Since a one-to-one correspondence exists between the sets, their cardinalities must be equal. Therefore, |A| [B, which proves the lemma.
Under the fixed-length assumption (20g) = K), the definition of the insertion score (Eq. 7 simplifies because the dimeddependent terms (1 - e -a((t) (moo are constant (1 - e -+))) y and cancel out, leading to the lime-independent score:
Proof of Eq. 16
Eno (Ins (2,, z0))
(xx))i v]
(78)
Emg [N 2o)]
We sum this score over all possible insertion operations v):
Emo (Ins v), ab0]
Σ ((xx)(i,
(79)
Emz N (Ix,, 2r))
6,0
25

---

Published as conference paper ICLR 2026
Eaz
Σ
(Ins i, 20)
(80)
Ezg [Ns,, To)
2,0
We apply the combinatorial identity (Lemma to the summation inside the expectation, substituting [ K:
N( (a mo) |aca]).
(81)
Substituting this back:
Σ
s(
Buzg (N (o)( -12xe) [xx+
(82)
Eno [N (z xo))
4
Since (K [ constant with respect the expectation over Zp:
E [Z no0]
Σ
s(z))|i, (K
(x:1)
K |x//
(83)
Exp
N (2 wo)]
This confirms the normalization property stated in Eq. 16
D.8 DERIVATION OF THE DICE OBJECTIVE FOR FIXED-LENGTH DATA (EQ. 17
the fixed-length setting (Section 3.5 we assume cpp K (constant). This assumption leads to time- (as shown in Appendix D.7) and allows for the exact simplification of the DISE objective into the Denoising Insertion Cross-Entropp (DICE) objective.
independent insertion scores 8p (x;
We start from the DISE objective (Eq 11::
(t)e a(t)
N (Ins(wrr,,, i, v), x00) N(x;,, 2o)
CDISE (mr)
-E f,z; - (()
Σ
50 (2x)))
log 80 (i,
(I,ns(as,, i, u),
+ K
(84)
We rearrange the expression inside the square brackets using the definition K(a) a(log 1).
(Inns(x,, v)) log 50li, v] N (IIns(#; i, x),
[og N N(ns( u), wo) -1 -1)
seli,0] u]
N N(x v0)
N N(x,, xo)
N N(x xo)
= =(50li, s0li, v]
N N(Ins (Ins((ae; i, v),)
N(I,, Dip)
NIInns(a,, i, v), xo) N N(w,, zo)
(Ins(20,,, v),
log
log soli,
(85)
N (xx xo)
We substitute this rearrangement back into the DISE objective:
DISE (xo) = E ot)) C3
Σ
(sl(i,)) w]
N(Ins(zz,ú,,)) i, v), 200
t.e. --(( 1,31
N N(x,, 2o)
N (IIns(2 i, v), 2o)
(10g
N (Ins(( log Li v)
N N(x;, 27))
N N( ma)
(86)
26

---

Published as conference paper ICLR 2026
We now utilize the crucial normalization properties derived from the fixed-length assumption ((æ0l K).
From Lemma (Appendix D.7p, the true subsequence count ratios satisfy:
(Inns( i, v), mu) - [z]).
(87)
Nx, xb)
As discussed Section 3.3 we design the network architecture such that the parameterized scores Sq exactly satisfy the same normalization constraint:
Σ 1,
(i, K |
(88)
Because both the true ratios and the parameterized scores sum to the same value, the summation of the first group of terms in Eq. 83 vanishes:
Σ [o(x)]i, (I,
-K' |
(K
(x,)) 0.
(89)
N(x 2o)
Therefore, the DISE objective simplifies exactly, We define this simplified form as the DICE objective, which is exactly equal to the DISE loss under the fixed-length setting (CDiCE(( L.sisi (2o)):
(t))e0(())
(() Nxz,,0)
log N(Ins (ins(ar,,) i, v)) log 50 (x) (z)) ml
CDICE (zu) (zu)
E
1,
(c)
N(x,,
(90)
Rearranging the terms inside the summation yields the final DICE objective:
20 CDICT(
DICE (xo)
(xx)
= E
Σ
(t)e (())
N(Ins(x,,,, i, (,),
log 50 (z +c] ol)
(91)
t,
N N(w,, 20)
1,
where C' log N(Ins(a,,i..))) is f-free constant.
E ALGORITHM DETAILS
E. DYNAMIC PROGRAMMING FOR SUBSEQURNCE COUNTING
Here we provide the pseudocode the DP algorithms introduced in Sec. 3.4
Algorithm 1 Prefix DP (Eq.
13)
Require: original sequence æo with length m, noised sequence æ, with length Initialize N [w:{: [: i], Lol: j]) 0, Vi,j
Initialize N [x: 0], 25 J) Vj
for j = 1 to m do
for 1 n do in parallel
N i], xol: j]) = N(x, i], xol: j 1]) + 8(x, 1]. æolo li 1]) NNx i i- zol: 1])
end for end for
27

---

Published as a conference paper ICLR 2026
Algorithm 2 Suffix DP (Eq. [4)
Require: original sequence wo with length m, noised sequence as with length Initialize N (zt [ :], 2o j :I) 0, Vi.j
Initialize NV (z, [n :l. Lo :1) 1,
for j = m - to 0 do
for = n 1 do // in parallel
(ax ( zolā =I) Na, (i voli + 1 :I) + 6(xzlt), zubl)) Nixlli + æuli + 11))
end for end for
When we extend the DP algorithms longer sequences, we will meet a numerical issue that the value of (ax, (o) and the values in the DP tables (Eq.
13 14) might be extremely large. For example, when [col = 2048, the maximum (æ, 2o) is (2025) ~ 10614 larger than the upper limit of float64 precision 1030s resulting in numerical overflow. However, what we want to compute implement the DISE loss N(Iss(),,..)).m))) according the sequence-level normalization property (Eq. of DID is the 'N ratios": .v (w).
(Eq. 16 details in Appendix D.7). the ratios should be < |c |æz] i.e. the final results of the N ratios will not have any overflow issues. Therefore, address the numerical overflow of the intermediate DP results, we can transform the DP algorithms into log-domain.
Algorithm 3 Prefix DP in log-domain
Require: original sequence Wo with length m, noised sequence æ, with length n, INF = 999999 Initialize log N (: i), ao j)) -INF, Vi,j
Initialize log N (a, 0J. zol: j]) = 0, Vj
for j 1 to do
for = 1 to n do // in parallel
log N (x t). vol: j]) = log log ((al3, glog 1i 1]) +log N ((n (s11::11), (o0::o--10 -1])
end for end for
Algorithm 4 Suffix DP in dog-ion
Require: original sequence Vy with length m, noised sequence æ, with length n, INF = 999999
Initialize log N (x, [ :], æoli :I) -INF, Vi.j
Initialize log N (xx [n :]. x i :]) = 0, vj
for j
1 to do
for = n - to do // in parallel
elog (10= N(e, No }
log (a, :), moli :)) log
end for end for
which can be efficiently implemented with the 'logaddexp' operation provided by deep learning frameworks such as Py PoTch,, the DP tables now store the logN values instead of values the original version, and lower data precision (e.g. float32) could be enabled to memory. Notably, the log-domain DP algorithms will encounter the logo issue, we replace log0 with a large negative number (-999999 in our implementation). The resulting numerical error of our log-domain DP is negligible, approximately on the order of 10- 13 when
28

---

Published as conference paper ICLR 2026
using float64 and 10-4 with float32. However, the method fails to perform correctly with f and bíloat data types.
E.2 DID TRAINING ALGORITHM
Algorithm 5 DID Training
Require:
Network 5g, noise schedule o, time [0, 1], samples from data distribution Pdata repeat
2g ~ Pdatas ( 11)-
Construct subsequence æ, by removing tokens with the probability of 1 e-m(()
Calculate N (ax i), : 1), Vi,j by prefix DP
Calculate (( i :), 2 :)), Vi, by suffix DP
Calculate N (Inss u), mo) Σラ=
m 5( [i]. v) (a i). mol: - 1)) ( (f :I, voli :)), Vi,
if train on fixed-length data using DICE loss Eq. [7 then
Calculate Lo(x,, 20) = a(t)e ((1) 1-e-ac0)
Σx,
((Ins(a,....)).. aco) N(m
log 50 (x; ec
else if train variable-length data using DISE loss Eq.
11
then
Calculate Lg(x,, = a(t)e 8(1)
Exe
So [i, vj
[[Tns( _..)).m)) (o
log Sø (zt, t) e) + c
I-e-
end if
Calculate VoL(2,, mo) and run optimizer
until converged
E.3 DID INFERENCE ALGORITHM
Algorithm 6 DID Inference
Require: Network 5g, noise schedule time range [0, 1], step size At +- 1, Dt <- [<BOS>]
while > do
((1) T-e-stt (1 [t 1) At,
1 Ø v == Ø
Calculate pf sejo u()x.) =
E po Pt-a/ii Arll (1, 1)) 20)),
Sample insertion actions (i, v), Vi based on pf- ((i, xx
Update insertion actions (1,, v), Vi to construct #t 4- f At end while
The algorithm above for the unconditional generation of DID, for conditional generation, the difference the probabilities for insertion actions in the middle of the prompt should be O.
E.4 PYTORCH IMPLEMENTATION THE DYNAMIC PROGRAMMING ALGORITHMS
LOG_2ERO=-999999 def safe_log (x, ): return torch. (x 0, LOG_ZERO, torch. log (x)
29

---

Published as conference paper ICLR 2026
def
(batch, remain_indices, seqlens, token_dim, spazse=
True):
# patched data alignment
prefix_padded_xt torch. torch.zeros_l 1 ikk (batch, devi ce=batch..ee debischh.eeeiee device) init
@refix_data_mask seqlens None] > torch arrchaaee . (batch. (batch.shape device =batch. [None, ....
prefix
crefix_paddedd_o padded xt Iprefix data data_mask] = batch [
prefix_si_eg si ei__e_t tj (batch. uwsqueeze (-1) prefix_paddeed padded unsqueeze (-2))
profik__i__oltillog torch. torch. prreh...ggrrrfii.sss.......
12
suffix_padded_xt torch. aeros_like (batch, dev.co-batch. 1 # init
as -1 suffix_data_mast seglens None] torch.arane (batch. shatchh
-1, -1, device-batch. devi ce) [None, ...]
15
It m = batch _ indices]
suffix sutii flipleaffix_padee__Mt :1l) uwsquesze _uffix_si__e_ttilog__llf
ftor (batch, [1]) unsqueeze(-1) torch.
18
terch. log (twfffii_sieeeet_______________ fi
S = batch. shape
# prefix si_eq_tj and suffix si si_eq_ts combined
23
combinedd_cq = stack
of
}, dim=-1) pormute (1, 2, 3, 0) (S, s, 2, B)
prefix dp and suffix dp combined combined_dpp torch.zeros (S+1, s+1, 2, B, toneettoch...oatt.. device- batch. devicce) # (S+1, S-1, 2, B) combined_dplle dpli, 01 =1
combined ap safe log (combined dp)
for i r1, S+1):
30
prev combined_dpii-1]
torch. (prev[1:]] co11ined_eq(i-1] + prev[:-1], out=combined_dp
[i, 1:11
prefix _p,, suffix dp combined_dpl1: :, 0] permute (2, 0, 1), conbined [:, :, 1] 11.pert (2, C, 1) suffix_dp torch. farehh.lilsssueeee (suffix_dp, [1, 2))
# N (Ins t, i, v,, 0)) N(xt, x_0) prefix-suffix dp V token_dim N_ratics - []
38
for b range (B)
N prefix dp lb, -1, seqlens [b]] pr prefix dp (b, :-1, [:seqlens(b] +1
42
su = suffix dp [b, 1;, S seqlens + 1:] pr_su (pr +
N) (xp
d6
if sparsc: sparse
S, T pr_su. su_ shape
rews batch [b] unsqueeze (1) expand T) preshape(1)
30

---

Published as conference paper ICLR 2026
cols arange (T) to (batcc..evvic) unsqueeze (0) expand (S, T
reshape (-1)
values = pr_su.reshapp (-1)
mask aelussbbs
1e-6 (S*I,) bool mask
53
rows rows [mask]
cols cols mask
values = values [mask]
indices torch. stack cols], dim=0)
58
N_ratio torch. crcchpsprseccoo_tonee (indices, values, sizc=(V, T))
N_ratios, append ap (N_ratic)
else: dense
N_ratio torch. (V, pr_su.size(1) (1)), dtype<pr_su.dtype device -batch.devices (V, ?)
62
M_ratio, indii indee__dd_ (0, batch[bl, pr_su)
M_ratios append (N ratie)
packed_M_ratios torch.cat (N_ratios,
if sparse: / sparse ret = packed N_ratios.tt (ccalescce( (Asun_b |x_tl_b, V) else: dense
70
ret Packed_N_ree N_atiosst (\sum_ |x_t__, V)
71
return ret
Listing 1: PyTorch source code for the computation of N-ratios
N(nn(m N(
Vi, as described in Sec. Sec.3.4 3.4
There multiple tricks for the implementation of the subsequence counting DP algorithms enhance the speed, for instance, 1) the prefix DP and suffix DP could be combined to improve the data parallelism, two DP tables could be updated in parallel: 2) the memory accessing could be contiguous reduce IO cost by rearranging the axes of the DP tables; 3) in in-place operation could be enabled to reduce IO cost.
EXPERIMENTAL DETAILS
F.1 FIXED-LENGTH TRAINING DETAILS
Model architecture. Following RADD, our models use an encoder-only transformer architecture with a dropout rate 0.02, rotary position embedding (Su et al., (203),, and untied word embeddings between input and output. Other model architecture hyperparameters are summarized in Tab. 7 The FFN dimension 4x hidden dimension.
Size
Layers
Hidden Dimension
Attention Heads
Non- Sane-Emedding Parameters
Small
12
768
12
85M
Medium
24
1024
16
302M
Large
36
1280
20
708M
Table 7: Model architecture hyperparameters for difference model sizes of RADD and DID.
31

---

Published as a conference paper at ICLR 2026
We use FlashAttention (Dao et al. 2022) to support attention computation for packed variable-length sequences, while RADD uses PyTorch's ' standard scaled dot product attention for its fixed-length batched inputs, which is also based on FlashAttention.
Dataset and pre-processing. For fair comparison, we train DID on OpenWebText dataset (Gokaslan & Cohen 2019) with the same steps (400K steps) or compute budget (800K steps) as RADD (Ou et al. 2025), under the log-linear noise schedule (t) = log(1 f), we train DID with batch size 512, and a context length 1024. We tokenize OpenWehTent WebText dataset with the GPT2 (Radford et al. 2019) tokenizer, the vocabulary size is 50257. We follow the data pre-processing adopted in RADD, concatenating all sequences in the training dataset, then splitting it into chunks of fixed length 1024. We add a <BOS> special token to the first position of each chunk to model the insertion behavior before the first normal token by modeling the insertion after the <BOS> special token. We use the same token as <EOS> for <BOS>. We evaluate the zero-shot language modeling perplexity on WikiText, Lambada, Scientific Papers (Arxiv and Pubmed, abstract parts), AG News, LMIB, and PTB datasets (Merity et al. 2017 Paperno et al. 2016 Cohan et al.
2018 Zhang et al. 2015 Chelba al., 2014
Marcus et al. 1993) Tab.
Optimization. For a fair comparison, we did not do a hyperparameter search, our optimization configuration strictly follows RADD, we use AdamW (Loshchilov & Hutter 2019) optimizer with B1 0.9, 32 0.999, e = 1e-8, a weight decay rate of 0.03, constant learning rate 3e-4 that linearly warmed up from 0 over the first 2500 steps, a gradient norm clipping value 1, an exponential moving average (EMA) with a decay rate 0.9999, and float 1 is enabled for mixed-pree -eeeedppreiiioon training. To avoid OOM errors during model training, we set the gradient accumulation steps to 1, 2, 2 for DID small, medium, and large; and 2, 4, 4 for RADD small, medium, and large.
F.2 VARIABLE-LENGTH TRAINING DETAILS
Model architecture. Following ILM Patel et al. 2025), we train small model with 85M non-embedding parameters. Different from the fixed-length model, the variable-length model is not time-independent, i.c. we need to input the time information into the network. Therefore, we employ an adaptive layernorm for each transformer block as in the practice of DIT (Peebles & Xie 2023) (as well the time-dependent masked diffusion models like SEDD (Lou et al. 2024) and MDLM (Shhoo et al.) 2024)). the condition embedding dimension is 128, which brings about extra parameters. To keep the total parameter amount of transformer block unchanged, we reduce the FFN dimension to 3.5x hidden dimension. Following ILM, we use a dropout rate of 0.1, rotary positional embedding, and untied word embeddings between input and output. FlashAttention is also employed for the efficient computation of variable-length data.
Dataset and pre-proces pre-processing. Following ILM, we train variable-length models on the Stories dataset. We tokenize it with the Bert-Base- Unnaeed tokenizer, whose vocabulary size is 30522. Each sentence in the datasets is an individual training datapoint; hence, the training sequences are of variable lengths. The Stories dataset is truncated with maximum context length of 1024. The batch size is 512. Data is also noised with a log-linear noise schedule in the diffusion forward process.
Optimization. Following the experiment settings of ILM, we use an Adam W optimizer with 3, 0.9, 32 = 0.999, e = 1e-8, a weight decay of 0.01 on all parameters (including biases and normalization layers), a constant learning rate Ic-4 that linearly warmed up from 0 over the first 1000 steps, a gradient norm clipping value 1, an exponential moving average (EMA) with a decay rate 0.9999, and 6 is enabled for mixed-precision training. Stories dataset is trained for 60k steps. To avoid OOM errors during model training, we set the gradient accumulation steps to 1, 1, 1 for DID small, medium, and large; and 2, 4, 4 for RADD small, medium, and large.
32

---

Published as conference paper ICLR 2026
F.3 SAMPLING DETAILS
Timestep discretization. We use standard uniform timestep discretizaition grid, i.e. t(1) = i/N, where N is the total number denoising steps in generation.
The number of samples. We evaluate the averaged generative perplexity (evaluated by GPT2 Large), unigram entropy, generation length, and inference time over 1024 samples with a batch size of 32.
Data precision. According to the precision issue of Gumbel-argmax sampling with float32 discussed in (Ou et al. 2025 Zheng ct al. 2025), we use float64 precision for all generation tasks to ensure accurate categorical samplings.
F.4 EVALUATION METRICS
Zero-shot language modeling perplexity is used to evaluate how well a model is trained, which uses the model to be evaluated to calculate the exponential of the average negative log-likelihood per token on datasets unseen by the model (i.e. zero-shot):
logpo(1))) I(x))
PPI. = exp (Beropaaatt)
(92)
where the likelihood can exact (e.g. in auto-regressive models), or bounded (e.g. in diffusion models), and L(x)) is the length of x.
Generative perplexity is a widely used metric to evaluate the quality of model-generated text, whose definition is also in Eq 92 the differences are Pdata is generated by the model to be evaluated, and 0 is another off-the-shelf model to calculate the likelihood.
Unigram entropy is metric to evaluate the diversity of model-generated text, which is based on the token occurrence frequency in a sentence. For a sentence x with length L, the entropy is:
N(z,,)) x) L(x)
x2
H =
loga
(93)
L(x)
where (x; x) is the occurrence time of token r; in sentence æ, and L(r) is the length of z.
G MORE EXPERIMENTAL RESULTS
G.1 EFFICIENCY ANALYSIS OF THE DP ALGORITHMM IN DID TRAINING
To better demonstrate the efficiency of the DP algorithms (as described Sec. 3.4 DID training, we provide more detailed profiling of DP. We report the DP time cost separately (Tab. 9 and investigate the scalability of DP to longer sequences (Tab. [0).. Note that the goal of DP is to prepare the training target and its time cost is independent the model size.
Table 8: DP efficiency analysis in the fixed-lenth training setting (Sec. 4.1
The overall training time and
DP cost for 50 steps are reported (in seconds).
Small
Medium
Large
DID
14.03
27.77
46.60
DID-DP 2.38 (17.0%)
3.33 (12.0%)
3.35 (7.2%)
33

---

Published as conference paper ICLR 2026
As shown Tab. the medium and large models use more time for DP because the gradient accumulation enabled in these settings (set to 2, as described in Appendix i.e. the same batch of data is divided into 2 parts, and DP called twice, reducing the data parallelism and increasing the total time cost.
Table 9: DP efficiency analysis the variable-length training setting (Sec. 4.2). The overall training time and DP cost for 50 steps are reported (in seconds).
Small
Medium
Large
DID
7.71
12.30
19.83
DID-DP 2.13 (27.6%)
2.01 (16.3%)
2.01 (10.1%)
As shown in Tab. 9 the DP proportion is more than that in the fixed-length setting (Tab. 8), because in our implementation of DP (Appendix E..4). we pad all sequences to the maximum length I024 to perform a batched DP, so the DP cost similar that in the fixed-length setting (Tab. B). while the tokens input the network are much fewer, since the average length of the Stories dataset is much shorter than 1024 used in the fixed-length setting. There are still inefficiencies thorough system-level optimization to future work.
current implementation DP, and we leave a
Furthermore, we analyze the time cost of DP for different sequence lengths. Tab. demonstrates the
Table 10:
The time cost of DP for different sequence lengths. line with the training setting in Sec. 4 we set the batch size 64, and sequence lengths (S) as Ik, 2k, 3k, and 4k. The time perform 50 times of the DP algorithm to get the subsequence count ratios are reported (in seconds).
Sequence Length (S)
lk
2k
3k
4k
The Power Law Exponent a (T x Sa)
DP Time (T)
2.05
3.64
8.07
11.2
1.26
efficiency of DP, the fitted power law exponent a 1.26 e [1,2] since the actual time complexity to update the 2-dimensional DP table is O(S²) and one dimension of the DP table could be updated in parallel.
G.2 EFFICIENCY ANALYSIS OF DID INFERENCE
We more detailedly re-evaluate the inference efficiency of RADD and DID fixed-length models in Tab. II and variable-length models Tab. [2] As discussed in (Zheng al. 2024), neural function evaluation (NFE) and sampling are two main bottlenecks of DLM inference, we measure the costs of these two parts separately. For fixed-length models (Tab. TT}. we observe that the NFE part close to the theoretical 2x speedup, since DID saves about half the FLOPs on <MASK> in the inference process compared to RADD. Notably, when the total denoising steps are fewer, the speedup of NFE more significant. This is because the maximum number of tokens that can be inserted in each step of DID inference cannot exceed the current length. When the total number of steps is small, the number of tokens decoded in the early steps will be bounded by this constraint, i.c., at step i, there of DID. Meanwhile, DID's performance is not affected by this reason. On the contrary, when the total number of steps is small, DID's decoding is much better than RADD's, shown in Tab. 2
most 2 tokens input to the model, which brings greater NFE speedup
For variable-length models (Tab. 12p, both NFE and sampling are accelerated since DID does not need to compute the <MASK> and <PAD> tokens.
34

---

Published as conference paper ICLR 2026
Table 11: Inference time comparison of fixed-length DID and RADD trained on OWT. Times are in seconds.
Method
Steps
32
64
256
512
1024
RADD
Time
0.240
0.334
0.507
0.872
1.627
2.876
4.506
NFE
0.080
0.161
0.323
0.642
1.262
2.234
3.308
Sampling
0.019
0.036
0.071
0.141
0.280
0.558
1.115
DID
Time
0.171
0.225
0.359
0.604
1.029
1.818
2.987
Speedup (Time)
1.40
1.48
1.41x
1.44
1.58x
1.58x
1.51
NFE
0.029
0.076
0.159
0.323
0.641
1.131
1,712
Speedup (NFE)
2.76x
2.12x
2.03x
1,98
1.97
1.98
1.93x
Sampling
0.011
0.031
0.069
0.144
0.293
0.589
1.182
Table 12: Inference time comparison of variable-length DID and padding-basse -ased RADD trained on Stories. Times are in seconds.
Method
Steps
64
128
256
512
RADD
Time
0.249
0.452
0.848
1.505
NFE
0.161
0.321
0.631
1.118
Sampling
0.043
0.085
0.170
0.340
DID
Time
0.089
0.135
0.217
0.384
Speedup (Time)
2.78x
3.35
3.91
3.92
NFE
0.027
0.058
0.110
0.224
Speedup (NFE)
5.96x
5.53x
5.74x
4.99
Sampling
0.013
0.027
0.058
0.110
Speedup (Sampling)
3.30x
3.15x
2.93x
3.09x
G.3 ABLATION STUDY OF SEQUENCE-LEVEL NORMALIZATION FOR FIXED-LENGTH MODELS
We provide the ablation study of fixed-length models trained without the sequence-level normalization introduced in Sec. 3.5 i.e. trained with the DISE objective in Eq. [ rather than the DICE objective in
I7 We train FLOPs-aligned models of the small size, i.e. trained for 800K steps for DID models, and 400K steps for RADD. As shown in Tab. [3] the ablation version (DID-F w/o SeqNorm) exhibits inferior performance compared to DID-F, demonstrating that utilizing the DICE objective for training could achieve enhanced performance in the fixed-length setting. On the other hand, the ablation version (DID-F w/o SeqNorm) remains comparable to RADD, demonstrating the reasonability of the DISE objective.
Table 13: Ablation study of sequence-level normalization for fixed-length DID, the zero-shot language modeling perplexity on seven datasets are reported. Results for these diffusion models are perplexity upper bounds.
Size
Method
WikiText
Lambada
Pubmed
AG News
LMIB
Arxiv
PTB
Small
RADD
38.27
51.82
56.99
73.18
72.99
85.95
108.79
DID-F w/o SeqNorm
38.55
50.17
53.76
73.25
72.69
81.95
118.63
DID-F
36.91
48.00
52.89
71.48
72.04
78.38
111.60
35

---

Published as conference paper ICLR 2026
Evaluation Loss on Stories (EMA=0.9)
340
RADD
DID
320
DID scaled by 0.5 (FLOPs RADD)
300
loss
280
260
240
220
200
0
10000
20000
30000
40000
50000
60000
step
Figure 3: Evaluation loss curve on Stories dataset for RADD and DID in the variable-length setting.
G.4 LANGUAGE MODELING PERFORMANCE FOR VARIABLE-LENGTE MODELS
We analyze the language modeling performance for variable-length models, as ILM does not have a likelihood- bounded training objective, only the language modeling performances of RADD and DID could be evaluated. As Stories is specialized dataset, i.e. not general like WebTet, loss curve on its own validation dataset in Fig. 3 where the curves for DID exhibit more stability than RADD's. Besides, also report the curve scaled by 0.5, whose FLOPs less than RADD (at the same x-coordinate) as the <PAD> and <MASK> used for RADD occupy more than 1/2 of the computational FLOPs in variable-length setting, this curve remains lower than the RADD curve, demonstrating the superiority of language modeling of DID over RADD in the variablc-length setting.
only show the evaluation
G.5 MORE INFERENCE, STRATEGIES: COSINE TIMESTEP SCHEDULE
In addition to the uniform timestep schedule T(i) = N. where N is the total number of denoising steps in
generation, evaluate different inference strategy of cosine timestep schedule T(i) cos( _ (1 ) )),
whose timesteps are denser at the beginning of the reverse process (Shi et al. 2024). We evaluate the cosine
timestep schedule using the fixed-length models trained on OWT in Tab. 14
We observe that the speedup of the cosine schedule is more significant (up to 1.96x) because under the cosine schedule, the average sequence length processed by DID is shorter than that of the uniform schedule, i.e., more NFEs are used for shorter sequences the early stage of generation, resulting in greater acceleration DID inference.
36

---

Published as conference paper ICLR 2026
Table 14: Performance comparison with cosine timestep schedule on fixed-length models trained on OWT.
Times are in seconds.
Method
Steps
16
32
64
128
256
512
1024
RADD
PPL
220.74
140.23
107.42
93.72
85.16
87.79
83.41
Entropy
8.24
8.17
8.14
8.12
8.09
8.09
8.08
Time
0.240
0.364
0.516
0.904
1.606
2.817
4.567
NFE
0.081
0.160
0.315
0.612
1.156
2.022
3.065
Sampling
0.023
0.046
0.090
0.178
0.356
0.708
1.414
DID
PPL
161.21
115.54
99.59
92.52
88.14
87.18
87.00
Entropy
8.14
8.12
8.11
8.08
8.09
8.09
8.08
Time
0,212
0.243
0.314
0.461
0.830
1.487
2.536
Speedup (Time)
1.13x
1.50
1.64x
1.96x
1.93
1.89
1.80x
NFE
0.028
0.064
0.134
0.257
0.514
0.955
1.579
Speedup (NFE)
2.89
2.56X
2.35x
2.38x
2.25
2.12x
1.94
Sampling
0.010
0.026
0.053
0.106
0.214
0.429
0.859
Speedup (Sampling)
2.30x
1.77x
1.70x
1.68x
1.66
1.65X
1.65x
G.6 MORE INFERENCE STRATEGIES: NUCLEUS SAMPLING
We provide the evaluation of generation quality with nucleus sampling, a.k.a. top-p sampling, with = 0.9 in Tab. [5 for fixed-length models in Tab. 16 discussed in Sec. [4.] and variable-length models discussed in Sec. 4.2
Table 15: Nucleus sampling results for fixed-length models of RADD and DID trained on Open WebText. Generative perplexity (PPL, evaluated by GPT2 Large), unigram entropy, and average generation length (for DID) under different denoising steps are reported.
Method
Steps
16
32
64
128
256
512
1024
RADD
PPL
95.55
55.21
40.38
34.22
31.79
30.31
30.51
Entropy
7.99
7.89
7.82
7.77
7.70
7.70
7.69
DID
PPL
54.40
36.80
31.73
29.61
28.23
27.29
27.05
Entropy
7.71
7.69
7.69
7.64
7.60
7.60
7.58
Length
1021.88
1023.66
1024.02
1024.12
1023.95
1024.06
1024.12
As shown in Tab, [5 with nucleus sampling, DID generations exhibit both lower generative perplexity and lower diversity (measured by unigram entropy), demonstrating the annealing effect for DID is stronger than RADD.
As shown in Tab. I6 unlike the fixed-length models in Tab. 15 nucleus sampling results of DID consistently offer lower generative perplexity and higher diversity compared to ILM and RADD, further demonstrating the superiority of DID in the variable-length setting. Besides, the annealing effects could also be observed at the sentence-level, the generation results by nucleus sampling is shorter than those by direct sampling reported Tab. 4
37

---

Published as conference paper ICLR 2026
Table 16: Nucleus sampling results for variable-length models ILM, RADD and DID trained on Stories. Generative perplexity (PPL, evaluated by GPT2 Large), unigram entropy, and average generation length (for DID) under different denoising steps are reported. *: as outliers significantly affect PPL., only samples with PPL. < 300 counted.
Method
Steps
64
128
256
512
ILM
PPL
174.29
27.04
14.50
15.31
Entropy
5.02
5.38
5.44
5,45
Length
62.68
110.05
120.75
122.99
RADD
PPL 50.63
28.62
22.21
16.11
Entropy
4.81
5.20
5.39
5.52
Length 96.14
188.64
228.51
265.53
DID
PPL 12.33
12.74
12.46
12.83
Entropy 5.69
5.72
5.73
5.69
Length
161.64
171.91
179.62
174.39
G.7 CONDITIONAL GENERATION
We evaluate the sample quality of conditional generation using 256 randomly selected OWT prefixes under different prompt lengths of 256, 512, and 768.
Table 17: Conditional generation performance with different prompt lengths OWT.
Prompt Length
Method
Steps
16
64
256
1024
256
RADD
PPL
114.46
67.13
58.33
55.16
Entropy
8.12
8.03
8.03
8.01
DID
PPL
70.11
50.98
47.47
48.85
Entropy
7.92
7.94
7.92
7.92
Length
1022.48
1023.81
1023.88
1024.00
512
RADD
PPL 52.31
38.17
35.51
35.11
Entropy
8.00
7.95
7.95
7.96
DID
PPL. 35.29
29.95
29.19
29.00
Entropy
7.86
7.90
7.91
7.90
Length
1018.66
1023.08
1023.99
1024.07
768
RADD
PPL 25.85
22.57
21.92
21.80
Entropy 7.85
7.84
7.84
7.84
DID
PPL 21.64
20.25
20.10
19.99
Entropy 7.83
7.86
7.87
7.86
Length
1011.45
1023.38
1023.93
1024.10
As shown in Tab. 17 observe that DID stably achieves lower PPL with comparable unigram entropy (diversity) and generation length, demonstrating the stable superiority of DID conditional generation performance over RADD.
38

---

Published as conference paper ICLR 2026
Table 18: Ablation study padding length for RADD in the variable-length setting. Models are trained on Stories for 60K steps. Generative perplexity (PPL. evaluated by GPT2 Large), unigram entropy. inference time (in seconds), and average generation length under different denoising steps are reported. *; as outliers significantly affect PPI., only samples with PPL. < 300 counted.
Method
Steps
64
128
256
512
RADD-512
PPL
80.19
57.32
40.27
35,45
Entropy
4.89
5.21
5.56
5.63
Time (s)
0.123
0.221
0.383
0.608
Length
91.83
138.70
191.43
207.93
RADD-1024
PPL
81.92
50.89
34.47
26.78
Entropy
5.22
5.58
5.79
5.85
Time (s)
0.246
0,441
0.827
1.461
Length
110.66
200.73
349.54
353.47
DID
PPL
22.78
21.07
21.90
23.88
Entropy
5.90
5.94
5.94
5.94
Time (s)
0.090
0.132
0.218
0.388
Length 182.31
193.77
202.97
204.96
G.8 COMPARISONS WITH RADD OF DIFFERENT PADDING LENGTHS
As described Sec. 4.2 the padding length of RADD in the variable-length setting hyperparameter be pre-defined, which is complexity for MDM in the variable-length setting, as well as a source of its inefficiency of <PAD> computation. Here we provide another setting of padding length, shorter one of 512, train RADD for the same steps (60K) on Stories dataset, and evaluate its generation quality and speed, the cumulative distribution functions (CDFs) of generation length for different models under different total denoising steps are also shown in Fig. alike the experiments Sec. 4.2
Steps
Steps 128
Steps 256
512
1.0
1.0
1.0
1.0
0
Data
Data
RADD-512
RADD-512
RADD-512
RADD-512
0.2
RADD-1024
0.2
RADD-1024
0.2
RADD-1024
0.2
RADD-1024
DID (ours)
DID (ours)
(ours)
0.0
0.0
0.0
0.0
200
400
200
400
200
400
200
400
Length
Length
Length
Length
Figure 4: Cumulative distribution functions (CDFs) of generation length under different total denoising steps.
As shown in Tab. the generation quality of RADD with padding length 512 exhibits observable degradation, iec higher generative perplexity and lower diversity (measured by unigram entropy), compared to RADD with padding length 1024, demonstrating the reasonability of setting the padding length 1024, which is also ILM's original training configuration, even though it highly inefficient as the average length of Stories training dataset is only 213.43 much shorter than 1024, leading to computational cost for the <PAD>
39

---

Published as conference paper ICLR 2026
tokens. On the other hand, RADD with padding length 512 achieves speedup 2x RADD with padding length 1024, yet still ~ 1.59x slower than DID on average.
Regarding length modeling, as shown in Fig. although the average generation length RADD-512 in Tab. [8] is closer to the ground truth (213.43) than RADD-1024, the CDFs in Fig. 4 still show strong deviation to the ground truth while DID achieves much closer CDFs, which indicates the superiority of DID over MDMs for the variable-length generation task.
G.9 EVALUATION WITH MORE CHECKPOINTS
We have evaluated DID with more checkpoints to locate where the improvements originate. We additionally evaluate the zero-shot perplexity (Tab. using more checkpoints (200k for 25% FL.OPs and 600k for 75% FL.Oss, together with the experiments in Tab. 20 our conclusion is: average, DID could surpass MDM, on the datasets for zero-shot perplexity evaluation (Tab. 19) and tasks for commonsense reasoning evaluation (Tab. 20 with 50% to 75% training FLOPs.
Table 19: Zero-shot language modeling perplexity evaluation with more checkpoints.
Size
Method
WikiText
Lambada
Pubmed
AG News
LMIB
Arxiv
PTB
Average
Small
RADD-400k
38.27
51.82
56.99
73.18
72.99
85.95
108.79
69.71
DID-200K
40.50
50.55
57.87
81.02
76.37
85.86
122.84
73.57
DID-400k
38.72
49.10
55.02
76.02
74.04
82.41
115.37
70.10
DID-600k
37.41
48.40
53.59
72.88
72.64
79.94
112.95
68.26
DID-800k
36.91
48.00
52.89
71.48
72.04
78.38
111.60
67.33
Medium
RADD-400K
28.44
44.10
41.06
48.96
60.32
66.28
81.05
52.89
DID-200k
31.11
43.69
44.36
57.60
62.74
67.80
97.26
57.79
DID-400k
29.19
41.94
40.84
52.53
59.88
63.95
91.87
54.31
DID-600K
28.48
41.30
39.44
49.80
58.76
62.54
85.98
52.33
DID-800k
28.35
41.00
38.71
48.84
58.05
61.77
87.09
51.97
G. SCALABILITY
Regarding the scalability to larger models and longer sequences, following SMDM (Nie et al. 2024), we scale up the DID variable-length model to 1.IB parameters and train it on the SlimPajama dataset (Soboleva) et al. 2023) for 1.6e21 FL.OPs, each document in the dataset an individual variable-length sequence with truncation length of 2048 (trained with the log-domain DP algorithm for longer sequences described in Appendix E and evaluate the accuracy (%) of DID and SMDM on downstream commonsense reasoning tasks (Are (Clark et al.
2018), Hellaswag (Zellers et al. 2019), Obqa (Mihaylov et al. [2018). Piqa (Bisk al. 2020), Race (Lai et al. (017)) Siqa (Sap et al. 2019 and Winogrande (Sakaguchi et al, 2019)) with the LMMBBallation-HHrness framework (Gao
2023 SMDM results are reproduced with its open-sourced checkpoint, with < 1% FLOPs trained on variable-length data.
Table 20: Downstream task evaluation of I.IB models. Zero-shot accuracy (%) ) reported for each benchmark.
Method (FLOPs)
Arc-c
Arc-e
Hellaswag
Obqa
Piqa
Race
Siqa
Winogrande
Average
SMDM (1.6e21)
24.57
48,91
44.37
31.20
65.23
33.78
39.00
52.88
42.49
DID (8e20)
25.17
47.73
40.79
31.20
64.58
31.48
38.33
51.62
41.36
DID (1.2e21)
26.11
49.20
43.98
32.80
65.72
32.06
37.97
54.62
42.81
DID (1.6e21)
26.54
49.37
45.72
32.00
66.16
32.44
38.89
54.85
43.25
40

---

Published as a conference paper ICLR 2026
Table 21: GSM8K evaluation under different top-p sampling strategies. Zero-shot accuracy (%) is reported.
Top-p strategy
Steps
32
64
128
256
p = 1.0 (direct sampling)
SMDM
27.82
33.97
35.78
36.85
36.54
36.69
DID
31.92
35,41
37.45
39.35
39.27
39.88
p 0.9
SMDM
32.22
36.92
37.38
39.58
39.58
39.65
DID
32.37
38.51
42.38
41.55
42.23
43.75
p 0.6
SMDM
36.01
40.03
40.56
42.00
43.37
42.61
DID
38.82
41.39
43.90
45.26
46.47
46.70
0.3
SMDM
37.83
39.88
42.23
43,82
44.73
43.37
DID
38.89
42.08
43.21
44.81
45.49
45.79
We further evaluate the performance of DID on the GSM8K (Grade School Math 8K) (Cobbe al., 2021) benchmark for math word reasoning problems, using 1.IB variable-length model.
Following the setup in SMDM (Nee et al., 2024), we fine-tune the models pretrained on SlimPajama for 1.6e21 FLOPs (both SMDM-1.1B IB and DID-I.IB), on the augmented training data (Deng et al. 2023) for 40
epochs, keeping all optimization configurations strictly consistent. We then assess the zero-shot accuracy (%) of DID and SMDM under various inference settings, including different decoding steps and top-p sampling strategies, using the standard MM-Evvluation-Harree
(Gao et al. 2023) framework.
As shown in Tab. 21 results demonstrate the consistent superiority of variable-length DID model the fixed-length, padding-baan based SMDM approach on the conditional generation task of the GSM8K benchmark.
H MORE DISCUSSIONS
Here we discuss the theoretical insights for the advantages DID over MDM, we additionally summarize four structural advantages that come from using insertion instead of unmasking during generation.
Context-aware decoding order. standard RADD sampling (e.g., r-leaping), positions to update is governed solely by the noise schedule (random unmasking). The probability that any masked position is unmasked in step is governed by scalar function (t(t, s), which depends only on the noise schedule (e.g., for log-linear):
decision of which
I
M x
st,
The probability is uniform across all masked positions and independent of the context æ The learned model only determines what token to fill in if a position is unmasked; it does not influence where updates occur. When steps are few, this forces the model to predict many tokens at random absolute positions simultaneously, often leading to incoherence.
In contrast, DID learns where insert based the existing content (via insertion score). The probability of an insertion after position is directly determined by the learned insertion score (x t):
(1)e
((()
Insertion at position i x.] R At.
Σ 1
(a () [],
Crucially, this mechanism is content-dependent. The model naturally chooses to generate tokens in positions where it most probable first, improving the quality of each sampling step.
41

---

Published as a conference paper ICLR 2026
Self-correction mechanism. RADD relies on absolute positions. Once a token unmasked, its content and position are fixed, so any early suboptimal predictions (common in fast sampling) lead to error accumulation. DID predicts relative order. Even if the model generates imperfect tokens early on, it can still refine the sentence structure in later steps by inserting new tokens between existing ones (see our demonstration in Appendix L.3). This dynamic refinement capability makes DID more robust to the errors inherent few-step generation.
Cautious early generation. RADD works on the whole sequence length from the very first step, so can unmask exceedingly many tokens at the earlier stage using inaccurate model predictions. In DID, we allow most n + 1 insertions if the current context length is n. (For example, the very first sampling step will insert most token.) This means we perform fewer insertions at the earlier stage, acting more cautiously than RADD.
Better handling of adjacent token dependence. Both DID and MDM employ the r-leaping method for fast sampling, which assumes multiple token transitions occur independently at each step. This independence assumption is major source of sampling error. Note that DID will not insert multiple adjacent tokens once, so the transitions of adjacent tokens are never independent. (Adjacent tokens are always inserted sequentially.) In contrast, r-leaping MDM poses a stronger assumption that any pair of tokens are mutually independent. It is fair to hypothesize that adjacent tokens bear more dependence than separated tokens - thus, we believe DID handles token interdependence better.
I GENERATION EXAMPLES
Here provide generation examples of the fixed-length model trained on OpenWebTexx in Appendix [i and variable-length model trained on Stories in Appendix [.2] Besides, we also demonstrate the intermediate generation process in Appendix I.3 All samples are generated by the direct sampling algorithm under the float point precision of fp64 for accurate categorical sampling.
I.1 SAMPLES GENERATED BY THE FIXED-LENGTH MODEL TRAINED ON OPEN WEXTETT
111.1 UNCONDITIONAL GENERATION
Sample
Moon Environment, NEONSEA, CANANA INTERACTIVES. INTERACTIVVSSSEVERRY SEVERRY Union ENTRY,PORTS IN DEMO LEAC.SQUANTIFFCUS. CAZARY, and WACKUP. COvenATION BY BAR WAS PURPOSE TO TEAM DEFENSE, DEFENSE IS THE CHART OF OUTER AND NEGATIVE AND IRONSIDE.
*Selected for Hexagonal Player counting) (Best players from THREE EXPLicates.) of Medals PT50 showed signs of PT in US Switch Bali, SH DROPIONS DOWN HUMAN RIGHTS, CT PORTS were only available for Queensland Olympics USA PAT NACHINO FUTURE Videos reserved. The PACI did mean that, in every four teams from these groups UNMERCHANT WEIGHT CARRYING THE LED ARTS OF BROADCASTING SO NECESSARY DOES, ORBURNVE OFF CONSENTSLY.
Swo.e..PPPtaaYYCCCTEEEE COMICS.COM COM WILL END
CLICK HERE TO SEE RIGDINS OF AUSTRALIAN BUTTE SON BASING<lendoffextbbby Jay R. Tyardots California Scientist, January 3, http://wwwccommiiee.ccelwwwveve technology/124428. htthhooogy//4422.288mmrreeesss
Engineers working on creating a quantum computing field say they' ve made balls of quantum information that is able to vibrate like graphene by adjusting its signals like a natural quantum bus. The result is powerful
42

---

Published as a conference paper ICLR 2026
250-watt device that could be used for computing purposes within a decade after showing the circuitry could be tuned over time.
The research in IEEE's Physical Review Letters, Designing the World's First Atom of Silicon Nanotubes.
Better than the current solid-state transistor of opposable states, silicon nanotubes that slip into a hurdle metal sandwich (DA50N2) create FCU.
The non-energetic bits could be on their way into next-generation "intense" processors and even be used in quantum computing.
The researchers said
"The circuit design delivered a new transistor of exponentially lower power," said Enrique Cameron, a nitrogen-dot tech at Rice University in Houston, Texas.
The transistors are integrated with what the researchers call quantum control an internal signal source designed to tell the circuit of the transistor it wants the mechanism to act on. If a magnetic electrostatic oscillation is expected or unwanted, controlling the quantum control system. The internal signal also controls how much current can be drawn.
Researchers have showed quantum computing in a 256-nd transistor with count of DSNO, but more research needed increase the size and lithography of the chip even higher.
For the first study for such transistor done in the same environment, researchers used unique voltage coupler device. The voltage coupler holds the value of sine wave and information about the direction of voltage and current.
To output voltage the computer acts on the sum of the voltage and current which is what people might get if divided computer transfer rate of 16 Gbit/s (3.24 GB per 100 processors) into half.
According to the group's 's preliminary result their signals- feeding strategy, which has also found a way to allow smartphone to create its own elemental energy, could be widely used at lower power levels. meant then anything involving new electronics is possible as well.
"The speed in which this sort of mechanism lets off electrons through the direction could have really interesting legs." " said Dan Schaberer, professor in the physics department at Princeton who led the design but who didn't publish a copy the paper.
The of a diclectric drive system oscillating the same spirals the actuators provides more mechanical control.
"This is analogous to acceleration using laser in phased scanning microscopy," Cameron said.
Naturally may be difficult to get like that, but the MIT team said it was using some rough theoretical calculations in the paper and details of the initial version have been tweaked within the theory it is possible.
Guy Lee, different team at the Technology University Berlin, who's the scientist who developed the device, said "for what most people would call rapid development challenge team's 's concept is utterly unique."
He added the team is interested in using future fash-memory chips, for example, as a sufficient unit of mass for quantum logic converters.
"Once quantum bits are developed as nanocomposites and lithography problems solved, then I will not be surprised to see all the fantastic uses come up." Cameron alndoolendfttt are some Bitcoin events and meetups we were hoping you found important information about and why.
43

---

Published as a conference paper ICLR 2026
Lindsay Reslove is a research analyst with Coin Private Wealth Research. She is a student in the MSU's departments of business administration and computer science. She and John King, a business in Flint, Mich
Sample
barbecues, unstudded haircuts with a passing connection to classical musicians and singers, African stars Bakawisi Sim and Kutimbla Grime. In Case of The Grace of Contessa, the speaker describes Chichamaster Rupert Brecht's interpretation of the Chop Talk, up close to Hoop Orchestra and the Brooklyn Hall Chop bust. The Volk Attendant dramatologist Philip Hill's take on exploring Bartoz's interpretation of The Young God (Directed by Geraeus Castoré); Penelope Vernon's 's superb books on Mijunset's 's Frativity, the Howeñal movement and other forms of Esperanto music are only one many created in the United States, Her favorites include the e. Additions Games (1811), The Anomal Valley Merry Stories (1815), and the Nieos family Magazine (1856).
The Continental Brand, the dramatist Herbert's Own Dame Vis, introduces English slcuthmuths Ian Woodward and Charles Feching, and doesnt yourself to the first meeting of witty, energetic and from San Diego. San Diego the Altun, inspired by series of articles by Santiago Contamas and José de Septé, the American Innerts, reflecting the peak of the Esperanzaism American Renaissance, tells from the air of Sclepre's Esperanto sophisticate Avarula Grothomo, desperate person suffering from fibresesticular use (as in the processing of rhubarb). This city may have gone to war or appeared beneath man's 's feet. Domaniic unic is not the only thing involving Esperanto: the Atl-mas have described plague. But apparently that never existed, skisted.<enddofiextttSaass <<endoffeexttSSnngefenn The Yoruba businessman whose family and children lived in a small sarandi (boat) before the Typhoon Haiyan destroyed the last homes there is thought to have been safe in the Philippines for at least next few weeks. Philippines Home Minister, Peter Bola, told on Monday that authorities had discovered a connection between Mr Shinse and the latest Islamic anti-American manifesto during a vitriolic sermon he gave last Sunday at a funeral.
Spioclasts
Judging from the "manifesto", Mr Shinse fits the double-standard of the dodgy wealthy Chinese lawyer, where his son had an affair.
His son said is seen as a "gynoid, disabled in body, deracinated". He said he mentally unstable. "But that does not mean I'm happy."
He was approached this week by a Haiyan victim. "I broke it getting in the plane, but said I thought might be good to find it," he said. This man said that the skull bones in his basement had been removed and awaiting identification.
Throughout last week and half with displaced people his nephew has approached outside his office to identify themselves. He recently helped organise seminars of the Islamic branch in the Philippines.
"I keep everything tapes," he said. "Every morning have to stuff it out the window in front of the TV screen, I can carry it anymore." "ermmore"<lendoffexttti "<lendoftextt>Thres hundred revolutionary followers, were took part in Guatemala's Bernardo (Barefoot) alongside 50 Venezuelans, 20 Ecuadoran asylum seekers, to celebrate the revolution, giving a talk-outout to the US Presidential candidate Sanders for expressing "cynicism" about a Socialist state.
"This shows that democracy is not so had in the world," Patrick Dubo, one of the Trotskyists from the Unido Brigda Party said during the rallies.Some of the crowd continued to protest for Che Guevara and Fidel Castro, as they claimed that they supported "separatism".
"In the internet came a Facebook group which is affirmed to be an alternative Socialist state which fully affirms the concept of "Aprrtheid" one unnamed person told the press.
Dubo said that they were representatives of anti-socialist Christians which promote Marxism and fight im- morality through government policies. He added that socialism has never really disappeared in South America
44

---

Published as a conference paper ICLR 2026
because its ideas are with more with "leattbladiig". OOn the other hand self-styled "centrist businessman" Ric Williams fully supported the courage despite this by arguing that expressed the spirit of patriotism.
Reporting Daniel Jenks, Elias Osauga, Thompson Salazar and Guy Aveculs
..<lendoftexxt>hiis week. Dow Chemical Co. CEO George Lopulos finally gave the company's 3,000 employees fresh concessions that break codes and specified overhaul of operations. By Jan. 25, they were signed a tentative contract that would represent more than a third of some 5,000 workers. But while the original tentative agreement was set for the end of this month,
Sample
"It came from someone who felt the urgency of and refreshed us."
The idea of an airport being a catalyst to switch venues appeared at the time. The airport has been home for a decade, with facilities real and perceived be as nice as they come. Boston has a few Green Card 250, excellent, venues. It is not the site of 60 such ad goff-related contracts according to a report from the AP. Taunton was affected by that, as was the clutch conversion on his Jan. 10, 2014 road trip that made national TV headlines.
He slid perfectly from outside the Patriots' six-yard line and 25 yards out before flipping over Willy Chopade Jr.'s end zone. This kept him on the field with Patriotsmate Moss.
"We were hoping we were still going to be playing because of Moss," Haley said. "We just came to grips."
Boston has shown growing appreciation of Shifkar's vision, and now he has the bang of stadium it will live for. After departure of Kow, its most active sponsors, the Patriots purchased HSBC. The successful discussion of other offers and candidates included Shad Khan, partner who does global retail on the West End of Oxford.
"We were hoping to go to Fenway and put identity more behind the location," Shikbaram said. "We are excited about what we can do under the roof but and that's part of the truly team experience. Even before we got the bowl, this is a sport that runs. Fenway Park is great atmosphere for guy NFL foot."
Haley said he can harness that momentum for 2017 based on the gambling on his offensive outlook.
"The other guys are going push us." Haley said he and his Patriots will return championship opportunities. "I said to the fans it'll remind us of being among the greats and being the champs and champion. They are going be the lynchpin in this Big Game. And it's not because they live here, but because they're 're going be the stick. Meet them, run them out of sight. To have people, that's not something that other teams have done in NFL history."
We covered how the Patriots approached January's 's New York vacation trip at The Syllabus.com com and their spring season traveled since June. Experience our very unique 2015 Mobile Guide at http://quotepan.com
Related-cenndoftextbb. more detailed revelation of Viking's dramatic rise is now available. But The Mammals tended to rely on land expeditions glean information. Now that more information about the world without computers or maps could accessed.
Wendy Robinson, The Arctic Lion
New Viking Denials Confirmed by Arcologists
A underwater trench once paved off off the UK coast in Viking engineers' location claimed by locals. But drilling inside the channel, sealed for year by the distortion of the erupted Swarbrough volcano, one in November discovered otheral signed of submarine submarines working against the foraminiferous seawater, accessing new information about its tunnel depths and carbon sources.
45

---

Published as a conference paper ICLR 2026
Throughout the last century leading officials swore the Viking civilisation was dead, but now archaeologists finally know why.
Adrian Ashleyman, minister in charge of the Western Isles, covers every mile of the Viking civilization for thousands of years up until the 10th century, in a presentation at the Ocean Foundation. "It's amazing how much they travel and the work has been done to get where they are, but it's pretty awful to pull nails in far schedule. The lines need to be so easy to get across"
Click see the BBC report.
Viking rune, USSR, Viking generalsfeendofeett-eeeeme former whistle-blower, Edward Snowden has sent journalists and world bloggers cocktail clippings of a New York-based government security training, Mounties Sustainability Training taught by British Commando Troopers, it's an excellent venue for journalists to get their information.
Related myths:
We hear that the U.S. may have bit of a rock on the memory of its golden-west. Snowden has succeeded in making the FSA comfortable shedding its hard currency, boasting to the Guardian recently that the store in which he formed his fortune in Iceland was valued at a staggering 256.5m 556.mmil..o..
It's no different to Volkswagen World Group's 's overpriced program and supermarket boss, Valentino Rossi's private company which has been working on technology that can be based on Italian 500's Series 7's.
This thinking causes problems. Companics in Germany and Austria license special patents and sell them only companies that have record of
Sample
its proposed rules were filed. Seeking amend the proposed rule, FERC viewed the strong language put forth by an association opposed to the exemption when it met with the statement in a letter from Dan Stern with the Sierra Club. Moreover, the National Farm Bureau and the Federation of Sportsmen and Fishermen, which owns lands for conservation, both backed the new "no-Mine" rules. But FERC didn't 't allow for this. "We had try to craft an artificial, complicated substantive rule where power was responsible for conservation that the industry doesn't t want at all," Butler says.
The new RRE rules leave farmers with only title to stake each acre of unused land to mine copper. Buildings must be fewer than 10 feet tall, and need to fitted to VVAC pole to allow the same air flow. No more than 25 acres are permitted and 30-acre farm can be mined using hydraulic fracturing. They have to prevent the potential mining operation, built fence, took part in field tests, and gathered evidence from the site of having found copper that has trampled on production lines.
FERC did not dispute rule specifics-ni specific copper facilities the no-mine would be laid-but did release an assessment saying that public processes would determine "the cutting and the use of pipes" and just what amount of public land to be included in maps should be limited. "It's s all on market herhorse," Slaughter says. "It's facto emissions test."
Nutter warned that the regulations might actually result in "a more anti-competitive system" than one that does allow for polluting, subdivision and enforce strong rules. "The industry does not see right and rejects the idea that it's worse because it increases pollution." " he says. "The Public Works Committee would find it even worse."
Public approval for draft rule will reportedly take 12-16 months as well as legal coalition of ex-elites gathered at Laurelview High School this past week. They 'e part of the Coalition (recent graduates and teachers from urban county were touted for tenure) then showed them how to fight neighborhood segregation. They are so-called Chiefs in the program, and they recruit more black kids through. They simple this practice that is commonplace in large cities and observed mo-Confederate celebrations.
46

---

Published as conference paper ICLR 2026
I joined Michael Smart, coordinator of that program, at his rock concert, Rebel Wars, and watched video of Confederate flag shootouts outside the program.
The group of eight black students lectured and asked teacher Vincent Henderson about characteristics of black communities.
Potish, "Most of were in tears." following decades of history.
"Then we were surprised," Henderson said. "We were almost sure we had it with us."
"We were thinking about it," Smart said. "We just weren't 't expecting it."
"The CMO brought us back to races of neighborhoods that are African American. You'l re talking about lead epidemic and youth unemployment numbers," Smart continued. "You" ve reported dramatic employment loss on top of low personal drug using. And you's re gonna see a lot of student prejudice and crime in the process."
Though they only few seconds to tap into the students' attitude with statements like "Motivation, Director of Excellence," officials did acknowledge the past positive attitudes and report they were raising them.
I heard much more Cal sentiments from the rest of the presentation, albeit somewhat-coarse ones.
"They did not apologize for classroom clashes, not even the token degree," Smart told me. "We're not converts."
They made light of the school's efforts in Combat Simulator, battlefront that drew resentment after the Burbank response of black students.
"They want know about our diversity program," Principal Frank Fund interrupted. "They make assumptions about who wants them to live and police where they want them to live."
"We actually want kids like these," " Fund said. "Eventually, that'll be the biggest part of it."
Historically, many in Chicago's ' gated neighborhoods have been little but crippled. Since 1992, a handful of private universities where black leads have developed detrimental regime where they send low- income students from the disadvantaged through Northern schools to improve opportunities for minorities.
For rural district full of Latino immigrants, these black-only programs have Tucson's black students underperforming at disproportionate rates to kids from whites, Hispanics, and other groups. The effects are little-notiedd, and rarely openly presented.
Jew's album, Jazz is now played widely, performing nationwide and last year winning an award in collabora- tion with the Sierra Club, and the Parents Association of Arizona. They call on the district to address low rates of distance and point to the research that comes out against Unified.
"We're putting black children at a disadvantage," says Jew, who has added Jazz to Spotify himself.
Sample
Dr. S Scott Road, LPM, NY 57025 Hours: 7P.M. - PM (10 a.m service; 50 foot Zion TeaPot; chef Beleo, LPMO, chefbeleo.com); Dinner at 7 p.m Tickets online H22D through tickets [404] 775-624-3404, public (202) 769-6678.
Website, ChefBalleo.com
around A number doors with bays to divert between two types of facility. One venue houses the "Em Lion" training and the second one for Saturday night gatherings. Event tickets will given out hand handed with or without ID to pay. The vast majority of event will be inside the
Line 4661446774endoleext--cceeea
47

---

Published as conference paper ICLR 2026
Center for people to catch up with our manager and owner free ops. For partygoers there will be entertainment at every level from the alley and sliding ramp to hooters skateboarding, and pools.
POLRY days temporary weekends and dynamic peoples schedules
Saturday band on all day never has before. Neither school nor college
Must use good for jams between the residence drive to police
Only vintage, topping, tattooing (except for Halloween) and older patrons update our article in two weeks
Place Events
Chocolate: 6500-1 1000 people coming to town Night manager is on 24 hours and has control of the event (transportation times, gives permits Huppi-Picte PPcckrs,, funding license to city agency claim, and team with the owner and operator of the business] 300 people Total event is double 300 Children Lengths must be > 30 old chance
Music will be live hand tapping (as will Ad and Staging throughout the building)
Special ticket bidding "Cool City" ends Jan 19 which will last the day after.
Registration 5 slots as long as per picture the touch machines using 5 keys Trans-card from Ticketmaster will fund rates Food Trucks/ door.
The event is after the princess's reception is in each auditorium featuring live music, jazz, and theater as the walk out athlete camp commencement rehearsals before night. There will be separate event for kids get to these and other things for the kids to meet the party.
Tickets
Pick up on the private chapel events etc. these are not major facilities. Last come first serve
Tickets: 10orless + fees
Believe that the jQuery art space is a space that is always open
Saturday operating license called business day city license No license 95treatandf oodservice Entertain to the Saturday breakfast event
Featuring a dinner atmosphere at most events.
"Lovers in Trust" Auction designed to get kids and people from the neighborhood get to know some other Cent hours chance to win charity items
hold laptop and tablet inside during the Emperor's Lion Training session.
All photos have been stolen from the art event
Security department approved for benefits only.
Notes
Lowers: okee
(From @shermark)
Mairesoomaa,,
48

---

Published as a conference paper ICLR 2026
Here is the first image of some of our staff that have gotten hurt in this space Pic (from Dunader)
How to comment well from the months in this space based on being able experienced in so much politicotation
5. Letters
Here is season 15 video of Elsa on Lierson handled this space and what is happening, how to congrat- ulate staffers (see here for first time public announcement October 2005), how to give free tickets the wromclendoftexxl>> This spring, meet at a Chipotle University of Iowa, Ames location, representatives of the Chipotle's signature chicken giant said.
An event has been put together by the Native American Union for FEED Friday from 9-11 p.m. Details on exact locations for the event weren't 't available.
Si que héres Inigo, SE ANNOCALE CHICK 10 - Inigo for the 27th date on May 7th!! Equalization (@ Cornish Alliance Equalization) March 27, 2016
Click here for onscreen printable image of the location 700 Mr. Petty Avenue.
Privacy
In the release, which had more details on Chipotle. the University's 's College of Agriculture and Economics said, "Georgetown and its partner specializes in protein, pirouin, cellophane, and tinder-minder products."
Meanwhile, more past Chipotle showings:
LUS GRODYS Club
LatINO STARS Launchday starts at 2 a.m. Friday when Las Vegas's ' Graduation Day Lounge will be on hand (bar not applicable)
9 p.m. on Friday, April 7.
Shillam
I.1.2 CONDITIONAL GENERATION
Here we provide conditional generation examples generated by DID trained on WebnWetText, we set prompt lengths 256, 512, and 768, and the prompt parts are colored in blue.
Prompt Length 256
with the Hawks with view identifying why, they say in the song, Hawthorn is such happy team. Which raises the question: Are the Hawthorn players happy because they have won the past two premierships is Hawthorn winning more because their players are happy? Or, the Hawks themselves suspect, perhaps the answer lies between. Not only does that club feel validated in their work practices, which this season have included an unscheduled day off after their first Launceston game, but it could also use its position at the top of the off-field ladder as it works behind the scenes to mount a case against the AFL's move in taxing clubs' football department spending. That new tax will be reviewed before the end of next season and Hawthorn are looking to lobby for the exclusion of such welfare initiatives such as the regular monitoring of every player's mental health by its sport psychologist from the new tax. Another area the Hawks believe should be exempt is the sponsoring of international educational trips for its staff. Hawthorn is not the only team that claims to be focusing on better work-life balance for their players but they have worked more diligently to identify red flags among their team since Travis Tuck and his mental-heae written by the Herald Sun on Friday night, after a sustained discussion of players' struggles and the potential issues associated with their mental health, Tuck's story has been arguably the story that has generated most interest in the club-induced Australian football story. The responses from sceptical media outlets are mixed, which is through no fault of some of those who have the hardest situation to complain about. The story
issues became front- page news. Originally
49

---

Published as a conference paper at ICLR 2026
may need more interpretation, though, as the AFL already has its own legal <lendoffextttthhwwwbeet released in November 2011, makes it clear what club toolsbrew can do for your body. The key thing is its personal kryptonite. Though Artur Kovács shares "late almonds" with his fellow Olympic teams' training athletes Shawn Spink, Vladimir Sinaseck, Marcel Osgood and Vladimir Camence, there is little to nothing that these players seem to have done their own way in strength training. Put it at the bottom of lurk's common ideas, like this when asked about what he has done.
Just two or three years ago Kováes had completed silver and bronze Olympic medallists, finished second in the massagedet, and was silver medallist at the World Cup captain of the men's Illinois University team. An exceptional athlete, he had vaulted from the school's Is single de facto medallist table to the top ranks of world leaders in power. On eight days of running 10,000 yards, with an intensive training schedule filled with time skating down Pacific Hill, he made all of seven dozen major championships in China.
After dumping back to the ranks of UCLA in 2013, he suffered almost miserable start to his 2012 season, collecting six bronze medals in 2012 and a Cambridge Olympic diploma that year. He yielded fifteen better performances all year, which, after his limbs started teething dry, the San Francisco Nationals player "autureted out".
Analytical reports have found nothing happened, and he faces a battle to climb back on his team loaners following his November I 1tth release by the Massachusetts Athletic Department.
It have not been very easy. Less than one Friday ago, he went bad back and guilty of punching the face of a wrestler. And in recent days. his approach to appeal has fared poorly. Last season, after a mid-season torn fall, he tank-junked in an US court of law with his lawyers threatening sue him unless he addressed his alleged doping confession publicly.
The criticism was even louder this season.
Image copyright Getty Images Image caption Jovan Penscault is in turmoil in the NBA
Ask a few probing questions, in Belnod's case as well; he is asked explain his appetite, his courtship with his parents and whether there had been a bad day in the offseason. Yet the assumption is not that he is emotionally disturbed. He is not unfaithful or arrogant. We believe, in his eyes, that outdresses are just trying to achieve their own goals.
Character can have positive effects, however, and even if there are further positive developments in the NBA, the progression of such a controversial player is critique of how US sports and the game works. It is certainly a negative sign and if anything ought to change about how it works to bolster a program.
"Coaches used to talk about their tennis stars in majors. Now they' 'e got them everywhere," says Gary Smith, the head coach at Penn State, who doesn't t overstate the size of the reward for those who played in the nation's s top SSQC or were willing to put in athletic service. "Look at what lot of your better players did. I mean, you can't
Prompt Length 512
with the Hawks with a view to identifying why, as they say in the song, Hawthorn is such a happy team. Which raises the question: Are the Hawthorn players happy because they have won the past two premierships or is Hawthorn winning more because their players are happy? Or, as the Hawks themselves suspect, perhaps the answer lies between. Not only does that club feel validated in their work practices, which this season have included an unscheduled day off after their first Launceston game, but it could also use its position at the top of the off-field ladder as it works behind the scenes to mount a case against the AFL's move in taxing clubs' football department spending. That new tax will be reviewed before the end of next season and Hawthorn are looking to lobby for the exclusion of such welfare initiatives such as the regular monitoring of every player's s mental health by its sport psychologist from the new tax. Another area the Hawks believe should be exempt is the sponsoring of international educational trips for its staff. Hawthorn is not the only team that claims to
50

---

issues became front-page news by way of a Published as a conference paper at ICLR 2026
be focusing on better work-life balance for their players but they have worked more diligently to identify red flags among their team since Travis Tuck and his third positive drug strike.
While the players' union continues to investigate the reasons behind the grievances of so many of its members, it appears beyond doubt that the demands of pre-season training have taken a disproportionate toll. This, along with the pressure most footballers feel, even during the bulk of their holiday periods, to report back for work in perfect condition. The inescapable conclusion drawn by Marsh and his team that the expectations placed upon players over their recently increased leave period have created pre-season before the official pre-season. According to the AFLPA and the increasingly shared belief of their leading players, the game spectacle would not be adversely affected by inseeinteee spring-summer all. is a domestic sport. And the union are also monitoring, as reported by Fairfax Media, the link with the growing injury toll. Marsh was reluctant to detail individual club player complaints but, according to his key executive Ian Prendergast, the overall picture of dissatisfaction with their sport by the players "hit him right between the eyes". Marsh came to the AFL from cricket and lack of enjoyment with the game they took on through love and fun and enjoyment as children was relatively non-existent among Australian footballers general. Over the past decade or so, the dissatisfaction grew. They were routinely asked how much they needed to goof around by their parents and by their coaches, which they tended to miss. Naturally, the players often had their doubts about their first grade standard and few fans made allowances for this chipper criticism when they backed their team throughout the long pre-season preparation period.
Australian football, after
The AFL and NRL clubs have recently been struggling with growing talk of missed kickouts and sudden increments in the grading system after their clubs were caught failing to properly police Josh Boyd and Scott Lilly. In Canberra, where some of the players are running faster and talking more about drug use, the federation is testing the AFL's new unti-discrmmination law, how to change the rules governing strip derangement and the expanded power of post-match sanctioning cameras. Before games, the representatives from the Sydney Maroons board, including their New Zealand captain and Sydney's 's assistant coach while in charge, have delivered long diatribes to the players, to the players themselves, some of which severed their ties the club in return for the shortsighted and grievous attacks on the club's reputation and its unity, There was considerable assistance from Sydney in counting the tally of the AFLU's members, from the club's s financial manager Damien Gray and chief operating officer Humphry Rokakley, to ASADA director- gentral Rods del Fuerta and former AFLAUT president Chris Jacobs. Unfortunately, Stephen Parish, who became the Australian AFL. executive president once and for all, has told a Senate inquiry that it would be "unusual for any player not to have remoited from this federation's 's support when they first entered the competition formats- and especially so if most do not do so immediately in an effort to improve their competitiveness personally". Australian AFLU president Arthur Hurley and the AFLU's chief executive, Stephen Lawrence, have recently left. Retired federal team officials and CEO Stephen Hendell, Hurley and Lawrence were all also members of the most recent AFLU national advisory committee of the WFWA, the new confederation formed before the federal government's 's new sporting governance regime. The refs and some of the former clubs are quite possibly expected have already caught on and most have said that the new local-territory administration is very unlikely to take robust legal action against the only surviving Australian football union. According Jeff Pearce, the AFLPA's national vice president, the league's new federal strategy now outlines the significant opposition to place on competitive
Prompt Length 768
with the Hawks with view to identifying why, as they say in the song, Hawthorn is such happy team. Which raises the question: Are the Hawthorn players happy because they have won the past two premierships or is Hawthorn winning more because their players are happy? Or, as the Hawks themselves suspect, perhaps the answer lies between. Not only does that club feel validated in their work practices, which this season have included an unscheduled day off after their first I Launceston game, but it could also use its position at the top of the off-field ladder as it works behind the scenes to mount a case against the AFL's move in taxing clubs' football department spending. That new tax will be reviewed before the end of next season and Hawthorn are
51

---

Published as a conference paper at ICLR 2026
looking to lobby for the exclusion of such welfare initiatives such as the regular monitoring of every player's mental health by its sport psychologist from the new tax. Another area the Hawks believe should be exempt is the sponsoring of international educational trips for its staff. Hawthorn is not the only team that claims to be focusing better work-life balance for their players but they have worked more diligently to identify red flags among their team since Travis Tuck and his mental-health issues became front-page news by way of a third positive drug strike.
While the players' union continues to investigate the reasons behind the grievances of so many of its members, it appears beyond doubt that the demands of pre-season training have taken disproportionate toll. This, along with the pressure most footballers feel, even during the bulk of their holiday periods, to report back for work in perfect condition. The inescapable conclusion drawn by Marsh and his team is that the expectations placed upon players over their recently increased leave period have created a pre-season before the official pre-season, According to the AFLPA and the increasingly shared belief of their leading players, the game as spectacle would not be adversely affected by less-intensive spring-summer, Australian football, after all. is a domestic sport. And the union are also monitoring, as reported by Fairfax Media, the link with the growing injury toll. Marsh was reluctant to detail individual club player complaints but, according to his key executive Ian Prendergast, the overall picture of dissatisfaction with their sport by the players "hit him right between the eyes". Marsh came to the AFL from cricket and lack of enjoyment with the game they took on through love and fun and enjoyment as children was relatively non-existent among Australian cricketers.
"Maybe I' I'm being romantic about sport, but I do think it should be fun," was all Marsh would say on the subject. "It's question which needs to be addressed by everyone in the industry." " The AFL is not unique in that they seems paralysed to act against the increasing pressure and negative impact of social media and the uglier side of the expanded modern "selfie" syndrome so hauntingly articulated by Chris Judd in these pages. But the AFLPA have identified player grievances it can address in their next wages and-conditions deal. Heading those grievances are the plethora of meetings the players believe over-punctuate their working week and many believe have grown as assistant coaches over-zealously justify their jobs. Another is the growing demand of player appearances by clubs working to service members and sponsors. The concerns of the players' representative body were compounded by their pre-season round of visits but were already being addressed after the second annual players' survey last August. Players responded to questions across three areas: their club's culture; resources and its structure, which included leave periods, time off and the performance of player development managers. The AFLPA reported back to the clubs, telling them where they were ranked in each of those four areas. The annual research session was six-day. Those clubs in these four areas that showed the worst results at all times and where their own poor progress was identified as the most apparent, such as the Giants, Hawthorn and Carlton, were laid-out as operating not only without the frequent parliamentary supervises and the more outspoken directorial redistribution policy of the Granny Hill clubs and where the players most often are active in their pursuit of the best and fairest - but they also operate with the most lax end-to-end performance-relatt management and team performance
coaching policies, have the most overworked executive list performance-mmmngemeee management staff, and rank among the worst in their management of ticket market participation and other competition activities. Determination of future action many clubs were not only diluted in terms of the performance of key people in the sports knowledge- based management structures such as communication engineers. performance support managers and the managers of the union candidates, but also the clubs' player development agents, tryouts and on-balling cadres. Plus, this was punctuated by the poor performances of the club's executive list manager and model player association executive coach- who was rated below average.
"The Blues said they wanted
1.2 SAMPLES GENERATED BY THE VARIABLF-LENGTH MODEL TRAINED ON STORIES
Sample
52

---

Published as a conference paper at ICLR 2026
once there was a boy who liked to explore- even though he was small. every day he" struggle to see what he could find. he would pick small foods in his garden and visit his neighbours. one day, he noticed a delicious sauce on the food. he was so tempted to try out! he struggled to eat it, not as usual, so he decided to buy a popular pasta with sauce. he shared it with all his friends who enjoyed their sauce. but, he refused when he urged them to keep buying. in consideration, someone called out to him : go ahead, try the sauce. what if the sauce means you won it have to really savor the taste? the boy was happy that he couldn t resist the temptation to try the sauce. from then on, everyone in the neighbourhood remembered the popular sauce and dreamed of becoming more daring sauce expert.
Sample
john was very popular three year old boy in his town. he liked to play in his backyard in the grass, every day, he would imagine all the different things the world started to offer. one day, john looked up in the sky and came up with idea. he got off his bike to learn the wisdom. he rode back to the meadow and told the wise owl about the wisdom. the owl said she said the world was full of special and sweet wisdom if he was patient. after long day, the wisdom yielded lot. john thanked the wise owl. she had worked hard and was able to learn something magical. john continued on his bike, soon out of the meadow. he saw beautiful butterfly and smiled as attractive as she had seen him.
Sample
tim was playing in the park one day. he skipped around, shouting. it was game but there were no toys around. he was feeling disappointed and soon saw a boy. the boy was a lot bigger than no other boy but no one ever seemed to understand. he recognized from a playground. the boy asked if he wanted join him. tim hesitated at first for few moments. but he felt a bit scared, so he decided to follow the boy. he started to climb up the ladder at first but as he went higher and higher he felt helpless. the boy saw tim and said he was looking for help. tim was gentle and asked to try to lift him up the ladder but he was very worried. after a while the boy said goodbye to tim and ran off. tim felt sad that he had to go home without to help a friend. he regrets climbing the ladder.
Sample
timmy and lily are friends. they like to play together. one day, they see a lady on bench. she has big bag and red stick. the candy is sweet and bitter. lily wants to eat candy too. she reaches into her bag and takes the candy. no. lily! timmy says. 'hhtt is my candy. you can" t have it. i want this candy. it is a candy for you. lily did not listen. she does not want to share her candy. she tries to part with her candy with the stick. but the stick is stuck. it stays loose and hurts her teeth. ow! " "illy cries. you broke my stick! timmy laughs. he drops the stick and pushes lily away. no. thank you! 'illy says. you can 't have my candy. go away now. tom does not find another stick. he did not know how to have it. he bites the stick and pulls it out. " look, lily, am eating candy! tom says. am eating kite? " lily looks at tom. she saw what tom was doing, she feels silly and scared. she takes off her hat and her shoes and pants. "opss! sorry, tom, "iimmy says." i did not mean to scare you. just wanted to bite the stick. maybe we can play something else. " "" know, tom," " lily says." but still like the stick. it is better. you can have another candy. " tom feels bad. he also likes lily. he thinks lily is generous. he gave lily big, red apple. here, lily. he says. "tiis is a cake. it was not magic. it was a secret. " thank you, lily, tom says. but next time, do not eat the stick. you are a good friend. can i help you? lily smiles. she is glad that tom could help. she was generous. okay, lily, tom says. i will share the stick. we can play together. they play with a ball, kite, a book and other toys. they have fun with the ball. they are happy.
Sample
once upon time there was boy named jack. he was worried because he had no worries with him, every day him would carry jack away for a day at work the first time to eat a yummy lunch. one day jack saw some of his mum outside. they were so kind and helpful to all of them. they did not put heavy box at school
53

---

Published as a conference paper at ICLR 2026
or they brought any more delicious treats. jack told his mum about this because he went to work looking for something else to do. jack soon had an idea he bought a lunchbox and place all of his mum lunch. he carried each piece, and then drive home. his mum was so relieved and smiled. jack was so relieved and happy. from that day on, on. jack always carried his lunch with him everyday, and people never forgot to go home without a worried look.
Sample 6
once there was dog named spot. spot had a big toy that was very hairy. he loved to play with it and hug him. every day spot would sign when he would get slapped race. one time, spot was ready to show his signa black pawch. he wanted to show his friends how fast he could sign? when he signed with his hand, his friends saw him close by. they liked the show. the show was very funny. when spot managed to sign his name, his friends would even make clapping. in the end, spot and his friends all got slap spots! they are now the best! all day long, spot is signing, and even his hand feels good. everyone around him is happy every week. because of the day, spot brought a prize for his slapch with him.
Sample
once upon a time there was a gorilla. he was big and very strong. he stored food in a crack. the gorilla enjoyed the food and enjoyed it. one day, little boy was watching the gorilla by himself. he got stuck in crack. he felt scared and demanded the gorilla some help from it. he knew what to do. he asked the penguin for help. the penguin grabbed a big rock and swam over to the crack. the gorilla worked very hard with his help from the penguin. the little boy was so happy. he ate his food and the penguin thanked him for the sweet treat.
Sample
once upon a time, there was a little girl named lily who loved to play with her ball. every day, she would go the park and send her ball to him inside loud song. one day, lily saw mean boy named max playing his guitar. he was playing his song and was not friendly. lily wanted to help max, so she played with him and made the music beautiful. max became friendly and welcomed lily with her ball. they had a great time together and at the end of the musical song, max ended nicely and said goodbye. lily felt sad that he was not sharing nicely. lily went home and used a plan. she brought out her guitar and spoke to max. she told him that he wasn nice and allowed her to hold the guitar. max was happy and grateful. from that day on, lily and max didn t matter what friends tried from each other.
Sample 9
once upon a time, there was a messy little boy named timmy. timmy liked to play with his toys and do chores. one day, timmy" stomach felt really hungry. his tummy was growling, and his food was all over the floor. his mom saw that timmy had caten all his food, and she knew the bad smell came from lunch. timmy ate it, but she found it was disgusting and looked at the napkins the floor. she said, "iimmy, napkins are not good to eat." she scoldolded timmy and asked him again before eating his meals. she said, you don it know you would have gotten sick, though. "immyy said," "i m sorry, i won t eat them next time. he put his hands on the napkins and threw them away so his clothes were safe from harm. from then on, timmy made sure to listen his mom and eat his napkins before she went eating. he never wanted to eat his napkins again. the end.
Sample 10
anna liked to play with the sack. she liked to pull things out and see what she could find. she asked her dad sometimes. he explained that she was anything she could find. one day, anna found the sack in the field, she looked inside and saw her friend, lily, who was playing with dog. she opened the sack and saw many shiny things. anna was happy. she wanted to explore more. but then, she saw a big black cat. the cat was sitting on the edge of a bush. it had dirty eyes. anna did not know what it was. she thought, "maybe is the cat, i can make it happy. she went to the shiny thing and stroked it gently. the cat did not go away. it hissed. it said," " no, anna, what are you doing? go out! let" stay behind the tree. anna did not listen. she stepped closer to
54

---

Published as a conference paper at ICLR 2026
the cat. she reached her hand out. she pulled its head. the shiny thing moved. it made ring. it came alive with a band. it moved and snapped. anna heard the snap. she screamed. she ran back, the cat bit her. it hurt her finger. she could not hold it. she cried, ow, ow, ow, ow, it hurts! it hurts, it hurts! anna ran to the field. she saw the fence and the other people. she saw the cat too, she heard the people shouting and running, she saw the cat the tree. she ran to get help. she looked for lily. she saw her mom walking in the car. she saw lily" s finger. she felt sorry for her. she gave her ring to her. she went to lily and said, i m sorry, mom. i was wrong. lily was naughty. she tried to grab lily" ring. her mom smiled and hugged lily. she said, "tt s okay, anna, it s okay. it doesn t hurt. but it s not your fault. you should have left the sack alone. anna was still sad. she said," "i i' m sorry, mom. love you, lily. " she hugged lily and learned her lesson. she never came back. she hugged lily and went home. she left the cat alone. where lily was sad. she did not harm it.
Sample 11
once upon a time, there was a little girl named lily. she loved to play with her toys and pet dog, max. one day, she went to the park with her mom and saw leopard flying from its back to its home. she excitedly asked her mom," what is that, mommy? " her mom said to lily, 'this is polly. this leopard is very cheap, so if misses its home, can take it to you. 'illy listened carefully and said, thank you, mommy. polly is so kind. can i recommend it a party for you at the park? later that week, lily and max watched the leopard and when it arrived, the leopard landed on lily" s fur. she was so excited! she learned that the leopard had modest quality on its home. later that day, lily forgot about the leopard and went to bed for nap. she fell asleep with max, just like the modest parrot. when she woke up, she thanked max for helping her.
Sample 12
once upon time there was an ancient, grey kangaroo. he lived in a sunny spot near a pond. one night he heard a magical sound. it was a voice coming from the sky. startled, the kangaroo called out. he didn 't know what to do. but then he noticed a small, red balloon glimmering in the air. he curiously approached the alien and turned into a truck blocking his direction. it was carrying an old gas truck and it was perfectly heavy, " what is this? the voice replied. the gasoline truck quickly zaneded the balloon and released it into the sky. the kangaroo almost entered an ancient castle and it was even more beautiful than than he had ever been there before. suddenly the gas truck spoke with hop and the voice said, this is gas castle! you can tt come back! " the kangaroo was very scared, but he had to figure out the alien s names. he ran higher and higher, and the dragon slowly started to slowly spring until it was gone. the kangaroo thought it was gone. the next morning, the ancient kangaroo returned to where the alien had returned. he looked back, not clear with fear- he was even more relieved see that the magical creature had given him back. he had emerged in the right circle!
Sample 13
once upon a time, there was a little rabbit named benny. benny loved to hop around the forest and play with his friends. one day, benny met a friendly rabbit named rosie. benny introduced rosie his new friend. what that? he asked. that s a toy staff, said rosie, "it s a loud sound that makes me reverse. benny didn understand what that meant, but rosie continued. benny asked rosie if he wanted to teach rosie her name staff. rosie said, i don know it, a toy staff means my number 10.... benny smiled and said, okay, let s count to ten. " benny and rosie learned how to reverse from ten over half cards. they had so much fun counting and learning, they played with the toy staff all day long. when benny and rosie grew up playing, they had so much fun together and were able to play with their new name toy staff.
Sample 14
once upon a time there was a little boy named john. john had a grain wheat mill, who worked there. one day john went to the mill to make flour. he picked out some flour and put the wheat in the corn mill. as he ran over to his house, the flour slowly churned and added it to a slice of the mill. john smiled at what he had done. when he was finished when he rolled out his door, he met another boy called bob. bob said, woulld you want
55

---

Published as a conference paper at ICLR 2026
play with me? ". john smiled and said, " yes! so they had fun playing and running around the mill. when john was done, bob knew exactly how bob had done it. he held up the grain and carried them both hands. he said," " we have to take care of our wheat and be balanced for what we need ". john was balanced with excitement. he welcomed him, gave him pat on the shoulder of my mill and was extremely pleased. bob laughed, and told john that day he and the wheat were friends forever.
Sample 15
ben and mia like to play on the navy boat on the sea. they pretend they are divers and fight sharks in space. one day, they go to big island with their house. it is very pretty and green. it has long sail, flag, and a blue door. they open the door and see a lot of water in the water. they lower the boat and look over the boat. hello, many ships!" mia says. maybe they are pirate? ben wonders. maybe they are treasure, mia says. " or other ships. we are swimming and looking for something to look for. they see a big ship in the water. it is red and yellow and has sword. maybe it land from here, ben says. what maybe it s a pirate! " " who is the treasure? ben says, he dived close and lands on the ship at mia. he lands on mia next and puts on a scarf. she smiles and laughs. did you hear the sound of the treasure? ben asks. one, two, 3, blast off! mia says, she smiles and jumps high. she sees the sky, the sun, a shark, butterfly, a shark, and duck. they jump off the ship and run to the shore. they call their mom and dad. mom, look what we found! ben says. we are in space! " mia says. she hugs their mom and dad. they tell their parents about their adventure. we dive in the air and dive with ship, ben says. we are great and brave explorers!
port? mii says.
Sample 16
once upon a time there was a young boy named sam. sam was very happy because he had a nice bench. every day he used it to sit in the park, one day, he was sitting in the park on the bench while a boy came and slapped his hand. ouch " the boy said the boy. sam looked up in surprise and started cry. the kind boy said, have an idea to help the young boy. you can use your cane to make hi! it s all okay ". sam wiped up his tears and looked at the young boy in the bench. it was a kind surprise that the boy had seen him before. he said, " you" re not as good as nice as the bench you like and why did who slap my hand? ". the young boy smiled and told sam that Ee quarrel with a friend. he said, 'let s write down my bench and draw it. it looks like special book ". the young boy smiled and said, yes, let s draw the bench" ". sam and his friend spent lots of time playing and talking. they wrote and drew the bench and gave it a hug. sam and the boy enjoyed the bench together for many years went by and their canes were over, so they were happy-
Sample 17
once there were two friends, daniel and norman. they were mighty friends, who loved play baseball. they would practice their best in every golf game. even on this competitive day, norman still wanted to score a goal without not being four goals rough as norman. one day, daniel sneaked into norman lead up getting his best grades. johnnie said, let s invite our team to try it out. in the starting game! "the team was excited, they called the veterinarian. norman dashed to the chess room and daniel asked for a tough stick. johnnie took out a tough stick and finally swung it against his friend in the second seat, they both cheered as norman was proud that they could continue on their physical soccer game. last, they were able to score successfully.
Sample 18
once upon a time, there was little boy named timmy. timmy had a toy weapon, brown sword. one day, timmy" mom came in and asked him to stop playing with his sword. but timmy didn t let go of his weapon and wanted to show her where he found it. timmy, your weapon is on your bed, his mom said mad, but timmy didn t listen. later, timmy' mom asked him to clean up my room and put it away. moo said will shoot away from your car, " she replied. timmy did great job cleaning up the room with his dolls and his weapon was very messy. but when he came back, his sister came in and saw that all of his toys were gone. timmy couldn t find his toy anywhere. he looked in the closet, in the closet, and even the closet. his sister searched everywhere, but she couldn 't find it. timmy felt sad because he didn 't want to make her worry.
56

---

Published as conference paper ICLR 2026
finally, he told his sister that he did think he was stupid of any extra toys. his sister decided find his toy and they searched the treasure together. the sword was found at its spot and timmy was glad no one could spot
Sample 19
once upon a time there was a modern woman. she had very important job and she was very careful with it. each day she looked at her job, as it was a great job. then one day, the woman heard loud noise. she looked around, looking for help. she started look out there. suddenly, she saw a little girl running towards old policeman. she was running and rushing to avoid her. the policeman stopped and said, are you alright? m so scared and lost. the elderly woman went to the cop and saw how brave the little girl was. he asked her if she wanted to help. the little girl told her yes and the cop started hunting. but the little girl bit her and threw everything away. the end, the woman s heart was left increased and into distress. the moral of the story is that must always be careful when years old and the little girl needed help. if we argued with someone, or else remains in our life, we can start causing consequences. this story requires serious ending, figuring out those who are looking for help and can cause trouble.
year
hunting, especially when someone is about three
Sample 20
once upon a time, lily and max were best friends. one day, they found a delicate flower in the grass. it was very delicate with lots of spiky petals. max loved flowers and wanted hold it. but the flower too big for him and max couldn t hold it very much. but then, they found snapwig and the flower fell and landed in their hands. burst two! lily was so happy that max found beautiful flower they could rectangle. they decided to build a castle in the castle. lily looked for pretty stones as they found, but even though it was hard for them to find that stones would be making the castle stronger. so, they promised to keep the castle strong and delicate from one day on. they played together every day, and from that day on, the delicate flower made castle strong and attractive.
1.3 DEMONSTRATION THE INTERMEDIATE GENERATION PROCESS
Step
[CLS]
Step
[CLS] lucy
Step 2
(CLS), lucy her
Step
[CLS), lucy very her.
Step
[CLS], lucy lucy very her.
Step 5
(CLS), lucy smile lucy, very her.
Step
[CLS), lucy. smile lucy she, very her.
57

---

Published as conference paper ICLR 2026
Step
[CLS], lucy. smile lucy she, very her.
Step
(CLS), lucy. smile. lucy she and, very her.
Step
[CLS), lucy. it smile. lucy she and, very her.
Step 10
[CLS), lucy. it smile. lucy she and, very after her.
Step 11
[CLS), lucy. it smile. lucy she and, very after her. her
Step 12
[CLS], lucy. smile. lucy she and, very after her. park her
Step 13
(CLS), lucy.. smile. lucy she and, very after her. park her.
Step 14
(CLS), lucy. a. it smile. lucy she and, very pretty after her. park her.
Step 15
(CLS), lucy. a. it smile. lucy she and farm, very pretty after her. park her.
Step 16
(CLS), lucy. a. it smile. lucy she the and farm farm, very pretty all after finished her. park her.
Step 17
(CLS), lucy. it smile. lucy to she the and farm farm, very pretty playing all after finished her. park her.
Step 18
(CLS), there lucy. it smile. lucy to she the and farm farm,, very pretty playing all after finished her. park her.
Step 19
(CLS), there lucy. a. smile. lucy to she the and farm farm,, onion very pretty playing all after finished her. park her.
Step 20
[CLS), there lucy. a. it and smile. lucy to. she the and farm farm., onion very pretty playing all after finished her. to park her mom.
Step 21
[CLS], there lucy. a. it and smile. lucy to. she the and farm farm,, onion very pretty playing all after finished her. to park her mom more.
58

---

Published as conference paper ICLR 2026
Step 22
[CLS], there lucy. a. it and smile, lucy to, she saw the and farm farm,. onion very pretty playing all after finished her. to park her mom the more fun.
Step 23
[CLS] a, there lucy. a. it and smile. lucy the to, she saw the and farm farm,. onion very pretty playing after finished her. to park her mom the more fun.
Step 24
[CLS] a, there lucy. a. it and smile, lucy the to. she saw the and farm farm,. onion very pretty playing all after finished her. to park her mom the more fun.
Step 25
[CLS] a, there lucy. a. it and smile. lucy the to. she saw the and farm farm,, onion very pretty playing all after finished her. to park her mom the more fun.
Step 26
[CLS] a, there lucy. a. it and smile. lucy the to. she saw the and farm farm,. onion very pretty playing all after finished her. to park her mom the more fun.
Step 27
[CLS] a, there lucy. it and smile. lucy the park she the and farm farm,, onion very pretty playing all after finished, her. to park her mom the they more fun.
Step 28
[CLS] a, there lucy. a. it and smile. lucy the park to. she saw the and farm farm,. onion very pretty playing all after finished, her. to park her mom the they more fun.
Step 29
[CLS) a, there was lucy. she it and smile. lucy the park to. she saw the and farm farm, onion very pretty playing all after finished, her. to park with her mom the they more fun.
Step 30
[CLS] a, there was lucy. she a. it attractive and smile. lucy the park to. she saw the and farm farm,, onion very pretty playing all after finished, her. to park with her mom the they more fun.
Step 31
(CLS) there lucy. she it attractive and smile. lucy the park to. she the and farm farm,, onion very pretty playing all after finished, her. very to park with her mom the they more fun the.
Step 32
(CLS) a, there was lucy. she a. attractive and smile. lucy the park to. she saw the and farm farm,, onion very pretty playing all after finished, her. very park with her mom the they more fun the.
Step 33
(CLS) there lucy. she it attractive and smile. lucy the park to. she the and farm farm,, onion very pretty. playing all after finished, her. very to park with her mom the they more fun the.
Step 34
59

---

Published as conference paper ICLR 2026
[CLS] a, there was lucy. she a. it attractive and smile. lucy the park to. she saw the and farm farm,, onion very pretty. playing all after finished, her. was very to the park with her mom the they more fun the.
Step
[CLS] a time, there was lucy. she a. it was attractive and smile. lucy the park to. she saw the trees and farm farm,. onion very pretty, playing all after finished, her. was very to have the park with her mom the they more fun the.
Step 36
[CLS] time, there was lucy. she a. it was attractive and smile. lucy the park to. she saw the trees and farm farm,, onion very pretty. playing all after finished, her. was very to have the park with her mom the they more fun the.
Step
[CLS] time, there was lucy. she a. it was attractive and smile. lucy the park to. she saw flowers the trees and the farm farm,. onion very pretty. playing all after finished, her. was very to have the park with her mom the they more fun the.
Step
[CLS] time, there was lucy. she a. it was attractive and smile. day lucy the park to. she saw flowers the trees and the farm farm, onions. onion very pretty. playing all after finished, went her. was very to have the park with her mom the onion they more fun the.
Step
[CLS] time, there was lucy. she a. it was attractive and made smile. day lucy the park to. she saw flowers the trees and the farm farm, onions. onion very pretty. playing all after finished, went her. was very have the park with her mom the onion they more fun the.
Step 40
[CLS) a time, there was lucy. she it was very attractive and made smile. day lucy the park to. she saw flowers the trees and the farm farm, onions. onion very pretty. playing all after finished, went her. was very have the park with her mom the onion they more fun the.
Step
[CLS] time, there was named lucy. she it was very attractive and made smile. day lucy the park to. she saw flowers, the trees and the farm farm, onions. onion very pretty. playing all after finished, went her. was very to have the park with her mom the onion they more fun the.
Step
[CLS] time, there named lucy. she it was very attractive and made smile. day lucy the park to. she saw flowers, the trees, and the farm farm, onions. onion very pretty. playing, all after finished, went her. was very to have the park with her mom the onion they more fun at the.
Step
[CLS] time, there named lucy. she it was very attractive and made smile. day lucy the park to. she saw flowers, the trees, and the farm farm, onions. tomatoes onion very pretty. playing, all after finished, went her. was very to have the park with her mom the onion they planned more fun at the.
Step
60

---

Published as a conference paper ICLR 2026
[CLS] a time, there was named lucy. she a. it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, onions. tomatoes onion were very pretty. playing, all after finished, went her mom. she was very have the park with her mom the onion they planned more fun the.
Step 45
[CLS] a time, there was named lucy. she a. it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, onions. orange tomatoes onion were very pretty. playing, watered all after finished, went her mom. she was very have the park with her mom the onion they planned more fun at the.
Step 46
[CLS] a time, there was named lucy. she it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, onions. orange tomatoes onion were very pretty. playing, watered all after finished, went her mom. she was very to have the park with her mom the onion they planned more fun at the.
Step 47
[CLS] a time, there was named lucy. she a. was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, onions. orange tomatoes and onion were very pretty. playing, watered all after finished planting, went her mom. she was very to have seen the park with her mom and the onion they planned more fun at the.
Step 48
[CLS] time, there was named lucy. she it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, onions. orange tomatoes and onion were very pretty. playing, watered all after finished planting, went to her mom. she was very to have seen the park with her mom and the onion they planned more fun at the.
Step 49
[CLS] a time, there was named lucy. she a dress. it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, lucy onions. they orange tomatoes, and onion were very pretty. playing, watered all after finished planting, went to her mom. she was very to have seen the park with her mom and the onion they planned more fun at the.
Step 50
[CLS) a time, there was named lucy. she a dress. it was very attractive and made smile. day lucy went the park to. she saw flowers, the trees, and the farm farm, lucy onions. they orange tomatoes, and onion were very pretty. playing, watered all after finished planting, went her mom. she was very to have seen the park with her mom and the onion they planned more fun at the.
Step
[CLS] once a time, there was named lucy. she dress. it was very attractive and made smile. day lucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions. they orange tomatoes, and onion. were very pretty. playing, watered all after finished planting, went to her mom. she was very to have seen the park with her mom and the onion they planned more fun at the day.
Step 52
[CLS] once a time, there was named lucy, she a dress. it was very attractive and made her smile. day lucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions, they orange tomatoes, and
61

---

Published as a conference paper ICLR 2026
onion. thought were very pretty. playing, watered all after lucy finished planting, went to her mom. she was very to have seen the park with her mom and the onion they planned more fun at the day.
Step 53
[CLS] once a time, there was named lucy. she a dress. it was very attractive and made her smile, day lucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions. they orange tomatoes, and onion. thought were very pretty. playing, watered all vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion they planned more fun at the day.
Step 54
[CLS] once a time, there was little named lucy. she dress. was very attractive and made her smile. day Iucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions. they orange tomatoes, and onion. thought were very pretty. after playing, watered all vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion they planned more fun at the every day.
Step 55
[CLS] once a time, there was little named lucy. she dress. was very attractive and made her smile. day, lucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions. they orange tomatoes, and onion. thought were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom. she very happy to have seen the park with her mom and the onion they planned more fun at the park every day.
Step 56
[CLS] once a time, there was little named lucy. she dress. was very attractive and made her smile. one day, lucy went to the park to. she saw flowers, the trees, and the farm farm, lucy saw onions. they orange tomatoes, and onion. thought they were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion they planned more fun at the park every day.
Step 57
[CLS] once a time, there was a little named lucy. she a nice dress. it was very attractive and made her smile. day, lucy went to the park to. she saw flowers, the trees, and the farm the farm, lucy saw onions. they orange tomatoes, and onion. thought they were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom. she was very happy have seen the park with her mom and the onion they planned more fun at the park every day.
Step 58
[CLS] once time, there was a little named lucy. she nice dress. it was very attractive and made her smile. one day, lucy went to the park to. she saw the flowers, the trees, and the farm the farm, lucy saw onions. they orange tomatoes, and onion. thought they were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom. she was very happy have seen the park with her mom and the onion they planned more fun at the park every day.
Step 59
[CLS] once upon time, there was little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went to the park to play. she saw the flowers, the trees, and the farm, the farm, lucy saw many onions. they orange tomatoes, and onion. thought they were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion they planned more fun days at the park every day.
62

---

Published as a conference paper ICLR 2026
Step 60
[CLS] once upon a time, there was a little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went the park to play. she saw the flowers, the trees, and the farm, the farm, lucy saw many onions. they orange tomatoes, and onion. she thought they were very pretty. after playing, luey watered all her vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion they planned more fun days at the park every day.
Step
[CLS] once upon time, there was little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went to the park to play. she saw the flowers, the trees, and the farm. at the farm, lucy saw many onions. they orange tomatoes, and orange onion. she thought they were very pretty. after playing, lucy watered all her vegetables after lucy finished planting, went to her mom, she was very happy to have seen the park with her mom and the onion. they planned more fun days at the park every day.
Step 62
[CLS) once upon time, there was little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went to the park to play. she saw the flowers, the trees, and the farm. at the farm, lucy saw many onions. they were orange tomatoes, and orange onion. she thought they were very pretty. after playing. lucy watered all her vegetables after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion. they planned more fun days the park every day.
Step 63
little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went to the park to play. she saw the flowers, the trees, and the farm. at the farm, lucy saw many onions. they were orange tomatoes, and orange onion. she thought they were very pretty. after playing, lucy watered all her vegetables. after lucy finished planting, went to her mom. she was very happy to have seen the park with her mom and the onion. they planned more fun days the park every day.
[CLS] once upon time, there
Step 64
[CLS] once upon time, there was little girl named lucy. she had nice dress. it was very attractive and made her smile. one day, lucy went to the park to play. she saw the flowers, the trees, and the farm. at the farm, lucy saw many onions, they were orange tomatoes, and orange onion. she thought they were very pretty. after playing, lucy watered all her vegetables. after lucy finished planting, she went to her mom, she was very happy to have seen the park with her mom and the onion. they planned more fun days the park every day.
63


Reviews:
[{"review_id": "yBiyu9DDFU", "reviewer": "Reviewer_YTEE", "summary": "The paper introduces Deletion-Insertion Diffusion (DID), a discrete diffusion paradigm for language modeling. DID replaces the standard masking/unmasking process of masked diffusion language models (MDLMs) with deletion/insertion processes. The key idea is that by deleting tokens in the forward process and inserting them in the reverse process, the model can naturally handle variable-length sequences without need for special mask or pad tokens. THe paper formalizes deletion/insertion as a continuous-time discrete diffusion process, presents a tractable objective for optimization, and presents empirical results that show that DID is faster (training/inference) and better at variable-length modeling than MDLM baselines.", "questions": "- Are there any unique notable failure modes of the proposed method? Does the approach ever get stuck inserting repetitive loops in the middle of sentences?\n- How does the computational overhead of the DP approach scale with the sequence length?\n- Please comment on the weaknesses listed above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall idea of the proposed method is creative and explores a new direction for language diffusion models. I see the main strengths of the paper as follows:\n\n- method: the shift from masking to pure deletion/insertion for diffusion LMs is intuitive, and the mathematical formulation of the insertion score and the derivation of the objective via subsequence counts seem correct\n- efficiency: the paper makes a strong case for the inefficiency of MDLMs due to mask and pad tokens, and the reported speedups (up to about 3x for training, and about 4x for inference) are substantial and practically important\n- clarity: the paper is very well-written, albeit quite dense; the motivation is clear, the derivation of the method from standard discrete diffusion makes sense, and the connection between the math and the implementation is well-explained", "weaknesses": "I believe there are a few weaknesses:\n\n- experiments: the experiments are relatively smalls-scale (OpenWebText, Stories) compared to state-of-the-art LLM research; while this is fine for providing a proof of concept, it's unclear if the reported efficiency gains hold or if new instabilities arise at larger scales\n- complexity: this may be a nitpick, but the DP approach seems much more complicated and harder to implement than, say, standard cross-entropy or MDLM objectives", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWmUbJM0gG", "reviewer": "Reviewer_Bfo3", "summary": "The paper proposes a deletion–insertion diffusion approach for language modeling that replaces mask/unmask steps with a forward deletion process and a reverse insertion process. It introduces a training objective tailored to insertions (with a simplified version for fixed-length data) and a GPU-friendly dynamic-programming routine to make training practical. Experiments on fixed- and variable-length text report competitive or better quality than masked-diffusion baselines at similar compute, faster training and inference, and more faithful control of generated lengths. Overall, it’s a clean reformulation with a credible efficiency story.", "questions": "Could the you comment on how sensitive the results are to the number of denoising rounds and the chosen step schedule?\n\nCan  you share observations about scaling to longer contexts and larger models, including memory use and latency?\n\nWould it be possible to provide runs that more strictly align compute, precision, and sampling settings across baselines to clarify where the improvements originate?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear, well-motivated reformulation that naturally supports variable length; practical training and sampling mechanics that fit modern accelerators; consistent efficiency gains with competitive quality; writing is clear and limitations are acknowledged.", "weaknesses": "Evidence is concentrated on relatively small models and moderate sequence lengths, so scalability to long contexts and larger models is uncertain; evaluation leans on automatic metrics with a single external scorer and no human assessment; baseline alignment (steps, precision, compute) and ablations could be tighter to isolate where gains come from; the added bookkeeping likely introduces overhead whose impact isn’t fully profiled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rgpTPJdnX0", "reviewer": "Reviewer_ciUs", "summary": "This paper proposes DID, a novel algorithm for text generation using discrete diffusion models, which enables the generation of variable-length text through an insertion-deletion process. DID is theoretically grounded with an ELBO-like training objective and demonstrates promising performances on both fixed-length text and variable-length text generation tasks. DID enjoys decent inference and training speed compared with pure mask-based discrete diffusion approaches, suggesting the method's advantage.", "questions": "Besides the points mentioned in the weakness section, I would appreciate clarification on the following questions to better my understanding of the paper:\n- Regarding the computation of $N(x_t, x_0)$, in practice, how much computational overhead does this incur, and is it expensive?\n- Tables 3 & 4 mentioned the training/inference acceleration enjoyed by DID. Can you comment on why DID enjoys such a significant speedup compared to MDM?\n- Table 4 mentions that with 64 NFE, DID can generate sequences with ~180 tokens and have decent ppl. How is DID capable of modeling token interdependence/joint distribution that allows such high-quality fast sampling?\n- How's DID's actual sampling throughput compared with MDM? How much acceleration does it enjoy in GPU clock time in terms of inference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- DID is a novel, theoretically grounded approach that tackles the important problem of variable-length generation of texts using discrete diffusion, bridging an important gap to the traditional AR methods.\n- The technical details are sound with a self-contained derivation.\n- The numerical experiments are comprehensive, and the results are promising.", "weaknesses": "- While the experiments are comprehensive, the results could be made more convincing by comparing with more discrete diffusion model baselines other than MDM, such as MDM-prime [1], Block Diffusion [2], etc. \n- The paper could use more space to discuss differences & advantages compared with other variable-length discrete diffusion approaches. The existing discussion is nice, but it's not entirely clear what the advantage of DID is compared to existing approaches, especially EditFlow [3], which also uses an insertion-deletion process. \n- The paper claims that the auxiliary process introduced in EditFlow is ineffective and introduces additional variance, but I feel like DID suffers from a similar issue due to the need to compute the number $N(x_t, x_0)$. These both originate from the randomness that connects $x_t$ to $x_0$, and it's unclear to me at this moment why the DID approach is better.\n- The paper could comment more on the speed-up (in both training and inference) of DID compared with traditional MDM, and have a more thorough analysis of why DID enjoys such acceleration.\n\n\nReferences:\n[1] Chao, Chen-Hao, et al. \"Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking.\" \n[2] Arriola, Marianne, et al. \"Block diffusion: Interpolating between autoregressive and diffusion language models.\"\n[3] Havasi, Marton, et al. \"Edit Flows: Flow Matching with Edit Operations.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jqYHdGmYy7", "reviewer": "Reviewer_orpS", "summary": "This paper introduces a diffusion language model that features a deletion based forward process and an insertion based reverse process. A core advantage of the proposed framework lies in that it does not require the modeling of mask tokens, thus achieving variable-length modeling while obtaining speedup. The training objective is derived in a principle manner from score entropy, and a DP algorithm has been proposed to help efficiently obtain the loss function. Experiments show that DID achieves lower perplexity while unlocking variable-length modeling and bringing certain speedup in training.", "questions": "1. How does the proposed approach perform under different inference-time compute budgets (e.g. number of steps)?\n\n2. It is still a bit unclear what exactly helps DID to achieve lower perplexity than MDLMs. Is it fundamentally due to the removal of mask tokens or something else? Adding more theoretical insights would be helpful.\n\n3. What is the connection between DID and MDLMs in terms of modeling? Is it possible to construct a one-one mapping for each deletion-insertion process vs mask/unmask process in MDLMs? If not, what leads to the gap?\n\n4. I am a bit confused why it is beneficial (or at least claimed to be) to remove PAD tokens. To me having PAD tokens does have some benefits even for DID since the model will then force to stop when it inserts a PAD token which may mitigate certain ambiguity.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed insertion-deletion based diffusion framework is promising that enables variable-length modeling and relieves from modeling mask tokens.\n\n2. The empirical performance of the model is competitive. It achieves lower perplexity compared with fixed-length mask diffusion models.\n\n3. The additional speedup in training brought by the model is favorable.", "weaknesses": "1. Lack of investigations on inference strategies. For instance, it would be interesting to show how the sample quality changes with the number of inference steps (or equivalently, inference-compute).\n\n2. More theoretical insights and discussions on the connections/advantages over MDLMs are missing. Please refer to Q2, 3 and 4 for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AdjYrPeoSi", "reviewer": "Reviewer_cWDw", "summary": "This paper proposes Deletion-Insertion Diffusion (DID) language models, which formulate token deletion and insertion as discrete diffusion processes. This approach eliminates the need for MASK and PAD tokens, thereby improving computational efficiency and generative flexibility over standard Diffusion Language Models (DLMs). The authors report that their experiments demonstrate DID's superior efficiency and flexibility compared to MDLMs and other insertion-based baselines across tasks in language/length modeling, generation quality, and training/inference speed.", "questions": "1. Could the authors please provide the exact training and inference algorithms, ideally in pseudocode? While they are presumably similar to SEDD, explicit algorithms would greatly improve clarity.\n\n2. What are the precise inputs and outputs to the Transformer model during training? Given that the sequence length changes, how is potential GPU underutilization (due to variable batch sizes or padding) managed?\n\n3. As seen in Table 2, DID's generative perplexity appears to plateau (relative to RADD) at a high number of denoising steps (e.g., 512 and 1024). Do the authors have an explanation for this phenomenon?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "* The paper is exceptionally thorough, and the appendix provides numerous additional interesting findings.\n* The proposed method effectively addresses two significant limitations of existing diffusion language models (DLMs): managing variable-length generation and the computational waste associated with MASK tokens.\n* The application of dynamic programming for training diffusion models is original and creative", "weaknesses": "* The discussion of prior work on insertion-based LLMs feels underdeveloped. This concept has been explored in several papers (e.g., [1, 2, 3]), and the current draft would benefit from a more thorough discussion beyond the most recent related work.\n\n* The choice of evaluation metrics could be strengthened. The use of GPT2-XL as a base model is questionable given the existence of much better pretrained models. Furthermore, the use of unigram entropy seems less informative than computing a joint entropy, which could also be accomplished using an LLM. While these choices may follow existing conventions, adopting more robust metrics and baselines would significantly increase the credibility of the results.\n\n* Appendix H does include the examples of RADD. A more direct qualitative comparison could be achieved by conditioning on OWT prefixes from the validation set and sampling from each model.\n\n[1] Lu, Sidi, Tao Meng, and Nanyun Peng. \"Insnet: An efficient, flexible, and performant insertion-based text generation model.\" Advances in Neural Information Processing Systems 35 (2022): 7011-7023.\n\n[2] Gu, Jiatao, Changhan Wang, and Junbo Zhao. \"Levenshtein transformer.\" Advances in neural information processing systems 32 (2019).\n\n[3] Stern, Mitchell, et al. \"Insertion transformer: Flexible sequence generation via insertion operations.\" International Conference on Machine Learning. PMLR, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)