Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
Authors: Mingyu Jo
Jaesik Yoon
Justin Deschenaux
Caglar Gulcehre
Sungjin Ahn
Venue: ICLR 2026 Poster (2026)
Keywords: Discrete Diffusion
Generative Models
Deterministic Latent Pathway
Self-Conditioning
Primary area: generative models
TL;DR: We propose Loopholing Discrete Diffusion Models (LDDMs), adding a deterministic path that keeps “soft” token information across steps, improving non-autoregressive text generation with more stable, coherent, and stronger reasoning outputs.

Abstract: Discrete diffusion models offer a promising alternative to autoregressive generation through parallel decoding, but they suffer from a sampling wall: once categorical sampling occurs, rich distributional information collapses into one-hot vectors and cannot be propagated across steps, forcing subsequent steps to operate with limited information. To mitigate this problem, we introduce Loopholing, a novel and simple mechanism that preserves this information via a deterministic latent pathway, leading to Loopholing Discrete Diffusion Models (LDDMs). Trained efficiently with a self-conditioning strategy that avoids unrolling the full denoising trajectory, LDDMs achieve substantial gains—reducing generative perplexity by up to 61\% over prior baselines, thereby closing (and in some cases surpassing) the gap with autoregressive models, and producing more coherent text. Applied to reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such as Countdown and Game of 24. These results also indicate that loopholing mitigates idle steps and oscillations, providing a general and effective path toward high-quality non-autoregressive text generation.

Published as a conference paper at ICLR 2026
LOOPHOLING DISCRETE DIFFUSION: DETERMINISTIC BYPASS OF THE SAMPLING WALL
Mingyu Jo Jaesik Yoon 4, Justin Deschenaux", Caglar Gulcchre Sungjin Ahn 1.5+ KAIST, RPPFI, 3 Miirosoftt SSA,, NYU
ABSTRACT
Discrete diffusion models offer a promising alternative to autoregressive genera- tion through parallel decoding, but they suffer from sampling wall: once categor- ical sampling occurs, rich distributional information collapses into one-hot vectors and cannot be propagated across steps, forcing subsequent steps operate with limited information. To mitigate this problem, we introduce Loopholing, a novel and simple mechanism that preserves this information via a deterministic latent pathway, leading to Loopholing Discrete Diffusion Models (LDDMMs). Trained efficiently with self-conditioning strategy that avoids unrolling the full denois- ing trajectory, LDDMs achieve substantial gains-reducing generative perplexity by up to 61% over prior baselines, thereby closing (and in some cases surpassing) the gap with autoregressive models, and producing more coherent text. Applied to reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such as Countdown and Game of 24. These results also indicate that loopholing mitigates idle steps and oscillations, providing a general and effective path toward high-quality non-autoregrressvv text generation.
INTRODUCTION
Discrete diffusion models have recently emerged as promis- ing alternative to autoregressive models for tasks such as text generation (Austin al. [021; Campbell et al.
(6.6)
et al.
2023 Sahoo et al. 2024 Schiff et al. 2024 Zhao et al.
2022 Lou
Ceneraaivveee Perploity (c)
100
(5.08)
(3:00)
(5.4
664
6.50
(3.44)
(363))
(5:00
2024
au 2022 Gat al. 2024 Wang al. 2025a). Un-
(..5))
SEDD Absorb
80
MDLM
like autoregressive models, which generate tokens sequentially from left to right, discrete diffusion generates entire sequences in parallel through iterative refinement across multiple denois- ing steps. This parallel generation not only enables substantial speedups for long-horizon sequence generation but also allows the model leverage global sequence context rather than be- ing restricted to left-context only.
(ours)
AR
(1
(557)
20
12n
200
512
1024
2008
The Maximum Number Sampling Steps
Figure 1: Unconditional gen PPI. measured with GPT-2 Large (sentence entropy in parentheses).
However, despite these advantages, empirical studies show that discrete diffusion models still lag behind autoregressive
models in generation quality (Gat et al., 2024 Zheng et al., 2024), indicating a substantial room for improvement. For example. recent studies have highlighted specific issues in discrete diffusion models, such as idle steps (Chao et al. (025) and temporal oscillation (Wang et al. (0255)).
As part of such effort to improve discrete diffusion models, in this work we first focus fun- damental phenomenon that may underlie these inefficiencies, which we term the sampling wall. We define the sampling wall as form of information collapse, in which rich categorical distribu- cions-capturing plausible token candidates and their relative likelihoods-are reduced to one-hot vectors after sampling. We call this a "wall" because, in standard discrete diffusion models, once sampling occurs, the original distributional information is lost and cannot be propagated to sub- sequent steps. Motivated by this observation, our central hypothesis is that explicitly propagating distributional information beyond the sampling wall across denoising steps can alleviate key limita- tions of discrete diffusion models.
"Correspnndence Mingyu Jo imingyu.jo@kaist.a.kkk and Sungjin Ahn (sungjin.ahn ahn kaist.ac.kr).

---

Published as a conference paper at ICLR 2026
In this paper, to realize this idea, propose simple and novel mechanism, termed Loopholing, together with a corresponding family of models, Loopholing Discrete Diffusion Models (LDDMs). Our key idea in the Loopholing mechanism is introduce a direct, deterministic pathway that trans- fers the rich contextual latent state to the subsequent step. This pathway complements the existing stochastic path; thus, in Loopholing, each denoising step produces vector and deterministic continuous vector. While this design introduces a recurrent dependency across the denoising trajectory, which would require full unrolling for training, we make training feasible at randomly sampled time steps without unrolling by introducing self-conditionin ap- proach (Chen et al. 2022] Jabri et al., (022) tailored for Loopholing.
outputs: stochastic one-hot
The main contributions of the paper are as follows. (i) Identifying the Sampling Wall Problem:
We identify the sampling wall problem as a fundamental characteristic that may underlie various inefficiencies in the standard discrete diffusion models. (ii) Introducing Loopholing: We propose
the Loopholing mechanism, and Loopholing Discrete Diffusion Models (LDDMss) (iii) Strong Empirical Results: We demonstrate the effectiveness of LDDMs through various experiments. On the OpeOOppeeeeeeeee dataset (Gokaslan & Cohen 2019), our model improves the MDLM (Sahoo et al. 2024 test perplexity from 23.82 to 21.90. In terms of Generation Perplexity (Gen PPL). as shown in Fig. our method achieves substantial gains, reducing Gen PPL by 55% relative to MDLM and by 61% % relative UDLM. Against autoregressive models, the gap shrinks from 3.17x higher Gen PPL. with MDLM to only 1.43x with our method. Remarkably, when applied to UDLM, our approach not only closes the gap but even surpasses the autoregressive baseline, whereas the standard UDLM lags behind by 2.15x. In reasoning, loopholing mechanism boosts the accuracy on Countdown4 (Gandhi et al. 2024 from 45% to 56.3% over the MGDM baseline (Ye et al. 2024).
2 PRELIMINARIES
Discrete Diffussin Models. Diffusion models (Sohl- et al. 2015 Ho et al. 2020) are probabilistic generative models defined by a fixed forward process that progressively corrupts data with noise, and learned reverse process trained to recover the original sample x. For discrete data, this framework is adapted by representing a categorical data sample as a one-hot vector x E V, where V = {u 10, K E. Uk 1} is the set of all one-hot vectors over vocabulary of size K. For clarity, we describe the formulation in the simplest case of a single categorical variable.
The forward process corrupts the initial data sample x over continuous time variable t E [0, 1), producing a sequence of progressively noisier latent variables Zz- A common approach for this is to use an interpolating discrete diffusion model (Sahoo et al., [0224] Schhff et al., 2024; Shi et al.j 2024 von Rütte et al., 2025), where the marginal distribution of the latent state 2r, conditioned on the original data x, formulated as a categorical distribution that interpolates between the data x and fixed prior distribution π.:
2 |x) = Cat(2z; 0,x + - TT)
(1)
where Cat(: p) denotes a categorical distribution parameterized by probability simplex p. and ar E [0, is monotonically decreasing noise schedule with ou and a1 0.
Masked Diffusion Models (MDMs) are a subclass of discrete diffusion models in which the prior is set m, the one-hot vector corresponding
special [MASK] token (Sahoo al. 2024 Nie et al. 2025 Kim et al. 2025). Under this formulation, the forward process can be interpreted as the gradual masking of tokens over time.
In MDMs, the reverse process generates data by progressively denoising a sample, beginning from the prior distribution in which all tokens are masked. Formally, seeks approximate the true reverse posterior (2, x) for any 0 < a < < 1. In practice, this distribution is parameterized using a neural network xe (ze, t) trained to predict the original data x. In the following, we use X6, as a shorthand for X (Zz. t). whenever this does not cause confusion. The resulting approximation of the posterior takes the form:
(6z, (Zu),
2: + m,
q(z, X8,1)
(m 1-ai
(2)
Cat Zz;
2, m,
where 6x, denotes the Dirac delta function. This ensures unmasked tokens are preserved, while masked tokens are resampled according to the model's predictions and the given schedule.
2

---

Published as a conference paper at ICLR 2026
For training, MDMs optimize simplified Negative Evidence Lower Bound (NELBO). In tinuous time, this specific formulation provides a tighter objective than its discrete- counter-
parts (Austin et al. 2021 Kingma et al. 2021 In practice, the objective reduces to weighted
cross-entropy loss (Sahoo et al. 2024 Shi et al. 2024):
I[z = ml of
LELLO = [0.- 20
lxg (z,, t).
(3)
1-a,
where (0 1] denotes the uniform distribution, af is the time derivative of cr,, (---) represents the dot product, and I[-] is the indicator function, returning the condition holds and 0 otherwise. This objective encourages the model to accurately reconstruct the original tokens at masked positions.
Besides MDMs, there are also Uniform Diffusion Models (UDMs), which use a uniform distribution over the entire vocabulary as the prior (Austin al. 2021 Schiff et al. 2024 Sahoo al. 2025
Further details on UDMs are provided in Appendix B
3 LOOPHOLING DISCRETE DIFFUSION MODELS
To begin with, we first discuss characteristic of discrete diffusion models that motivated the design of the Loopholing mechanism. We refer to it as the sampling wall problem.
The sampling wall problem represents form of information collapse, where rich categorical distri- butional representations are reduced to one-hot vectors. Specifically, while Xº.: = X5 (Z., t) encodes far richer information about plausible token candidates and their relative likelihoods, this informa- tion discarded once single category 2g is drawn, leaving only the one-hot representation to be propagated forward. For example, consider two cases: X [0.49, 0.51] and x9. [0.20, 0.80). Despite reflecting very different situations, the denoising process cannot distinguish between the two if the second category is sampled in both cases, entirely discarding the predictive distribution that position. Without propagating this distributional information, a process based solely on sampling can become redundant (Chao et al., 2025) and prone to excessive oscillations (Wang et al. 2025b).
To address the sampling wall problem in discrete diffusion models, we propose novel mechanism, termed Loopholing, together with a corresponding family of models, Loopholing Discrete Diffusion Models (LDDMs). The goal of LDDMs is to operationalize the following central hypothesis in both the architecture and the training procedure of discrete diffusion models:
Main Hypothesis: Enriching the denoising process by propagating detailed contextual in- formation available prior to sampling discrete tokens-such as the categorical distribution parameter X9,1 alleviate the aforementioned issues and leads to improved performance.
Our key idea for implementing the above hypothesis in the loopholing mechanism is to introduce, alongside the standard sampling pathway, direct deterministic pathway that carries rich distribu- tional context information obtained before sampling across denoising steps.
In the following, we present Loopholing Discrete Diffusion Models (LDDMs) in two parts. First, we describe the generation process of LDDMs, detailing how distributional context is obtained and propagated throughout denoising. Second, we explain how LDDMs can be trained efficiently through self-conditioning approach. Notably, both the generation and training mechanisms of Loopholing are straightforward to implement, requiring only minor modifications to the standard discrete diffusion framework.
3.1 GENERATION WITH LOOPHOLING
In standard discrete diffusion models, shown in Fig. a(a), the modeling of the denoising process operates on sequence of tokens 2; (I.L.) 2 (1) 2, First, for each position in the sequence, it converts the one-hor hot input token z; into an embedding )z The embeddings of the sequence are passed through a backbone network such as a Transformer (Vaswani et al. (017)) layer to mix the sequence embeddings and produce a latent embedding !t) Finally, this latent embedding is fed into projection layer predict the distribution of the clean observation that position x (2)
(1:2) (z! (z((12) 1)-
3

---

Published as a conference paper at ICLR 2026
Xy (dp, ()
Xy (x.))
Einopholing (Na (gg .))))
Projection
Projection
Projection
he
Backaone
Baclibone
Backbone
Backboone
Bg (x.)
Ct )
Add
Eo(t,)
A
Add
Ep Ct.
Add
Emroeddiingl
Emoedoinng
Layerhior
Emteddinn
LayerNorm
Embawtiing
Layarhoar
Z
Zt
h,
Zr
0
Zt
sg(h")
Standard Discrete Diffusion
Generation with Loopholing
Training with Self-conditioned Loopholing
(a)
(b)
(c)
Figure 2: Architectural comparison of standard discrete diffusion models and the Loopholing Discrete Dif- fusion Models (LDDMss). (a) The standard architecture of discrete diffusion. (b) During inference, LDDMs propagate the continuous latent representation h, the subsequent step, creating deterministic pathway that preserves rich contextual information. (c) During training. LDDMs employ a ceff-conditioning strategy: first pass generates pseudo-context h which is then used to condition the second pass.
Crucially, while the backbone aggregates global context across the sequence, the sampling step applies the posterior in Eqn. Dindependennl to each token position. Therefore, for notational clarity, we describe the subsequent formulation in the simplest case of a single categorical variable z omitting the positional index f.
We observe that rich contextual information captured in both h2 and xz t) during this process. This information is rich because it accounts for complex relational interactions among tokens and represented as a high-dimensional continuous vector rather than a one-hot encoding. However, once the output representation 2, is sampled, this rich information collapses into one-hot vector. Therefore, the next denoising step, which takes only 2, as input, cannot exploit this information or build upon it, and must instead reconstruct much of it again from the limited one-hot representation.
From this observation, our key idea in the loopholing mechanism to introduce direct, determin- istic pathway that transfers the rich contextual latent state h, the subsequent step. This pathway complements the existing stochastic path; thus, in loopholing, each denoising step produces two outputs: stochastic one-hot vector and deterministic continuous vector:
(xo h t), h.) = Loophhling (z,, h t).
(4)
Formally, the denoising process with loopholing as shown in Fig. b described as follows. Let z; be the one-hot vector for token at step t. We initialize a latent state h1 = 0. At each denoising step & + 8, the model performs the following computations:
e, Ez().) + LN(h:), h, = t) xo(z,, b, softmax (b.))
(5)
where Eg is the token embedding function, fo is the backbone network, and ge is the output projec- tion layer. The previous latent embedding h, combines with the current token embedding Eg (z,) via Layer Normalization (LN) (Ba et al. (0166), creating deterministic contextual latent path. This prediction Xo(2, h t) is used to parameterize the reverse posterior (Eqn. 2) and sample 2,.
Note that while it is also possible pass the model's 's prediction X7(z,, h t). to the subsequent step, this distribution over the vocabulary typically has much higher dimensionality than h, in important applications such language modeling. For this reason, we pass h, in default architecture.
3.2 TRAINING WITH SELF- CONDITIONED LOOPHOLING
A key advantage of diffusion models is that training does not require time-consuming temporal unrolling. Instead, training can be performed on randomly sampled time steps, as q(z,)x) can be directly constructed for any arbitrary time step. However, the generation with loopholing requires propagating h, across denoising steps, which introduces a recurrent dependency. Maintaining this training efficiency within loopholing therefore significant challenge.
To address this, we introduce a self-conditioning approach (Chen et al., 2022 Jabri et al. 2022 to avoid unrolling the full generation path during training. The core idea, illustrated in Fig. c), is
4

---

Published as a conference paper at ICLR 2026
quert
mun
Predict
Sampling
man
Masking
man
Imain
man
MASK
who
MASK
MASK
HASK
interrupted
reading
reading
reading
reading
reading
reading
State
Denoising
Network
Distribution
Sampling Well
Steps without Progress
Excessive Oscillation
Generated
Sample
Dencising Process
Figure Illustration of the sampling wall in Masked Diffusion Models (MDMs), which induces two distinct failure modes. (I) Steps without Progress: Fixing on a single token can cause input sequence remain static across multiple denoising steps, leading to significant computational inefficiency. (2) Excessive Oscilla- tions: Sampling low- cow-probability token (e.g.. "loud"" can trigger excessive oscillations in subsequent steps.
simulate the context propagation process using two forward passes: for a given noisy input 2,, the model first computes a pseudo-context ho and then uses it in second, context-conditioned pass as input to make the final prediction. Specifically, the process for each training step is as follows:
1. First Pass (Pseudo-Context Generation): We perform loopholing denoising, but by setting the imput context state zero vector. This yields pseudo-context he and an initial prediction x (z. t) directly from the loopholing function:
(x Ee, 0, t, h²) fLoopholing (Ze, h, 0,t).
(6)
2. Second Pass (Context-Condition bedding h from the first pass as it is from the previous step during the generation process:
Prediction): The second pass takes the aseudo-latent em-
(z), he, f).t),))) h') fLoopholing (Zz, h, = )g
The stop-gradient operator, sgl-), ensures that gradients flow only through the second forward pass. This allows the model to learn how to consume its own representations as context without the prohibitive cost of backpropagating through time.
This training objective can then be expressed as:
ml of (log(x) sg(h"], .). t).x))
nopholing E1--1(001))
[[z,
(8)
where the expectation taken over a uniformly sampled time ~~ 11 1] and the corresponding noised input x, g(2|x)) Furthermore, following previous work (Jabri et al. (022), we employ self-conditioning with probability of p. Specifically, the model is trained on the self-conditioning loss with probability p. and the standard discrete diffusion loss (Eqn.3) otherwise. This approach encourages the contextual latent from the first pass to accurately capture the context of x while providing useful guidance for the second pass.
DISCUSSION: WHY LOOPHOLING WORKS
We hypothesize that the sampling wall manifests through two key inefficiencies in discrete diffusion. which are illustrated in Fig.
Steps without progress: As shown in recent study (Chao al. 2025 ) many denoising steps in standard discrete diffusion models reproduce the same samples as previous steps, resulting in idle steps (see Appendix (D.]. This is wasteful because it does not make any progress in the sequence. We hypothesize that loopholing provides way to address this issue by enabling continuous updates to the context latent h, even when the sample z, remains unchanged. Specifically, the deterministic recurrent state update in the h, space (Eqn. 4 ensures that contextual information evolves across steps, so that each iteration contributes to refining the output. As shown in our ablation study (Sec- tion 6.3), LDDMs indeed exhibit higher Temporal KL divergence during the early denoising phase, confirming that each step makes more meaningful progress along the denoising trajectory. This improved efficiency allows high-quality samples to be generated with fewer denoising steps.
5

---

Published as a conference paper at ICLR 2026
Table Comparison of test perplexities (4) of models trained for 1 million steps the One Billion Word (LMIB) and WeenWe (OWT) datasets. denotes retrained model.
LMIB
OWT
Masked Diffusion
SEDD Absorb (ouu ct al. 2023
S 28.39
24.01
MDLM (Shhoo et al.
2024
27.60
23,05
Uniform Diffusion
UDLM (Schiff al.
2024
331.1
25.51
Ours (LDDMs)
LDDM-M (ours)
25.95
21.90
LDDM-U (ours)
29.21
23.82
Excessive Oscillation: The sampling wall can induce oscillations during denoising. Although stan- dard discrete diffusion models consistently predict the same objective, Xa.t, each step, it discards the rich distributional information from the prior step's prediction. This forces the model to predict from scratch, relying only on the current, stochastically sampled sequence (Wang et al. 2025b). We hypothesize that the loopholing mechanism can lead to a more stable generation process. By providing deterministic information path, this mechanism enables the model to directly maintain contextual information about the target x throughout the denoising process. Indeed, our ablation (Section shows that LDDMs exhibit lower Temporal KL divergence in the later denoising phase and consistently lower Token-Predictts Entropy, indicating more stable generation.
5 RELATED WORKS
Discrete Diffusion for Language Modeling. The pioneering work (Austin al. 2021) introduced discrete denoising diffusion using transition matrices based on an absorbing "mask" state and uni- form noise. Subsequent models, including Score Entropy Discrete Diffusion (SEDD) (Lou al., 2023). Mask Diffusion Language Models (MDLM) (Sahoo et al.
2024). Uniform Diffusion Lan- guage Models (UDLM) (Shhiff et al. 2024) and Duo (Sahoo al. 2025), have further advanced this paradigm with improved training objectives, producing stronger language modeling performance. However, shared limitation is the reliance repeated categorical sampling across the sequence, which can degrade contextual coherence. Our work directly addresses this challenge.
Self- Self-Conditioning. Self-conditioning techniques improve consistency across generative steps by reusing previous model outputs or hidden states during training. For instance, Analog Bits (Chen) al. 2022) employs seff-conditioning to enhance sampling performance, while Recurrent Interface Networks (RINs) (Jabri et al.) (2022) use it to reduce the computational cost of training by avoiding backpropagation throughout generation trajectory. Inspired by these approaches, our loopholing mechanism integrates self-conditioning strategy to efficiently train the model to use the propagated latent embedding as internal memory.
EXPERIMENTS
In this section, we empirically validate the effectiveness of the proposed loopholing mechanism across various models and tasks. We demonstrate that by accumulating contextual information, our mechanism achieves superior perplexity and generation quality in language modeling, along with higher success rates on reasoning tasks. To further understand these improvements. we conduct a series of ablation studies that provide a detailed analysis of our method's key compo- nents. For reproducibility, we provide full experimental details, including datasets, hyperparameters, training procedures, and evaluation protocols, in Appendix
6.1 LANGUAGE MODELING
To demonstrate the efficacy of our loopholing mechanism, we first apply it to discrete diffusion language models. Specifically, integrate method into the Masked Diffusion Language Mod-

---

Published as a conference paper at ICLR 2026
Table Zero-shot perplexities (4) after million training steps OpenWebText. All reported perplexity values are upper bounds. denotes retrained model.
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
Masked Diffusion
SEDD Absorb
97.87
38.34
74.71
50.15 48.36
76.54 68.62
45.25 41.94
39.75
MDLM
86.33
36,30
66.73
37.52
Uniform Diffussin UDLM
77.28
38.48
81.41
51.68
76.81
46.18
41.19
Ours (LDDMs)
LDDM-M (ours)
85,80
33.27
69.53
44.22
62.55
39.74
34.96
LDDM-U (ours)
71.52
38.89
79.60
52.34
76.81
45.05
41.02
els (MDLM) (Shhoo et al. 2024) and the Uniform Diffusion Language Models (UDLM) (Schiff let al. 2024). creating LDDM-M and LDDM-U, respectively. We train these models on the One Billion Word (LMIB) (Chelba et al., 2013) and OpenWebText (OWT) (Gokaslan Cohen 2019) datasets. Following the training phase, we evaluate their performance based three key metries: likelihood, zero-shot likelihood on unseen datasets, and the quality of the generated samples. For fair comparison, all models were trained under identical settings, using the same datasets and training configurations, following the setup of [Shhoo et al. (2024). detailed description of our configuration provided in Appendix C.I
Likelihood Evaluation. As shown in Table I our loopholing mechanism consistently improves performance across both masked and uniform diffusion frameworks on the LMIB and OWT datasets. This result indicates that our approach is effective regardless of the underlying diffu- sion model type. Perplexity was measured by approximating the Negative Evidence Lower Bound (NELBO; Eqn. 3). with further details on the evaluation protocol available in Appendix C.1.2
Zero-Shot Likelihood Evaluation. To assess the generalization performance of our models, we evaluated the models trained on the OWT dataset across a diverse set of unseen datasets. A detailed description of these evaluated datasets provided in Appendix C.I.3
As shown in Table 2 LDDM-M, our loopholing-ehanneed MDLM, consistently outperforms the baseline MDLM on all evaluated unseen datasets except for LMIB. In contrast, LDDM-U exhibits only marginal improvements over UDLM, with the exception of the PTB dataset. We hy- pothesize that this performance disparity stems from the fundamental differences in how perplexity is calculated for uniform diffusion framework. Uniform diffusion calculates perplexity over all to- kens, making the metric highly sensitive to domain shift between the training and test distributions rather than the effectiveness of the loopholing mechanism.
Generation Quality Evaluation. The loopholing mechanism is designed to address the sampling
wall issue, property that is difficult verify solely through likelihood evaluation. To directly assess its impact on generation quality, we therefore employ two alternative metrics. First, we measure the perplexity of unconditionally generated samples using pretrained GPT-2 Large model (Radford ett al., 2019), which we refer to as generative perplexity (Gen PPL). Second, we utilize GPT-4.1 score the consistency and naturalness of the samples on a 0-to-10 scale, following the G-eval framework Liu et al. 2023). Specific experimental details are provided in the Appendix C.1.4
As depicted in Figs. and 4a), applying loopholing yields substantial improvements in genera- tive perplexity. For instance, at 1024 sampling steps. LDDM-M achieves Gen PPL of 49.13, more than halving MDLM's 108.94. Similarly, LDDM-U (28.76) shows about 2.5x improve- ment over UDLM (73.95). Critically, this performance gap is not limited to single step count; unlike their baselines which show signs of saturation, both LDDM-M and LDDM-U exhibit a con- sistent downward trend in perplexity as the number of sampling steps increases. This demonstrates that loopholing enables meaningful, continuous refinement at each step, directly mitigating the steps without progress issue. Notably, shown in Fig. 4a), LDDM-U also surpasses the strong auto- regressive baseline after 512 steps. Furthermore, these quality gains are achieved without sacrific- ing diversity, evidenced by the stable sentence entropy. This suggests that loopholing improves
7

---

Published as a conference paper at ICLR 2026
Generative Perplexity
G-eval Soores
Best p Values
100
um M
URM
1.0
L00M 14 bant
LOOM poun))
Generatuv Parplosty (1)
11.5
[9.57]
(8.66)
(5.00)
00)
(5
Scare
3.4
0.
[3.52]
[5.57]
Comeshencyy
PT8
Wildos
Lantbodd
Hovs
Pidmedd
The Maximum Number of Sompling Steps
(a) Unconditional generative perplexity measured using GPT-2 Large, with values in parentheses indicating the sentence entropy of the generated samples. (b) Evaluation of generation quality for consistency and naturalness using G-eval framework, rated by GPT-4.1 on 1-10 scale. (c) The optimal value of the self- conditioning rate (p) that yields the lowest zero-shot perplexity for each dataset.
Figure 4:
generation quality not by collapsing the output distribution, but by guiding the sampling process more effectively within a rich and diverse token space.
The G-eval results in Fig. 4b) further substantiate these findings on human-aligned metrics. The marked improvement in consistency suggests that by maintaining a richer contextual representa- tion throughout the whole generated sequence, loopholing helps the model produce more coherent and logically connected text. Similarly, the higher naturalness scores indicate that by propagating the rich contextual latent, the model generates more fluid and human-like sentence structures. To- gether, these automated and human-aligned evaluations confirm that loopholing provides robust solution to enhance generation quality in discrete diffusion models, Additional evaluation results on downstream tasks are provided in Appendix D.2
6.2 REASONING TASK
To evaluate the effectiveness of loopholing on reasoning tasks, we integrate it into the Multi- Granularity Diffusion Model (MGDM), masked diffusion framework designed for reasoning (Ye ctt al. 2024). resulting in the model we refer to as LDDM-G. We evaluate its performance on two arithmetic reasoning tasks that require high logical precision: Countdown (Gandhi al. 2024) and
Game of 24 Yao et al., 2023). The objective in these tasks is to generate valid arithmetic formula that yields target number using a given set of digits. Comprehensive implementation details are
provided in Appendix C.2
As presented in Table LDDM-G demon- strates substantial performance gains over the MGDM baseline across all evaluated tasks and model scales. For instance, with the 85M pa- rameter model, LDDM-G achieves a 16% improvement Game of 24 and almost 8% gain on Countdown 4. The loopholing mechanism drives these improvements by pre- serving contextual ambiguity, rather than pre- maturely committing to a single token per step. This allows to maintain a richer representation of the solution space, enabling a more effective exploration of the multiple pathways required in complex reasoning tasks and ultimately enhancing its capacity for structured, multi-step reasoning.
Table 3: Success rates (%) on the Countdown (CD) and Game of 24 (G24) tasks. Architecture
Params
CD 4
G 24
CD 5
MGDM
6M
45
12
5.9
85M
86.5
47
35.7
LDDM-G (Ours)
6M
56.3
28
10.3
85M
94.4
63
41.3
6.3 ABLATION STUDY
In this section, investigate the effects of our design choices by conducting series of abla- tion studies. Specifically, we analyze the performance variations with respect to different self- conditioning rates in training, assess the impact of propagating the continuous latent representation, and examine the changes in excessive oscillation upon applying the loopholing mechanism.
Reeported MGDM baselines may differ slightly from those in Ye et al. 2024) due to differences in evalua- tion criteria. Specifically, we consider solutions invalid if they numbers not provided in the input or derived from previous expressions, which were filtered under the original codebase.
8

---

Published as a conference paper at ICLR 2026
Generative Perplexily
KL Divergence
Entropy
120
0/6
Gonercativn Perplesity (-)
MOLM
LDDM-M
Divergence Beeween Steps
0.4
0.5
MDLM
LUUM-M (oun)
LUUM-M (Oun))
UDLM
0.3
UDLM
UOLM
LODMIU (ours)
(surs)
Entroy (1)
(ouns)
80
0.2
60
0.1
40
20
16 32 64 128f 250 517-g7
0.05 The Number Samaling Stegs
Exploration
Exploitation 800
1000
Letent Propagation Skeps
The Sampling Steps
(b)
(c)
Figure 5: (a) Generative perplexity across varying latent propagation steps. (b) KL divergence (log-scale) between the predicted token distribution at each step and the distribution from 20 steps prior (f-20) during the generation. (c) Entropy of the predicted token distrihutions throughout the generation.
Self-Conditioning Rate. We first evaluate the impact of varying the self- conditionio rate, denoted by p. We assess performance by measuring zero-shot perplexities on the datasets from Section 6..1 using models trained on the LMIB dataset. The results, presented in Fig. 4c) show that LDDM-M generally achieves its best performance across various unseen datasets when p set between 0.5 and 0.9. This suggests that this range provides an effective balance, allowing the contextual la- tent representation to be robustly learned and utilized through the two-pass eeff-conditionnn mech- anism, with detailed results shown in T
Latent Propagation Length. We further investigate whether the efficacy of the loopholing mech- anism accumulates over time. To see this, we assess the model's performance latent propagation length k. Specifically, using a model trained on the OWT dataset, we generate samples with 1024 sampling steps and, every k steps, reset the context latent to the self-conditioned latent instead of carrying it over from the previous step. This procedure limits the accumulation window, so larger l means longer propagation. As shown in Fig. 5(a), performance improves k increases, suggesting that the sustained propagation of accumulated latent information is effec- tive in generating higher quality samples.
function of the
Idle Steps and Excessive Oscillation. cillation, introduce two (TPE). For a sequence of length L. over T number of sampling steps, these are defined as follows:
To investigate whether Loopholing mitigates excessive os- KL divergence (TKL) and -reeictrreii Entropy
f.
(1K ITAE
DEL (x (1xxx.)
HTPE(t)
H(xb,)), L 2=1
(9)
where DKL is the KL divergence and H is the entropy. The TKL metric evaluates the rate of change in the token distribution across denoising steps (here measured with 20-step lookback). whereas the TPE metric assesses the level of confidence or certainty in the model's predictions at each step. We therefore interpret high TKL as reflecting faster progress along the denoising trajectory-closey) tied to the steps without progress phenomenon--werea. a high TPE value serves as an indicator of excessive oscillatory behavior during generation.
We measure these metrics on models trained on OWT dataset with 1024 sampling steps. In Fig. 5(b), we first see an interesting crossover points in the middle where the behavior between LDDMs and non-LDDM-baan -bsed models reverses. During the first half, the LDDM-based models show higher TKL than non-LDDDM models. This means that LDDMSs update their predictions more actively at each denoising step. We see this phase, where the model tries to search for the target topic to generate, as an exploration phase. Interestingly, the trend reverses during the second half of the de- noising steps by showing lower TKL than non-LDDM models. This means, LDDMs try to change more conservatively showing less uscillations. As shown in Fig. B(c), LDDMs maintain consis- tently lower token-level entropy. This indicates that the stable contextual information carried by loopholing allows the model to make more confident and decisive predictions across the denoising trajectory.

---

Published as a conference paper at ICLR 2026
7 DISCUSSION
Computation and Memory. While loopholing significantly improves the performance of discrete diffusion models, it also introduces certain limitations. Most notably, training requires about 30% more time compared standard models, although it adds almost no overhead at inference time. In addition, doubling the embeddings to support both the sampling and contextual pathways leads to increased memory consumption. Moreover, the current training formulation considers only single- step updates, suggesting potential benefits from explicitly designing multi-step training strategies to better exploit long-range dependencies through the context latent path.
Relations to Recurrent Neural Networks. An insightful perspective on loopholing and LDDMs is to view them through the lens of Recurrent Neural Networks (RNNs) (Goodfellow al., 2016). In this interpretation, the deterministic update path in loopholing corresponds to the hidden-state update of an RNN, while the stochastic and discrete outputs play the role of the RNN's output, which then fed back as input at the next step-akin to autoregressive RNN. However, the key difference lies in the training procedure: loopholing diffusion enables simulation-free training, whereas RNNs typically require rollout-base training. We believe that further exploring the connection between loopholing diffusion and RNNs would be a compelling direction for future work.
General Limitations. Furthermore,
current contribution is novel architecture supported by empirical evidence. However, a rigorous mathematical framework that incorporates loopholing into the standard diffusion framework has not yet been developed, marking natural direction for future theoretical work. In terms of scalability, our experiments have thus far been conducted on moder- ately sized models feasible within an academic setting, and extending loopholing larger scales will be important for fully assessing its potential.
CONCLUSION
In this work, we identified the sampling wall as a key limitation discrete diffusion models, where rich distributional information collapses into one-hot representations, leading to inefficiencies such as steps without progress and excessive oscillation. To overcome this, we proposed the loopholing mechanism and developed Loopholing Discrete Diffusion Models (LDDMs), which preserve and propagate distributional context latent across denoising steps through a deterministic latent path- way. Extensive experiments demonstrated that LDDMs improve fluency, naturalness, and semantic consistency in text generation and reasoning tasks, significantly narrowing the performance gap with autoregressive models. These results highlight loopholing as a general mechanism to enhance discrete diffusion, with promising future directions including multimodal extensions, theoretical understanding, and integration with broader non-autoregressive frameworks.
REPRODUCIBILITY STATEMENT
We provide detailed information datasets, model configurations, training procedures, and eval- uation protocols in Appendix C Our source code and scripts for data processing, training, and evaluation available at https: http::::://iittbb.... cotpuu-..c////mmmmma. ahn-ml mlmlddmm
ACKNOWLEDGMENT
This research was supported by Brain Pool Plus Program (No. 202HH1D3A22003103665 and GRDC (Global Research Development Center) Cooperative Hub Program (RS-2024-00436165) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT). This research was also supported by the AI Computing Infrastructure Enhancement User Support Program (RQT-25-09019) funded by the Ministry of Science and ICT (MSIT). Re- public of Korea. We thank the members the Machine Learning and Mind Lab (MLML) and Lan Tran for valuable discussions and assistance.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms, 2019.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 1 11981-179933 2021.
Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXivv1607.064500 2016.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense in natural language. In Thirty-Fourrth AAAI Conference Artificial Intelligence, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-28279, 2022.
Chen-Hao Chao, Wei-Fang Sun, Hanwen Liang, Chun-Yi Yi Lee, and Rahul G Krishnan. Be- yond masked and unmasked: Discrete diffusion models via partial masking. arXiv preprint arXiv:2505. 18495, 2025.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3000 2013.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with seff-conditioning. arXiv preprint arXiv:2208.04202. 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 100.050355557, 2018.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. discourse-aware attention model for abstractive summarization of long documents. arXiv preprint arXiv:/804.05685, 2018.
Justin Deschenaux, Lan Tran, and Caglar Gulcehre. Partition generative modeling: Masked model- ing without masks. arXiv preprint arXiv:2505. 18883, 2025.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404. 2306.0633, 2024.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 2024. URL htttt:://// /wnodoe www.mooo..rog/eeccr////000000000
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 133345-133385, 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. OpenWebTeextCorpus 2019.
1o/

---

Published as a conference paper at ICLR 2026
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: 7/www. www.deepleezm.bbooooooo ozq
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. arXiv preprint 3rXiv:22/2.11977 2022.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv:2502.06768, 2025.
arXiv preprini
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 09221666-21700 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint I4Xivv:412.69000 2014.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303. 16634, 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310 10. 16834, 2023.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):11-330, 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint SEKIv:1609.07844/ 2016.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint 1rX1v22410.18144 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992 2025.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.0377 03736, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 1606.060300003 2016.
William Peebles and Saining Xie. Scalable difffusion models with transformers. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 4195 1195-4205, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 30116-1130884 2024.
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang. Justin Chiu, and Volodymyr Kuleshov. The diffusion duality, arXiv preprint arXiv:2506. 10892, 2025.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint 1rXiv:1907.10644 2019.
Yair Schiff, Subham Sekhar Sahoo, Hao Phung. Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint 42 2024.
12

---

Published as a conference paper at ICLR 2026
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103161 2024.
Jascha Sohl-Dickstein, Bric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Dimitri von Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf. and Thomas Hofmann. Generalized interpolating discrete diffusion. arXiv preprint arXiv:2503.04485 2025.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00::00 2025a.
Wen Wang, Bozhen Fang, Chenchen Jing. Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chunhua Shen. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint arXiv:2508.09) 09138, 2025b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1111981988222 2023.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint 2.11:22410.14157 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Lingxiao Zhao, Xucying Ding. Lijun Yu, and Leman Akoglu. Unified discrete diffusion for categor- ical data, arXiv preprint arXiv:2402.03701, 2024.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02908 2024.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 1097-1100, 2018
Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao. Text generation beyond discrete token sampling. arXiv preprint arXiv:2505. 2025.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, we utilized a large language model to improve grammar, clarity, and overall readability.
B UNIFORM DIFFUSION MODELS (UDMS)
In contrast to the masking approach of MDMs, Uniform Diffusion Models (UDMs) employ a uni- form noising strategy (Austin et al. 2021 Schiff al.] 2024). In UDMs, the prior distribution is uniform distribution over the vocabulary, denoted as u = 1/K, where 1 is a K K-dimensional vector of all ones. This forward process gradually replaces an original token with a random token drawn uniformly from the vocabulary.
Similar to MDMs, generation performed by approximating the clean data in the true reverse posterior gzz [z, x) with neural network (z, t). The specific formulation for this approximated posterior is given by:
(22,/a,, xo(2,, ()) Cat Z3g Kazz, 2 + (an)= as oz) + (a, -x))x0 + (a,-a))(1-0)
Ka,
Ka, (xs: 2,) + C22
(10)
where O denotes the Hadamard product, Ctjs = % and X2 is shorthand for xa t). Unlike MDMs, which fix generated tokens, UDMs enable iterative refinement the entire sequence.
For training. UDMs also optimize the continuous-time formulated NELBO. The training objective takes the following form (Schiff et al. 2024):
a
K
K
(xo); xj (Ro); Ri
[NLLBO 1E,01(0011 (
log
(11)
Kat
8;
{88)
X;
where (x); the j-th element of vector x, = Ka;x (1- 04 1, Xg K Kax + Cr and i = je( (Zr); is the index of the non-zero entry in 2;. Recently, Duo (Shhoo et al. 2025)
proposed a low-variance objective for UDMs with improved empirical performance.
C EXPERIMENT DETAILS
C.1 LANGUAGE MODELING
C.1.1 EXPERIMENT DETAILS
MDLM and UDLM settings. For our implementation, followed best practices from prior work: we employed 64-bit precision for MDLM to ensure more accurate categorical sam-
pling (Zheng al. 2024), and we adopted the loss implementation from (Sahoo et al. 2025) for UDLM to improve its numerical stability.
LMIB. For the One Billion Word Dataset (LMIB) CCelba et al., 2013). we used the
bert-base ses--baaeuusee sasse--ncased tokenizer (Devlin et al. 2019) with fixed context length of 128 tokens. Sequences shorter than this were handled with padding. The model architecture is based
the Diffusion Transformer (DiT) (Peebles & Xie 2023) with rotary embeddings (Su et al., 2024). The model consists of 12 Transformer blocks, each with 12 attention heads and a hidden dimension of 768. A dropout rate of 0.1 was applied throughout the model.
For optimization, we used the Adam optimizer (Kingma & Ba 2014) with learning rate of 3e-4, betas of (0.9, 0.999). and an epsilon of 1e-8. The learning rate was linearly warmed from 0 to 3e-4 over the first 2,500 steps. Training was conducted for 1M steps on 8 NVIDIA RTX 4090 GPUs. The global batch size was set to 512, which was achieved by assigning batch size of 32 to each GPU and applying gradient accumulation over 2 steps. Additionally, we applied an Exponential Moving Average (EMA) with rate of 0.9999 and gradient clipping with threshold of 1.0. For LDDMs, the colficonditionin rate was set to p = 0.9, and to stabilize early training we initialized layer normalization parameters 7 0.
14

---

Published as a conference paper at ICLR 2026
used the gpt tok- enizer (Radford et al. 2019) with a context length of 1024 tokens. To maximize context utilization, employ sentence packing during preprocessing (Austin al. 202I). The model architecture and hyperparameters are largely similar to the LM1B experiment. Specifically, it based on a DiT with rotary embeddings and includes 12 Transformer blocks, 12 attention heads, hidden dimension of 768, and a dropout rate of 0.1.
OWT. For the Open Wex (OWT) dataset (Gokaslan & Cohen 2019).
Optimization was performed using the Adam optimizer (learning rate 3e-4, betas=(0.9, 0.999), epsilon=le-8), with a linear learning rate warm-up over the initial 2,500 steps. The model was trained for 1M steps using 16 H100 GPUs. The global batch size was 512. configured by assigning a batch size of 32 to each GPU. Similar to the LMIB setup, we applied an EMA with a rate of 0.9999 and gradient clipping with a threshold of 1.0. LDDMs used self-conditioning rate of 0.9, and we zero-initialized LayerNorm (B = 0) to stabilize early training.
Time Conditioning. To remain faithful prior work, adopt the canonical setting used by each baseline: MDLM and LDDM-M are trained without time conditioning following SaaooioogSShoooooo et al. (024);; SEDD (Lou 2023), UDLM (Schiff et al. (024), and LDDM-U use time conditioning
their original implementations. We treat time conditioning as orthogonal to our contribution-lophollioe
adds a deterministic latent pathway and can be combined with either setting-so we do not tune beyond faithfully reproducing baselines.
C.1.2 PERPLEXITY DETAILS
In discrete diffusion models, we approximate perplexity (PPL) using the Negative Evidence Lower Bound (NELBO; Eqn. 3
The perplexity for a sequence of length L,, X1.L, is defined and upper-
bounded as follows:
PPI.(x..)) = exp ---
-
log Togp(x: xxi)) = exp(-t log ((ogoooop(x:)))) S eexp (EIBO
(12)
Here, the marginal log-likelihood log p(x) is intractable to compute directly. We leverage its re- lationship with the Negative Evidence Lower Bound (NELBO), where - p(x) S NELBO(x). This relationship allows us to use the computable upper bound as our perplexity metric. Since the NELBO is computed via single Monte Carlo estimation over time steps t, which can be stochastic and exhibit high variance. To reduce the variance of this estimation, we adopt the low-discrepancy sampling technique from MDLM (Sahoo al.) 2024), which ensures that sampled time steps for each batch are more evenly spaced across the time interval [0.1].
Even with this improvement, the final value can be influenced by factors like batch size and hard- ware. Therefore, to ensure a fair and consistent evaluation, we compute all perplexity scores using fixed experimental setup: two NVIDIA RTX 4090 GPUs with batch size of 16 per GPU, under an identical software environment. (same Py Torch version, CUDA version, and library configurations).
C.1.3 DATASETS FOR ZERO-SHOT LIKELIHOOD EVALUATION
We evaluate our model's zero-shot likelihood on a diverse suite of standard benchmarks. The datasets include the Penn Treebank (PTB) (Marcus et al.) 1993), Wikitext (Merity al., (016)) Lambada (Paperno et al. (2016), AG News (Zhang et al., 2015), and a corpus of scientific articles from Pubmed and Arxiv (Cohan et al., 2018).
C. GENERATION QUALITY EVALUATION
Generative Perplexity. We evaluate the quality of generated text using generative perplexity (Gen PPL., computed with pretrained GPT-2 Large model (Radford et al.) 2019). Given a generated sequence of length L composed of discrete tokens x (I:T.) the perplexity is calculated as:
1. log (x(()) xx(=i)))
exp
(13)
1=1
15

---

Published as a conference paper at ICLR 2026
This metric reflects how likely the GPT-2 large model considers the sample, providing proxy for overall sample quality. For this evaluation, we generate 512 samples using the model trained on OpenWebText (OWT) dataset and report the average perplexity across all samples.
Sentence Entropy. As shown in Zheng et al. 2022), Gen PPL can be deceptively low when gen- erations have very low sentence entropy. To check for this, measure sentence entropy for each sample. Sentence entropy indicates how many diverse tokens are used within a sample.
For single generated sample of length I., let count(v) be the number of times token appears. The sentence entropy for that sample is:
count(e)
(count(F) (1)
log
(14)
L
VE
where V is the vocabulary. We generate 512 samples and report the average sentence entropy.
G-eval Scoring. We evaluate the quality of generated texts using the LLM scoring framework (Liu et al. 2023), with GPT-4.1 serving as the evaluator. For each model trained on WexTett (Gokaslan & Cohen 2019) dataset, sample 512 unconditional generations. Due to sentence packing during training, each generation might contain multiple sequences separated by the end-of- sequence token ([EOS)). ). For evaluation, retain only the first sequence.
Each sequence is independently rated by GPT-4.1 based on two criteria:
Consistency (1-10): Evaluates whether the generated text maintains coherent topic with- out contradictions or context shifts.
Naturalness (1-10): Assesses grammar, fluency, idiomatic usage, and freedom from spelling punctuation enrors.
Scores are assigned in a zero-shot manner using the prompt provided in Fig. 6 To improve the accuracy of the measurements, we performed four evaluations for each sequence with the tempera- ture set to 1.0 and assigned the average of these scores. We report the average score across all 512 sequences as the final measure of model quality.
C.2 REASONING TASK
Datasets. We evaluate our method on two arithmetic reasoning tasks that require multi-step rea- soning: Countdown (Gandhi et al., 2024) and Game 24 (Yao et al., 2023). The Countdown task requires the model tion, multiplication, and division) to reach a specific target number. For instance, in Countdown4, given the input numbers (24, 59, 23, 77) and target of 29, valid solution is to first calculate 24 - 59 = 83 and 77 - 23 = 54, and then use these intermediate results to reach the target with 83 54 = 29. Countdown5 extends this task to five input numbers, while the Game of 24 is vari- ant of Countdown4 where the target is always 24. We use the datasets released by Ye et al. (2024) without additional filtering or preprocessing.
given set of numbers and basic arithmetic operations (addition, subtrac-
Setup. Our experimental setup follows that of the Multi-Granularie Diffusion Model (MGDM) Ye et al. 2024). We use the MGDM as our base architecture and integrate the loopholing mech- anism to create what we call LDDM-G. MGDM extends the standard discrete diffusion objective with an adaptive token-level reweighting term and an easy-first TopK decoding strategy at inference.
Regarding TopK decoding, we made notable adjustment to the original implementation. The original MGDM recalculates probabilities over all tokens (both masked and unmasked) at each step, which permits overwriting already generated tokens-a form of remasking. This approach conflicts with the training objective, which focuses exclusively on predicting masked tokens. Therefore, following prior work (Nie al., 2025 Kim al. 2025). modify the decoding process to keep unmasked tokens fixed and apply the uncertainty- TopK selection only to masked positions.
However, Table 4 shows that the original MGDM-style TopK decoding leads to slight performance gains. This is noteworthy because the method relies on distributions over unmasked tokens, for which it was not explicitly trained.
16

---

Published as a conference paper at ICLR 2026
You will be given piece of text. Your task the on metrics. Please read and understand these instructions carefully, and keep this document open while reviewing.
Evaluation Criteria
Consistency (I-10)
high score (10) indicates that text maintains a single, coherent context throughout. A low score (1) given if the text shifts topic, contradicts itself, or loses logical flow.
Naturalness (1-10)
high score (10) means text is grammatically correct, idiomatic, and spelling
punctuation errors.
A low score (1) is given if the text contains frequent grammar mistakes, awkward phrasing, or typos.
Evaluation Steps
1. Read the generated text carefully and identify its intended context and message.
2. For Consistency, ask yourself:
Does the text stay topic without introducing unrelated ideas?
Are there any contradictions abrupt shifts meaning? For Naturalness, ask yourself:
Is the writing grammatically sound and easy to read?
Are phrases idiomatic, and is punctuation used correctly?
4. Assign each metric score from (lowest) 10 (highest) based the above definitions.
Text:
{sample}
Evaluation Form (scores ONLY):
Consistency:
Naturalness:
Figure G-Eval prompt template used with GPT-4.1. {sample} is replaced with the generated sequence to evaluate.
Table 4: Performance using original MGDM TopK decoding. Success rates (%) on Countdown and Game of 24 tasks using the original MGDM TopK decoding implementation, which includes probabilities over masked tokens. Numbers parentheses indicate accuracy improvements from using MGDM Topk decoding
Architecture
Params
CountDown4
Game of 24
CountDown5
MGDM (Retrained)
6M
47.6 (+2.6)
12
7.6 (+1.7)
85M
87.1 (+ 0.6)
52 (+5)
36.9 (+1.2)
LDDM-G (Ours)
6M
57 (+0.7)
29 (+1)
11.6 (+1.3)
85M
94.5 (+0.1)
64 (+1)
42.6 (+1.3)
Training. Both MGDM and LDDM-G models are trained for 600 epochs with batch size of 1024. We use the Adam optimizer (Kingma Ba 2014) with betas of (0.9, 0.999) and an epsilon of 1e-8. For the 6M models, we use learning rate of 1e-3, and for the 85M models, we use learning rate 3e-4; a cosine learning rate schedule employed for both. All models are trained on RTX 4090 GPUs. The model architecture is based GPT-2 (Radford et al. 2019). without the causal mask for bidirectional attention. MGDM is trained with hyperparameters a 0.25 and | = 2. For LDDM-G, apply seff-conditioning with p 0.9.
Evaluation. A generation is considered correct if the resulting arithmetic expression evaluates exactly to the target number without reusing inputs or generating invalid intermediate values. Our evaluation script employs stricter criteria than the original MGDM implementation. Specifically, the original evaluation only verified that each intermediate equation and the final result were mathemat- ically valid, without penalizing generations using numbers provided in the input not derived
17

---

Published as a conference paper at ICLR 2026
There way he would come here his own.
He ordered a cup of coffee, and then we just sat in silence. "So," Aidan finally said, "How's ' going?"
laughed. "Not much has changed since the last time I saw you."
"Ya know, you here lot," said Aidan
Figure 7: An example from the LAMBADA dataset. The goal is to predict the final word, "Aidan".
from previous expressions. We enhance this by explicitly filtering out generations using such invalid numbers, ensuring faithful and input-grounded reasoning.
D ADDITIONAL RESULTS
D.1 DETAILED RESULTS FOR SELF-CONDITIONING RATE
Table 5: Perplexities (1) of SEDD Absorb, MDLM, and LDDM-M with varying ceff-conditioning rates p. The model trained with diverse p values and evaluated with p 1.0. All scores reported upper-bound estimates.
LM B(trained)
PTB
Wikitext
Lambada
AG News
Pubmed
Arxiv
SEDD Absorb
28.39
108.63
78.61
99.44
61.57
75.09
142,19
MDLM
27.60
110.90
74.43
100.11
60.50
70.72
140.62
LDDM-M (ours)
1.0
26.34
101.92
70.51
92.17
57.71
69.25
140.29
= 0.9
25.95
99.92
66.87
89.62
56.72
67.38
136,04
= 0.7
26.14
102.75
66.55
90.64
56.43
67.09
134,73
0.5
26.39
100.12
66.58
89.63
57.99
67.01
128.06
0.3
26.66
101.88
71.90
90.16
58.46
68.08
135.33
0.1
26.88
100.52
69.02
90.38
59.10
69.69
136.25
D.2 DOWNSTREAM TASKS
We evaluated MDLM and LDDM-M on various downstream tasks using the 1m-eval-harness library (Gao et al. 2024). Both models were pre-trained on OpenWebText.. Our evaluation includes six multiple-choice Basks-ARC-Eand and ORC-Challenge (Clark al. (018). HellaSwag (Zellers) 2019), MathQA (Amini al.) 2019), PIQA (Bisk et al. 2020), and WinoGrande (Sakaguchi
ctt al. cct al.
2019 and one generation task, LAMBADA (Paperno et al., 2016). Since the library is designed for autoregressive models, adapted the approach for the masked diffusion framework.
Multiple-choice tasks. Following Deschenaux et al. (2025). we adapted the evaluation for MDLMs. Autoregressive models select the answer with highest log-likelihood via arg max; log p(y, where x is the context and y, is an answer option. However, MDLMs model the joint probability log p(x, y:) of the entire sequence, requiring a different approach.
Bayes' rule provides solution by connecting conditional and joint probabilities:
log p(y, x) = log p(x, y:) log p(x) x logp(x,yy
(15)
Since log p(x) constant across all answer options, ranking by joint probability is equivalent to ranking by conditional probability. We bound the joint probability using NELBO, compute Monte Carlo estimates, and select the answer with the lowest NELBO.
LAMBADA task. Unlike multiple-choice tasks where models select from given options, LAM- BADA requires generating the last word and comparing it with the ground truth. During evaluation, identified a critical issue with the dataset format. As shown in Fig. the final word lacks any terminal punctuation (period, question mark, etc.). While this poses no problem for autoregressive models that only consider preceding context, it creates significant challenge for discrete diffusion
18

---

Published as a conference paper at ICLR 2026
Table 6: Performance downstream tasks. LDDM-M substantially outperforms MDLM the generation task (LAMBADA) while showing comparable accuracy on multiple-choice benchmarks.
Model
LAMBADA
ARC-e
ARC-c
HSwag
MathQA
PIQA
WinoG
MDLM
40,46
36.49 36.03
25.17
31.81
21.51
57.62
51.85
LDDM-M
52.40
23.21
33.11
22.21
58.16
51,46
Table Impact of adding an MLP layer on perplexity across benchmarks. bounds.
All reported values are upper
LMIB((rained)
PTB
Wikitext
Lambada
AG News
Pubmed
Arxiv
MDLM
27.60
110.90
74.43
100.11
60.50
70.72
140.62
+ MLP
27.21
107.96
74.61
97.04
59.10
70.31
139.42
LDDM-M (ours)
25.95
99.92
66.87
89.62
56.72
67.38
136.04
+ MLP
25.87
99.54
66.96
89.33
56.43
66.05
127.71
models. When the input is formatted as said [MASK] [ECS], the EOS token signals the end of sequence, causing the model generate punctuation marks to properly terminate the sentence rather than predicting the target word "Aidan".
To resolve this issue, modified the model input by inserting additional [MASK] token before the [EOS] token. This extra position serves as a placeholder for terminal punctuation, allowing the model to correctly predict the target word in the original mask position. The added token is used only during inference and is not included in evaluation. For targets spanning multiple tokens, we adopted the iterative decoding implementation from Nie al. (2024), where we unmask one token at time based on confidence. These modifications resulted in improved performance and more reliable evaluation.
Results. Table shows that LDDM-M achieves notably higher accuracy on the generation task (LAMBADA) with 52.40 vs 40.46, while maintaining comparable performance on the likelihood- based multiple-choice tasks.
D.3 DETERMINISTIC PATH AUGMENTATION
To understand the effect of incorporating additional parameters into the deterministic path, we inves- tigate augmenting the deterministic path with a two-layer MLP featuring an expansion ratio of 4. In this modification, we update the latent embeddings as h = h, + MLP((h,, followed by layer nor- malization to produce the latent representation e; (2) 2z LN(h)). The original MDLM archi- tecture applies similar MLP directly to the token embeddings, where z Es (2:) MLP (EE (z:)) and the latent representation computed as e, = (z,) + LN(x)), The subsequent structure the same as before (see Section 3.1 The model was trained the LMIB dataset.
As shown in Table 7, this augmentation yields only marginal performance gains. This finding sug- gests that the primary benefit of our framework stems from contextual latent propagation across time steps, rather than from additional parametric complexity in the deterministic path.
D.4 APPLICATION OF LOOPHOLING TO AUTOREGRESSIVE MODELS
To investigate the broader applicability of our proposed mechanism, we adapted loopholing for standard au- toregressive language model. During training, the model performs two forward passes. The first pass generates a pseudo-context from the input sequence. This pseudo- context is then shifted by one position to the right (with the first position embedding being zero vector), and fed
Table Unconditional Gen PPL. 512 samples (1024 sampling steps).
Model
Gen PPL (L)
AR AR-Loopholing
34.40
34.21
19

---

Published as a conference paper at ICLR 2026
Table 9; Perplexities (L) of Autoregressive models trained OpenWebText.
OWT (trained)
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
AR (Baseline)
17.27
118.39
34.88
55.91
48.22 45.00
60.60 57.01
42.72
46.40
AR-Loopholing
16.57
115.99
32.98
55.19
41.23
45.65
Table 10: Perplexities (L) of models with matched computational budget, trained OpenWebText. All ported perplexity values are upper bounds.
OWT(trained)
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
MDLM (IM steps)
23.05
86.33
36.30
66.73
48.36
68.62
41.94
37.52
MDLM (2M steps)
22.54
84.40
35.27
65.68
48.07
66.26
42.08
36.75
LDDM-M (1M steps)
21.90
85.80
33.27
69.53
44.22
62.55
39.74
34.96
into the second forward pass along with the original token embeddings. This setup allows each to- ken's prediction to be conditioned the continuous representation of the preceding token from the initial pass. During inference, the model generates tokens sequentially, passing both the generated token and its corresponding latent embedding the next step, technique similar to another existing method (Zhuang et al. 2025).
However, this approach did not meaningfully improve sample quality (Table 8). despite slight im- provement in perplexity scores (Table().. We attribute this outcome to the fundamental differences in their generation processes. In discrete diffusion, the sampling wall problem is more severe because the loss of the predicted distribution occurs for many tokens at every step of the iterative refinement. In contrast, autoregressive models operate with fixed context, stably predicting only one token at time. Consequently, the information loss from single sampling step is less significant, and the primary challenges loopholing designed to mitigate are less prevalent in this framework.
D.5 IMPACT OF MULTISTEP BACKPROPAGATION
In our standard self ceff-conditioning setup. gradients backpropagated only through the second forward pass for computational efficiency. To see if allowing gradients to propagate through more steps would be beneficial, we conducted an experiment. Since train-
Table 11: Gen PPL. on 512 samples (1024 sam- pling steps) after 10K fine-tuning steps.
Model
Gen PPL (1)
ing from scratch is computationally prohibitive, we performed a short fine-tuning experiment for 10K steps on a model pre-trained on the Web dataset for 1M steps with self-conditioning. In this
LDDM-M
49.13
Fwd Pass (10K)
83.08
+ 4 Fwd Pass (10K)
60.07
fine-tuning stage, we extended the training process to three and four forward passes, including the sampling steps between them mimic the actual generation process with 1024 denoising steps. Gradients from the final pass were allowed to flow back to the second pass, while the initial pseudo- context generation pass remained detached.
As shown in Table II this approach led to a decrease in performance, as measured by gen PPL. We hypothesize this IS because the standard two-pass setup, the second pass learns robustly refine any context from the detached first pass, since it cannot control its input. However, in the third and fourth passes, the model can influence the context it receives from the previous step. This likely leads to the model learning dependencies on specific context patterns, which harms its ability to generalize during inference and degrades generation quality.
D.6 COMPARISON WITH MATCHED COMPUTATIONAL BUDGET
The loopholing mechanism requires two forward passes during each training step. While total com- putational cost (FLOPs) also includes backpropagation, we naively matched the number of forward passes as proxy for the overall budget. To ensure LDDMs' performance gains stem from the deterministic path and not simply the increased computation, we ran a controlled experiment.
20

---

Published as a conference paper at ICLR 2026
Table 12: Comparison of Sample Quality between Masked Diffusion and Autoregressive Models. All metries were evaluated on 512 samples generated from models trained on Open OeenWeTText. Metries from OpenWeb- Text validation set are also provided for reference
OWT Dataset Validation (512 samples) Gen PPL: 14.88
Entropy: 5.442
Self-BLEUU: 0.2498
Autoregressive Model (T=1024) Gen PPL: 34.40
Entropy: 5.573
Self-BLEU: 0.2450
SEDD
MDLM
LDDM-M
T
Gen PPI.
Entropy
Self-BLFFU
Gen PPI.
Entropy
Self-BLFU
Gen PPI.
Entropy
Self-BIFUU
195.96
5.731
0.1971
198.72
5.745
0.1896
248.02
5.739
0.1673
64
143.04
5.681
0.2100
145.48
5.704
0.2051
122.34
5.674
0.2014
128
123.52
5.658
0.2153
125.04
5,681
0.2078
83.10
5.641
0.2156
256
114.12
5.643
0.2178
114.05
5.659
0.2126
64.76
5.600
0.2241
512
111.18
5.629
0.2178
112.83
5.652
0.2089
53.56
5.569
0,2289
1024
109.05
5,629
0.2214
108.94
5.637
0.2132
49.13
5,545
0.2301
2048
108.
5.625
0.2168
107.20
5.625
0.2096
44.51
5.518
0.2295
Table 13: Sample Quality of Uniform Diffusion Based Models. All metrics were evaluated on 512 samples generated from models trained on OpenWebText.
UDLM
LDDM-U
T
Gen PPL
Entropy
Self-BLEU
Gen PPL
Entropy
Self-BLEU
95.90
5.591
0.2447
75.55
5.551
0.2541
64
86,24
5.578
0.2436
56.27
5.538
0.2572
128
80.39
5.571
0.2407
45.47
5.518
0.2592
256
77.64
5.560
0,2423
38.76
5.494
0,2519
512
77.78
5.562
0.2367
32.83
5.447
0.2451
1024
73.95
5.547
0,2429
28.76
5.418
0.2343
2048
75.45
5.537
0.2380
25.06
5.366
0.2320
We trained baseline MDLM for 2 million steps, thereby matching the number of forward passes of our LDDM-M trained for 1 million steps. The results confirmed that the L.DDM- M (1M steps) significantly outperformed the MDLM (2M steps), as shown in Table 14 and Table I0 This demonstrates that the improve- ments achieved by loopholing are attributable the propagation of contextual information, not larger computational budget.
Table 14: Unconditional gen PPL on 512 samples (1024 sampling steps).
Model
Gen PPI. (L)
MDLM (IM steps)
108.94
MDLM (2M steps)
108.22
LDDM-M (IM steps)
49.13
D.7 DETAILED QUANTITATIVE ANALYSIS OF SAMPLE QUALITY
This section provides the specific numerical values for the sample quality analysis from Section 6. detailed in Table 12 and Table [3 We evaluate Generative Perplexity (Gen PPL). Sentence Entropy. and Belf-BLEU. Self-BLEU Zhu et al. generated text corpus. For our experiments, we compute Self-BLEU scores using up to 4-grams with uniform weights (i.e., weight of 0.25 for each n-gram from to 4). Lower Self-BLEU scores indicate higher diversity, as they reflect less n-gram overlap among the generated samples. The autoregressive model used for comparison a standard Transformer architecture. The analysis highlights that while LDDMs maintain contextual information about the target across denoising steps, they also maintain generation diversity. We attribute this to the high diversity the initial stages of generation, as suggested by our ablation study (Section 6.3)
2018) is a metric that quantifies the internal diversity of
D.8 VISUALIZATION AND QUANTIFICATION OF IDLE STEPS
To verify the presence of idle steps in MDLM, we examine the number of tokens opened during single generation trajectory. First, to visualize this phenomenon explicitly, we tracked the number of opened (updated) tokens at each denoising step for single sample generated by MDLM with 1,024 sampling steps. As illustrated in Figure a significant number of steps exhibit zero token changes. These instances correspond to idle steps, where the model fails to update any part of the sequence
21

---

Published as a conference paper at ICLR 2026
despite expending computational resources. The scatter plot clearly shows clusters of zero-value points, confirming that idle steps are distinct events individual trajectories.
To quantify the prevalence of this issue, measured the average frequeney of idle steps across 512 generated samples. We evaluated the model with varying total sampling steps (2). For = 256, 512, and 1024, observed average of 4.8, 69, and 376 idle steps, respectively. These findings align with the observations reported in Chao et al. (2025) confirming that standard discrete diffusion models suffer from substantial computational inefficiency due these non-progressive steps.
Number of Opened Tokens at each Sampling Step
Namberr Changio
Number of Opened Tokens
400
1000
Sampling Step
Figure 8: Number of opened tokens at each sampling step during a single generation process (T' 1024). The data points at 0 represent idle steps where tokens were updated, highlighting the redundancy in the generation process.
D.9 SCALABILITY ANALYSIS
To examine whether the improvements from Loopholing hold beyond our main configuration, trained models at moderately larger scale (169M + 424M: Depth: 24, Heads: 16, Hidden Dim: 1024). Both MDLM and LDDM-M were trained for 100K steps OWT under identical settings.
Table 15: Perplexities (L) at two model scales, trained for 100K steps on Open WebTex. All reported perplexity values upper bounds.
Model
Size
OWT
LMIB
LAMBADA
News
Pubmed
Arxiv
Wikitext
MDLM
169M
27.20
80.10
55,45 47.49
116.54
89.00
51.47
44.27
45.46
MDLM
424M
22.49
68.55
78.67
64.68
40.98
36.15
34.90
LDDM-M (ours)
169M
26.14
80.00
49.94 42.52
102.67
79.84 53.82
47.58
40.59
39.93
LDDM-M (ours)
424M
20.20
64.97
72.13
36.88
32.89
31.10
Table 16: Generative perplexity (1) at 1024 sampling steps for two model scales.
Model
Size
Gen PPL (L)
MDLM
169M
108.90 98.99
MDLM
424M
85.62
LDDM-M (ours)
169M
49.13
LDDM-M (ours)
424M
40.10
As shown in Tables 15 and 16 LDDM-M consistently outperforms MDLM at both scales. At 424M parameters, LDDM-M achieves Gen PPL of 40.10 compared to MDLM's 85.62, maintaining roughly 2.1x x improvement. The consistent gap across the two configurations indicates that the loopholing mechanism does diminish with increased model capacity.
22

---

Published as a conference paper at ICLR 2026
D.10 CONTINUED TRE-NIIINING WITH LOOPHOLING
To assess whether Loopholing can be applied to existing pretrained diffusion models without train- ing from scratch, we start from standard MDLM checkpoint trained for 1M steps on OWT and integrate the Loopholing mechanism by attaching zero-initialized layer normalization layers. We then continue pre-training with the LDDM-M self-conditioning objective for up to 200K steps.
Table 17:
Perplexities (J) of continued pre-training from MDLM (IM step) checkpoint. All reported per-
plexity values are upper bounds.
Model
OWT
LMIB
LAMBADA
PTB
AG News
Pubmed
Arxiv
Wikitext
MDLM (IM step)
23.05
66.73
48.36
86.33
68.62
41.94
37.52
36,30
LDDM-M (IM step)
21.90
69.53
44.22
85.80
62.55
39.74
34.96
33.27
MDLM LDDM-M (10K)
22.39
65.83
45.70
83.45
61.70
40.57
36.55
33,92
MDLM LDDM-M (50K)
21.91
65.34
45.09
82.20
60.53
39.22
35.59
33.35
MDLM LDDM-M (100K)
21.71
64.57
44.44
80.77
59.83
38.97
35.35
33.02
MDLM LDDM-M (200K)
21.50
64.42
44.27
80.04
59.44
39.10
35.14
32.83
Table 18: Generative perplexity (L) of continued pre-training across different sampling steps.
Model
T=256
T=512
T=1024
T=2048
MDLM (IM step)
114.05
112.83
108.94
107.20
LDDM-M (1M step)
64.76
53.56
49.13
44.51
MDLM LDDM-M (10K)
91.04
82.45
78.31
74.65
MDLM + LDDM-M (50K)
75.73
66.57
61.96
56.69
MDLM LDDM-M (100K)
71.01
60.72
57.20
51.59
MDLM LDDM-M (200K)
66.06
55.81
51.51
47.04
The adapted model surpasses the original MDLM (1M) within only 10K continued pre-training steps in most benchmarks. After 200K steps, it closely approaches the performance of LDDM-M trained from scratch for 1M steps. In terms of Gen PPL at 1024 sampling steps, continued pre- training reduces from 108.94 to 51.51, nearly matching the scratch-trained LDDM-M (49.13). These results confirm that Loopholing can be effectively integrated into existing diffusion model checkpoints through continued pre-training.
D.11 ISOLATING CELE-CONDITIONING FROM LATENT PROPAGATION
To isolate the contribution our recurrent latent path from the effect self-conditioning, we com- pare three variants: (a) standard MDLM, (b) LDDM-M with latent propagation (our method), and (c) a variant that propagates the sampled token (drawn from Xy) to the next step instead of the pre- sampling latent state h,, which is closer to standard self-conditioning (Chen et al. 2022). All models were trained for 200K steps on SpenWeeText with the same configuration.
Table 19; Generative perplexity (L) 1024 sampling steps, comparing latent propagation with sampled token propagation, models trained for 200K steps on Open WebText.
Model (169M, 200K steps)
Gen PPL (L)
MDLM
103.52
LDDM-M (sampled token)
93.71
LDDM-M (latent)
55.04
While propagating the sampled token offers slight improvement over MDLM, it lags significantly behind latent propagation. This confirms that preserving pre-sampling information via continuous latent path is the primary driver of Loopholing's ' improvements, rather than the self-conditioning training procedure itself.
23

---

Published as a conference paper at ICLR 2026
D.12 RESULTS UNDER ORIGINAL MGDM EVALUATION SCHEME
For transparency, additionally evaluate LDDM-G and MGDM under the original evaluation and decoding scheme Ye et al. )2024).
Table 20: Success rates (%) under the original MGDM evaluation scheme of Ye et al. (2024).
Model
Params
Countdown 4
Game of
Countdown
MGDM (retrained)
51.5
32
85M
88.1
62
52
LDDM-G (ours)
6M
60.6
45
23.6
85M
95.3
82
55.2
The relative performance trend remains unchanged under the original evaluation scheme, and LDDM-G continues to outperform MGDM across all tasks and model scales.
E IMPLEMENTATION PSEUDO-CODE
Here is the pseudo-code for the generation process and training step of Loopholing Discrete Dif- fusion Models (LDDMs) with self-conditioning. While the forward function shared across all models, the generation and training procedures are presented for LDDM-M for clarity. For LDDM- U, the same procedure applics by replacing the posterior and training objective with Eqn. I0 and Eqn. respectively.
Algorithm LDDMs Forward Function
1: Require: Current token sequence 2 (1:L) previous latent context h, (1:L) diffusion timestep t.
2: Parameters: Token embedding layer Eg, backbone fe. output projection go-
3: Output: Logits o (1:L) updated latent context h. (1:L)
4: function LODMFORWARD (1:L) ,
.h(.L) .t)
(1:Z)
(1:2)
Ez(zi (111)
5:
Embed token sequence. b Fuse input with normalized latent.
e [1:L]
+- v f1:L) + [LyerNorm((h.-))
6:
h (1:L) Solef (ile)):)),1) 1)
7:
Update memory state via backbone. Project to vocabulary space.
0, (1:L) - go (2:1)
8:
return of (1:2) h (1:L)
9:
10: end function
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 LDDM-M Generation Process
1: Require: Total diffusion steps T, sequence length L, forward function LDDMFORWARDG.
2: Output: A generated sequence.
3: z
(1:L) < (MASK)...... [MASK])
Initialize a sequence of length L with MASK tokens.
4: h
(1:2) +0
D Initialize the latent embedding to zero vector.
5: for
i T do
6:
+ -i/T
7:
11/T
Os (1:1) h1 (1:1)
<- LLDDMFORWARDD (1:L) (1:1. .1) D Predict logits and update latent.
8:
(z h
9:
(1:L) (I:L) (1:L)
(z+
hh,
.) Softmax (1:b)) Predicted distribution of the clean sequence.
10:
for = + do
P Element-wise sampling (independent across f).
11:
2.
(6) ((zs
(0)
Zi X (z)
(1:L) (0) (1:L) (1:L) .t)
h.
The posterior g defined in Eqn.
12:
end for
(1:L) Z, (1:L)
13:
14:
end for
(1LL)
15: return z,
Algorithm LDDM-M Training Step with Self- Self-Conditioning
Require:
Clean data of length L, (1::L, noise schedule at, self-conditioning rate p [0, 1], forward function LDDMFORWARDg.
2: Ensure: Training loss L.
3: ~ U[001)
>> Sample a random time step. Each token is processed independently. Sample noised input via the forward process q (Eqn.
4: for l 1= 1- do
5:
zi
{Z) (0)
(2)
6: end for
7: h (1:L)
P- Initialize the contextual latent to zero.
8: With probability p:
9: ppeddo LDDMFORWARD (1:L)
.,,t)
10:
h (1:L) STOPGRAD(h (LLL)
Update with the detached pseudo-context
11: o (1:2) + LDDMFORWARDDD(Z.. (1:L. (1:L)
cond
x, (1:L)
( [1:L) h (1:L)
cond .) -- Softmax (1:L)
C E1[
(8) m a- 1x (z( (1:L) h bond t). x())
af
(6)
Calculate loss based on Eqn.
14: return L
F SAMPLES
Figures 9 12 present unconditioned samples generated with 1,024 sampling steps from models trained on WeenWetTex. We highlight topic shifts and incoherent sequences the baseline sam- ples. MDLM exhibits both failure modes, while UDLM maintains local grammar but still drifts across unrelated topics. In contrast, LDDM samples preserve topical consistency over longer spans.
25

---

Published as a conference paper at ICLR 2026
(lendoftext)) NA laner damage/mon slot degenerate, they should all tend to take longer than the average achieve three splits in NA. Any midlaner is just very small sample that can pick out if everything in place will affect an offlaner, say an Assassin or a Templar. More specifically, assuming that top mid laner has important mana needs undertaker as timings increases. Conceptss that go into wisping teamfights always revolve around a tank composition. Picking the meta we want are to at least mitigate shield leech early unless you have max hp zaelus, paladin and kidd. Even if you are only one tank ADC with greater skill stack or if you don2019t necessarily run tanky, you should always try to avoid aggressive mid laners. How strong you are, also depends on the main map your opponents have and the level of your presence on most ones. For sure that it needs be built that you also have experience on an ally you can use on most main maps for early game control once cross biofus.
Defensive defensive champions:
The tank shield against crits/cooldown and Ept/HOME shenanigans shouldn2019t take too long to turn off any tank supports. The Breacher Shield can also be used to overcome team stump!
Champions ADC Stats 4 5 10 7 8 16 18 7 Cooldown 2 6 pulses CC in 50 base armor Regem Crit
Power 14120 Resist AP at max level, 430 CCD Payroll 930 Deviate AP at max level 10 Cyclone 900 Ranged Poisoning 50% on AP skills at max level, 1444 NET abilities 800 BKB Pol 2475 WCP De- fensive champions in gold per attack, and by number of CR per dmg. hexcaprec.com/SP lane-jpg
Round 1: Mega Troll
EN!1172 1278 440 Gemgenius "Crazy" Crunchy Trade# Stats of huge Twisters reported Card Stats Quotes Tactical level MasterOver 9000 Advl 125 WMPUlt eggoBlue File Page 316WPP 944
Converse TT's ' build:
I know it would make easy a transition to Katainl would be wise to try to pick maniac alphas on Aklematic. Honestly, stuns are the worst in the game and are considered sign of turbulent terrain. TT's beautifully short animations on these are always going to be a meta property, when stuns on Elanna leavoured, jungling jungling and flying slow are the fishworks in The anime.
More precisely speaking, he can step up to TBA if she works like a climber on a butterfly, then on cat,ale and back.
I asked reddit for its thoughts on laning when he released the very big curse Emptables. Ever since i wrote about his hegemony on my other news site, it seems we are back to juke with the topic of playtime or BlackStorm. Should are cheap, breaker to get back? And so ask, if TT does a slouch on his build like Alistar ormalfurr, have a smart hunch that if Imp or TTeM can really tank harder play fragile role, after he can supposedly replace, the playmaker will pick on stryrolls or slower side up protector. Barreds on TT's build (crashless auto auto heal, he wouldn't really lose any value factor like Oriana or AK, and tanking AP and INTs) are of the powerful tank attorrupts of all time. Kennalden and baniac are in the niche, but bull %s are good at gatstrings and base. Pro coaching:
OP police Profile Blog Joined Jamuary 2010 New Zealand 38849 Posts #16 Wow!
The new sign from ABCR FriUndus member needs to de (Holdaround) as GPL support by Jim IM being joined all in part) been mostly worked on for the better half a year.
I think it would be useful if we can shake up the challenge a bit.
Using armor totems: Major
Gear-Charging noticing the misfamiliarity: Aprium might be related using schedule to charge all monsters Charge monsters using the charge Majority add names have proficiency of gold nodes in tanks and length of output power Core functionality are shown
https//wwww.stattrr.com
Apriim/
have crack at tech only for at least 45 seconds Beware of spiders to stray us, let's play meta yourself?
(/endoftext]))
or prefers an easy setting place various pros and cons for the purposes of sharing ideas with the Snet music. leaders can't It handle that curation either, they like merge to delete the items rolled and procs into requirements. Since very few people have one formed (ormed((endoftett]) endoftext))
Figure 9:
A sample generated by MDLM trained on OWT with 1,024 sampling steps. Topic shifts indicate
abrupt, unrelated subject changes within single passage. Incoherent sequences indicate spans where sen- tence structure collapses into meaningless token concatenation.
26

---

it, like someone Published as a conference paper at ICLR 2026
(Jendoftext) ) off people for the first time at that point, somehow do not prolonging your woes. Secondly, it is never easy to know the culture's motives. is also very easy to get a firm sense of what you should offer anyone if you do not know what their motives are. Sure, you don't oo always get the feeling that being wrong is the answer to the issue. But, will work against you. You understand what they were and they have answered the question. Probably the most widely used saying is that was 'after those misery and misery' BS and it is still a bad choice. There is only a single good option. Who is to target if they go bad over, why go to with people in black clothing.
Some of the approaches that harm the environment are incredible, but a deeper level, it is not true without the success of anything useful afterwards. But first, it helps to consider what our choices actually are. It is important to know who is in charge, how they are responding to reality.
Have you ever got transition presentation?
In case all you are studied making this question already, I can see myself suggesting you bring someone or woman to the business club. needs to be quite informed and clear, then there so much work ahead that it is fully satisfied just the agenda of bureaucrats in that evil Pigguonious tibboleth. Can you relate to the students at all as almost everyone else? Certainly making a decision consists of nothing more than time. That just allows you to be reassured that you never know them better. Not aware of their meaning and abilities. Not letting things go before it starts all over in some other place too That's a mistake. imagine bringing a person away would end the conversation. You just can't do that. Give in your data and give yourself platform to assess what means before you decide to mislead. Once you do, the comparator's 's problem for at least two months is the use of the ever changing strategy as an excuse for new situations. Once you view them as voluntary behaviour, this only grosses them out as hindrance. Shining the Spear
There wasn't t enough time to consider another question of bearings, these are ideas, opposed to necessarily things that matter. What that opposing about them that they totally matter. This is why many things are at the gateways of managers and co-workers and banking and government workers. Considered or non-working ideas can exist. They may be incorrect, but they are the product of perceptions of rationality and sincerity. No matter if they are wedged together and land somewhere between good guys because you could get op-room in exchange for playing your part, you still get it. Managers trying to protect the idea usually resign someone who opposes it. Do they do good for them than those who do? Or do they actually on better chances than those who oppose they ideas.
This was people. If anything, this is a stranger problem with people. Good reason is all there is to have. But, my question? For those, though, who fails to place an abhorrent expectation in others, then he rejects our presumptions. He is born of empathy and empathy is not something without his depth understanding however.
Suppose have kid who thinks our culture must be equal, that is, because entrepreneurs are liberation activists so whoever controls the culture, in order to be less oppressive, will go into striving for another form. Muscle is not thing that lead to power. It can make democracy. It can make a person who is different from people. Invisible people can grow power. People have the muscle to exercise that power. There is inner strength. People can be potent force against people that can't 't consciously attain strength. Person won. Some people. However, it is not something we have to choose. This is the opportunity. The opportunity is good. Yet it is easy being unable to solve puzzle.
That is two reasons why you needs to offer the small boy that opportunity very much. It is only distraction and to have to think wouldn't 't use distraction against oneself. Failure to think is the reason we take autonomous things away. You cannot have the will live, freedom to do what we choose or make this. Even so we' re tough at it. We haven't 't got the resources for your mind or your quest.
You can't. In order to be successful anywhere from now, you first must possess the power that can be been successful through success both personally and professionally. Even in the first few weeks, there are times when there is strong foree in your life who comes with many options or goals as you are comfortable with. know that you are always jealous when he introduces your well-carned stuff to friends or (endoftext))
Figure 10: A sample generated by UDLM trained on OWT with 1.024 sampling steps. Topic shifts indicate abrupt subject changes. Unlike MDLM, UDLM produces grammatically plausible sentences, but the passage still lacks topical coherence across paragraphs.
27

---

Published as a conference paper at ICLR 2026
(/endoftext) There are charges of battery and assault against Smith and is expected to be arrested at 7pm Mass. Thursday.Thoss who are on should contact BanTTheBoosoo.comm 5574. (574.(/endolle (/enooftext] ) As the Supreme Court is expected to rule next week over the Foreign Intelligence Surveillance Court's 's surveillance program to dig up the phone records of citizens and the Internet- based services they use without warrants, the Obama administration says it's close to the case, due to its broad purview.
800-707-
"The FISA Court has ruled, as it always has, broad executive searches, which violate First Amend- ment constitutional claims and rely on just a show of cause," said Malcolm Poena, senior counsel for intelligence services at the White House.
"I urge people to reconsider the decision and pursued more vigorously the FISC Court, and perhaps of greater importance, in Congress."
For nearly two years, the appellate court has ruled that intelligence officials are not allowed to gather phone records about Americans or foreign citizens without court or employer request, but also without showing-can only if critical telecommunications infrastructure "contains threat to the public."
warrants, and that the government can't 't ask for private stored information
"It is not clear why the administration should use DoMet before curtailing companies that honor terms for the surveillance against foreign targets," its predecessor, Marcy Wheeler of San Edward, said in statement last week.
Dennis Romero, founder founder of Public Knowledge, said that the legality of such requests is an ongoing issue because of different interpretation of the law, and that his organization is trying to draw distinctions between firms culpable and not culpable.
"A service to consumers is not always that of several firms," he said. "It is our call that the president lift orders requiring that. Likewise, is encouraged by orders enacted by previous administrations to reconsider this interpretation and to seek the full implications of the decision and to bring the case to the Supreme Court." "
Contact us at editors @ @twme.tcm...m.......oo (lendoftext)) Our Planet's Eclel now joins the fragmented bluster of the Orion cluster.
Scientists' 've mapped Earth's great primate cluster that once populated it throwing victory decades of thinking that ancient reptiles and gibes were born.
The Eclel originally formed by series of young animals losing their home, dropping off eggs on dying moors between North's 's Centaurus constellation and South's 's Adu constellation. The more than 280 million-year-ae to new location, the 'eureka' as it is now known.
(62.5 billion y years ago) reptiles set out on the austere journeys
Eventually they realised the shipshape was no longer safe, so they abandoned the colonies. "We have always wanted to get picture of the last nursery location and whether it flowed into another," said co-author Calvei Richmond of the Atriemia de Investigadamento e Cholsa de Brazil. "We are glad to potheses."
map does seem to overturn longstanding fossil record and traditional hy-
Richmond has spent year probing the Eel via three satellites on the ground but detailed map showing the cluster's 's history has been elusive.
The paper map uses observations from Brazil's '' MARS Digital Surveys EPOS Landgrab Connector, which has been taking off from around the globe.
The study appears in the high order journal 'Nature'.
Reference: 2012-03-10 DOI: 10. 10. 1688//paid.immbb 1rrob.2022.10(enno/fiii. Jamout Bling!
Jamout Bling one of the most popular, month monthly OrganConner MO. In the live group-match MOBA, players will have to compete against a variety of "smoothingles"" to attempt prizes. At least have us right, Jamout Bling is solid fun with a great opportunity to make name for itself. What is Jamout Bling?
Jamout Bling is a small community that operates primarily by hosting prepaid prize pools of ap- proaching 3,000, and the team made it easy for you to have fun with other Blayers. Entry fee is $300, and get ready!
Why jam our house with the community! Jamout Bling has run for about six months by a group of folks living within a block of Willow Creek! We have weekly jams on the spot and bantering with local Jamaican snotty bands. When's the Jam?
Until A Friday 10am!
As much as you want to have fun, come out! It's a very small crowd, bring your critters too, and we'll be hosting a dartboard, so just put your controller on if you have on. on. ((endofftxt))
Figure 11: A sample generated by the LDDM-M trained on the OWT dataset, using 1.024 sampling steps.
28

---

Published as a conference paper at ICLR 2026
[Jendoftext] OOne of the most enduring stories that will eventually lead us to this point is the "Oil War." Between 1992 and early 2000, the United States proved to be China's 's largest central defense partner, having currency turnover of over $100 billion. Astonishing America's s growing closer to Iran and Russia, China to turn away from Iran and relinquish its oil reserves in the Middle East. started blocking deals with the United States. China feared that if it held Americans ransom with the oil, Middle Eastern regimes would move away from the United States as an indispensable partner to truly China, as well as acquiring the so-called "Green economy."
If story went more like this, Iran might not even be surprised by this grand strategy. Instead, it decided respond by physically seizing some of the vast oil reserves north of the border, near Saudi Arabia near Basra. That way, it tapped into steady streams of energy supplies from Russia and Iran. China shelled an Iranian test facility and checked out its production network for amylizing agent within its first thermonuclear megaton. Ultimately its sole concern was to hold on oil.
The price for all the Iranians' oil misadventures was massive American imports. Today, E.G.A.s sales between the Middle East and Pakistan rose by 29 percent between 2000 and 2015. Ten years ago, the cost of America's 's largest drone was under single-digit dollars. Today, America's largest Predator drone costs whopping S32 billion. Imagine the consequences of Reagan's 's wars, such as the encroachment of the Soviet Union in the 1980s, and his successor's s wars as well Korea, Vietnam, and especially East Vietnam. Likewise, oil alone is no longer source of revenue but a big driver of investment to U.S. companies. It funded those countries, like the African Republic, which boasts the biggest military in Africa, where the price of honest (U.S). oil imports was about the price of oranges in 2000. Today, the U.S. trade on oil exports is close to $180 billion.
America's growing dependence on oil in the South Pacific, as well as all of Southeast Asia, need to grow even more as technology advances. "The core challenge to secure a balance to sustain growing liabilities." wrote Vern van den Walk, a senior technology analyst at Fortune Global Trading Service, Fortune's 's analytical unit located in London, in a note he wrote. "Not so much dictated by technology but will the name of the game." "
The huge challenge of cybersecurity will soon prove to be an important factor in the partnership's longevity, as well as the longer term consequences. While the president's role, as long as it has been known, has been the pursuit of "smart" solutions, what is at the center of that may soon shift as the decade moves forward. Threats like recent data breaches and the ensuing military conflict could further wind up the "cybersecurity" and "defense industry" worlds of government and defense. Dalea McGovern, a CTO at the upstart Lockheed Martin, designs the F-35 Multi-Role Vehicle. (Reuters)
It does not take a bit of imagination to help imagine the likely scenario down the road. Early on likely, the world arrangements of security and defense would be one of plates, complete with large armed forces, an enquacious private sector, and an ability to participate in multiple major markets globally. Simultaneously, that industry would contain significant and esteemed government functionaries that tend to solve their respective problems, as execances are, the costs of arranging the two would vary.
Between 2004 and 2015, military agencies handled roughly a gazillion Pentagon data requests/day. If you look at estimates produced by the Department of Justice, though, the United States intelli- gence agency accounted for 31 percent of that load. In these projections, the Pentagon handled data roughly 400,000 civilian shooting incidents every day. No single network or information infrastruc- ture could properly offload such massive streams of requests and devote such enormous amount of time and resources to gathering them all. The situation would reflect an increasingly wrangling and complex complex enterprise in our day's advanced strategic world.
Part of that potential comes from how inelastic things are now. Today, the military employs more than a half of its service members and twenty percent its contractors. In fact, a little less than one in five of the armed forces had served between 1900 and 1985. Before that, about couple of out of five men fought in the Civil War. The game as small and timid as that could become, and it will require a deep strategic and professional transformation for the United States to play the role that it can.
The Bad Most people don't imagine, but( out(//enoft
Figure 12: A sample generated by the LDDM-U trained on the OWT dataset, using 1,024 sampling steps.
29


Reviews:
[{"review_id": "rGOgD3dAJu", "reviewer": "Reviewer_dHL3", "summary": "The paper introduces Loopholing Discrete Diffusion Models (LDDMs), which modify discrete diffusion LMs by adding a deterministic latent pathway (h_t) that is passed between denoising steps, in addition to the standard sampled token outputs. The idea is to “bypass the sampling wall,” defined as loss of distributional information once you collapse logits into a one-hot token during sampling. The paper also proposes a two-pass self-conditioning scheme to train this recurrent latent path without having to unroll time.\nThe authors claim:\n\nlower perplexity than prior discrete diffusion baselines such as MDLM and UDLM on LM1B and OpenWebText,\n\nlower “generative perplexity” (measured by GPT-2) and higher GPT-4.1 “consistency”/“naturalness” scores for unconditional generations,\n\nand improved success rates on small arithmetic reasoning tasks like Countdown and Game of 24 when applied to MGDM.", "questions": "* Novelty vs self-conditioning: How does loopholing differ from existing self-conditioning techniques beyond storing an explicit latent ?\n* Please include an ablation where standard self-conditioning is applied to MDLM without your recurrent latent path, to show what the new component adds.\n\n* Isolation of latent path effect:\nCan you run controlled ablations for:\n(a) no recurrent latent but with self-conditioning;\n(b) recurrent latent but single-pass training;\n(c) caching and reusing previous x_{\\theta, t} instead of h_t? \n\t​This would clarify which mechanism drives the improvements.\n\n* Task breadth: \nThe current benchmarks (LM1B, OWT, Countdown, Game of 24) are short and low-complexity.\nCan you evaluate on actual generation tasks such as summarization (CNN/DailyMail, XSum), long-form reasoning (GSM8K, MATH), or instruction-following datasets?\nWithout these, it’s unclear if LDDM meaningfully improves text generation quality.\n\n* Perplexity vs real performance:\nSince the evaluation focuses on approximate NELBO perplexity and GPT-based metrics, can you show concrete qualitative examples or human evaluations demonstrating that loopholing produces better text (fluency, coherence, factuality) than MDLM or AR baselines?\n\n* Efficiency claim:\nCan you provide real compute metrics (tokens/sec, FLOPs/sample, latency for 1k tokens) to support the “higher efficiency” claim?\nDoes LDDM actually require fewer diffusion steps at equivalent output quality?\n\n* Baseline fairness:\nAre the MDLM and AR baselines trained with equal data, compute, and precision settings?\nHow sensitive are the reported improvements to these factors?\n\n* Robustness of evaluation modifications:\nFor MGDM, your evaluation differs from Ye et al. (2024). Can you report results under the original evaluation scheme for transparency?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "* The paper articulates a clear failure mode of discrete diffusion LMs: once you sample a discrete token, you throw away the richer distributional state and force the next step to start from a one-hot. They call this the “sampling wall.” \n\n* The proposed “loopholing” mechanism - passing a continuous latent h_t forward alongside the sampled token sequence — is architecturally straightforward and appealing for implementation. \n\n* LDDM-M beats MDLM on perplexity on LM1B and OWT (e.g. ≤25.95 vs ≤27.60 on LM1B; ≤21.90 vs ≤23.05 on OWT). \n* On unconditional generation, “Gen PPL” improves a lot (e.g. 49.13 vs 108.94) and GPT-4.1 gives higher fluency/consistency scores. \n* On arithmetic reasoning puzzles, LDDM-G improves success rates vs MGDM (e.g. Countdown4: 86.5→94.4% at 85M params).", "weaknesses": "* Evaluation is narrow and relies only on perplexity metric.\nThe paper repeatedly claims “closing the autoregressive gap,” but this is based entirely on approximate perplexity (NELBO upper bounds) rather than true log-likelihood or downstream task performance.\nThere is no evaluation on actual generative tasks such as summarization, dialogue, open-ended story writing, or reasoning. Consequently, we cannot conclude that the method improves generation quality. It is important to evaluate the models on actual dowsntream performances, some of the current widely used benchmakrs for evaluation of the large scale language models are summarization tasks, MMLU, MMLU-Pro, Math-500, GSM8k, and reasoning benchmarks, and code benchmarks like Humaneval, MBPP, ... \n \n* Limited novelty beyond standard self-conditioning.\nThe proposed “loopholing” mechanism - passing a continuous latent state between denoising steps trained via a two-pass stop-gradient trick — is almost identical to self-conditioning already used in diffusion and consistency models (e.g. Jabri et al., 2022; Chen et al., 2022).\nThe only real difference is that LDDM explicitly names and propagates a latent vector rather than the previous clean prediction. This is a minor architectural variant, not a new principle.\n\n* The “sampling wall” is largely a renamed version of known problems.\nWhat the paper calls the “sampling wall” (loss of information when discretizing predictions) is conceptually the same as the “idle steps” and “oscillation” problems already identified in earlier discrete diffusion work (e.g., MDLM, SSD-LM).\nThe paper renames these issues and overstates their novelty, without providing direct causal evidence that loopholing specifically resolves them.\n\n* Causal explanation lacks experimental proof.\nThe claim that the propagated latent  “bypasses” the sampling wall is asserted, not demonstrated.\nThere is no ablation isolating: the effect of the latent path from the effect of self-conditioning, or the effect of feeding forward x_{theta, t} versus h_t\n\t​\n* The observed differences could easily stem from generic stabilization due to the two-pass stop-gradient process.\n\n* Weak baseline fairness and inherited limitations from MDLM.\nLDDM inherits MDLM’s fairness issues:\nPerplexity is computed as an upper bound, which is not directly comparable to autoregressive likelihoods.\n\n\n* Trivial or overly simple benchmarks.\nLanguage modeling experiments are limited to LM1B and OpenWebText at relatively small scale.\nThe “reasoning” benchmarks (Countdown, Game of 24) involve symbolic arithmetic and tiny vocabularies.\nThese tasks do not capture the complexities of realistic generation (e.g., global coherence, factual reasoning, long context).\nThe method is not tested on any challenging downstream generation benchmarks (summarization, QA, GSM8K, MATH, etc.).\n\n* Efficiency and compute cost not substantiated.\nThe paper claims fewer idle steps and higher efficiency, but provides no wall-clock, FLOP, or token-throughput comparisons to MDLM or UDLM.\nTraining is actually ~30% slower due to doubled embeddings and latent propagation overhead.\nThere is no evidence that LDDM improves quality per unit of compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hLjb91m45u", "reviewer": "Reviewer_2QKi", "summary": "This paper shows strong improvements in generation and reasoning accuracy of discrete diffusion models by allowing each diffusion step to condition on the pre-logit hidden states of the previous step.  This avoids the problem of losing information in the discrete sampling step, despite the fact that these hidden states are not trained for this purpose.\n\nThis problem is fundamentally the same as the problem with backprop across a discrete sampling step, and the intuition for solving it is essentially the same as the intuition behind Gumbel softmax (i.e. mixing the complete predicted distribution with the sampled one).  But the authors identify the forward direction of this problem as a separate issue independent of the backward direction, and demonstrate its importance for diffusion models.  They also propose a practical architecture which implements this intuition in an efficient way, by conditioning on the pre-logit hidden states (mixing these pre-sampling embeddings with the embeddings from the sampled tokens) and not backproping error across this conditioning.", "questions": "Please explain if I have misunderstood the argument about Idle Steps and Excessive Oscillation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The model and its motivation are well explained.  The empirical results are strong, thorough and convincing, and include the obvious ablations.\n\nUsing diffusion for non-autoregressive generation of text is an important open problem, and this paper demonstrates a method for closing the gap in accuracy with auto-regressive models.  It also shows how iterative methods such as this can improve reasoning ability.", "weaknesses": "Identifying the problem of passing information forward across discrete sampling steps is not as novel as they claim, since it just the forward perspective on the well-known problems with backprop across a discrete sampling step.  But the method for addressing this intuition is novel, and it is novel to show that it applies even when the model has no backprop across this step.  It would have been interesting to see how this compares to a model which does backprop across this step, but this is mentioned as future work.\n\nEquation (2) is confusing because it uses the same variables to sometimes refer to the entire sequence and sometimes refer to individual tokens within the sequence.  I think it would be clearer to add explicit token indexes where appropriate.\n\nI wasn't convinced by the discussion of Idle Steps, since the graphs are averages and thus do not show whether individual paths have idle steps.\n\nFor the claim in lines 360-361, please mention which graph backs up this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqL9T2S5e6", "reviewer": "Reviewer_Sjct", "summary": "This paper identifies a key limitation in discrete diffusion models, which the authors term the sampling wall. This refers to the information collapse that occurs when the rich, continuous categorical distributions predicted by the model are sampled, reducing them to one-hot vectors for the next denoising step. The authors hypothesize that this loss of distributional information is a root cause of known inefficiencies in discrete diffusion, such as \"idle steps\" and \"excessive oscillation\" .To address this, the paper introduces Loopholing, a mechanism that creates a deterministic latent pathway to complement the standard stochastic sampling path. The resulting models, Loopholing Discrete Diffusion Models (LDDMs), are evaluated on language modeling and arithmetic reasoning. Experiments show that LDDMs significantly improve upon baseline models, achieving up to a 61% reduction in PPL. It also improves performance on reasoning tasks like Countdown and Game of 24.", "questions": "The paper modified the TopK decoding of MGDM to only apply to masked tokens. However, Table 4 in the appendix shows that the original MGDM decoding actually performs better for both the baseline and LDDM-G. Why does this approach that ‘conflicts with the training objective’ achieve higher performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "a. The proposed Loopholing mechanism is a simple and logical solution to the stated problem. Propagating the continuous latent state $h_s$ is a direct way to preserve information, and the architectural modification is relatively minor but highly effective.\n\nb. The authors successfully apply Loopholing to two different families of diffusion models (Masked Diffusion and Uniform Diffusion) and two different domains (open-ended language modeling and structured arithmetic reasoning ). This suggests that Loopholing is a general and widely applicable technique.\n\nc. The analysis of Temporal KL (TKL) and Token-Prediction Entropy (TPE) in Figure 5 compellingly supports the hypothesis that Loopholing mitigates oscillations and idle steps.\n\nd. Loopholling mechanism surpass of its discrete diffusion backbone even under the same computing budget.", "weaknesses": "a. While the results are excellent at this scale, it remains an open question whether these substantial relative gains will persist with much larger models.\n\nb. The paper notes that an attempt to apply Loopholing only during fine-tuning was unsuccessful. This limits the capability to improve existing diffusion models. Is it possible to continue pre-train with an existing diffusion model with the loopholing mechanism?\n\nc. Loopholing yields strong improvements for MDLM but marginal gains for UDLM (Table 2), except on PTB. The authors attribute this to domain shift sensitivity in UDLM perplexity, but this undermines the generality of the proposed mechanism. Are there any other metrics can be used to evaluate the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)