Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall
Authors: name: Mingyu Jo, name: Jaesik Yoon, name: Justin Deschenaux, name: Caglar Gulcehre, name: Sungjin Ahn

Discrete diffusion models offer a promising alternative to autoregressive
generation through parallel decoding, but they suffer from a sampling wall:
once categorical sampling occurs, rich distributional information collapses
into one-hot vectors and cannot be propagated across steps, forcing subsequent
steps to operate with limited information. To mitigate this problem, we
introduce Loopholing, a novel and simple mechanism that preserves this
information via a deterministic latent pathway, leading to Loopholing Discrete
Diffusion Models (LDDMs). Trained efficiently with a self-conditioning
strategy, LDDMs achieve substantial gains-reducing generative perplexity by up
to 61% over prior baselines, closing (and in some cases surpassing) the gap
with autoregressive models, and producing more coherent text. Applied to
reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such
as Countdown and Game of 24. These results also indicate that loopholing
mitigates idle steps and oscillations, providing a scalable path toward
high-quality non-autoregressive text generation.