ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
Authors: Jia-Nan Li
Jian Guan
Wei Wu
Chongxuan Li
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion
autoregressive
large language model
Primary area: foundation or frontier models, including LLMs

Abstract: Autoregressive models (ARMs) are hindered by slow sequential inference. While masked diffusion models (MDMs) offer a parallel alternative, they suffer from critical drawbacks: high computational overhead from precluding Key-Value (KV) caching, and incoherent generation arising from learning dependencies over an intractable space of token combinations. To address these limitations, we introduce ReFusion, a novel masked diffusion model that integrates sequence reorganization into the causal attention framework. By elevating parallel decoding from the token level to a higher slot level, ReFusion interleaves inter-slot diffusion-based selection with intra-slot autoregressive infilling, while reordering newly generated slots ahead of the remaining masks after each iteration. Consequently, this design simultaneously unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot-level permutation space. Extensive experiments on seven diverse benchmarks show that ReFusion not only overwhelmingly surpasses prior MDMs with a 34\% performance gain and an over 18$\times$ speedup on average, but also bridges the performance gap to strong ARMs while maintaining a 2.33$\times$ average speedup.

Published as a conference paper at ICLR 2026
REFUSION: A DIFFUSION LARGE LANGUAGE MODEL WITH PARALLEL AUTOREGRESSIVE DECODING
Jia-Nan 2, Jian Guan Wei Wurt Chongxuan Li
Gaoling School of Artificial Intelligence, Renmin University of China
Beijing Key Laboratory of Research on Large Models and Intelligent Governance Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE
4 Ant Group $ *Ant International
[1ij (iijaanae anan, chongxuan ruc. edu. cn
(jianguanthu, WLwei198503133 gma.i...111mm1mm1mmmmmmmm1.m.m.m com
Code:
Model:
ABSTRACT
Autoregressive models (ARMs) are hindered by slow sequential inference. While masked diffusion models (MDMs) offer a parallel alternative, they suffer from critical drawbacks: high computational overhead from precluding Key-Value (KV) caching, and incoherent generation arising from learning dependencies over an intractable space of token combinations. To address these limitations, we introduce REFUSION, novel masked diffusion model that integrates sequence reorganization into the causal attention framework. By elevating parallel decoding from the token level to a higher slot level, REFUSION interleaves inter-slot diffusion-basee selection with intra-slot autoregressive infilling, while reordering newly generated slots ahead of the remaining masks after each iteration. Consequently, this design simultaneously unlocks full KV cache reuse and reduces learning complexity from an intractable token combination space to a manageable slot-level permutation space. Extensive experiments on seven diverse benchmarks show that REFuSIon not only overwhelmingly surpasses prior MDMs with a 34% performance gain and an over 18x speedup on average, but also bridges the performance gap to strong ARMs while maintaining a 2.33 average speedup.
INTRODUCTION
While autoregressive models (ARMs) (Grattafiori et al. 2024 Yang al. 2025 Jaech al.
2024
have achieved remarkable progress across a wide range of tasks (Chen et al. [021 Wei et al. 2022
Lightman al. 2023 Li al., 2024), their inference throughput is fundamentally limited by
sequential, left-to-right decoding process that precludes parallelization (Chen et al. 2023 Cai et al.
2024 Zhang et al.) 2025). In contrast, masked diffusion models (MDMs) (Nic et al. 2025 Ye et
2025) operate via an iterative denoising process with no fixed generation order. This flexibility yields two significant advantages. First, it permits parallel decoding by assuming conditional independence among target tokens: their joint probability, given the context, is assumed to be the product of their individual marginal probabilities (Li al.) 2023). Second, offers the potential for the model to discover better generation orders than the rigid left-to-right trajectory (Kim et al.) 2025).
Despite these theoretical advantages, existing MDMs often suffer from two issues: (1) Architectural bottlenecks negate efficiency gains from parallelism. The flexibility of generation orders requires bidirectional attention in MDMs (Vaswani et al. 2017 Devlin et al. (201)), an architectural choice fundamentally incompatible with Key-Value (KV) caching used in ARMs (Radford et al. 201 8). That is, each decoding iteration forces a full re-computation of the KV states of the entire context, introducing substantial latency and making MDMs significantly slower than ARMs (Feng al.)
Equal contribution.
Corresponding authors: Wei Wu and Chongxuan Li.

---

Published as a conference paper at ICLR 2026
2025 (2) Intractable learning complexity hinders coherent parallel generation. MDMs typically decode multiple tokens with high marginal probabilities in parallel (Nie et al. 2025). However, the conditional independence assumption frequently fails for these tokens, particularly for nearby tokens, leading to severe incoherence (Huang et al. 2022 Luxembourg et al. 2025 Gwak et al., 2025). For example, in context where both "at once and "right now" are valid, an MDD might decode a spurious output "right once" by independently sampling tokens with high individual marginal probabilities but low joint probability. We attribute this failure to an immense learning challenge: modeling data distribution over an exponential space of possible token combinations is far more demanding than the fixed sequential dependency of ARMs. Consequently, current MDMs often remain undertrained for reliably identifying conditionally independent tokens.
To address these challenges, we introduce REFUSION, masked diffusion large language model leveraging sequence reorganization with causal attention framework. Specifically, we partition the masked sequence into fixed-length, consecutive uub-sequences, referred to as slots. During each decoding step, REFUSION performs diffusiion-bsse slot selection followed by parallel autoregressive infilling within these slots. Subsequently, the newly decoded slots are reordered to the front of the remaining masked slots (Figure B}. This design yields two critical benefits: (1) Full KV Cache Reuse: REFUSION seamlessly reuses the KV states of all previously generated tokens without sacrificing the MDM's flexible generation order. (2) Reduced Learning Complexity: By serializing adjacent tokens within a slot, REFUSION mitigates the conditional independence violations typical of MDMs. This significantly reduces the learning complexity, transforming an intractable token combination space into a manageable slot permutation space. Appendix A.1 compares REFUSION and existing MDMs in detail.
REFUSION's training mirrors its infer- ence dynamics. For each sequence. we randomly mask several slots, per- mute the clean slots, and reorder the input so that clean slots precede masked ones. The model is then opti- mized with a hybrid objective that cul- tivates its dual capabilities: an autore- gressive loss on the permuted clean slots for sequential generation, and a denoising loss on the masked slots for context-aware parallel reconstruction. Unlike traditional MDMs, which learn only from masked positions. this hy- brid objective supervises every token, boosting data efficiency.
A pass@ 1 (%)
Lower Throughput Higher Performance
Higher Throughput Higher Performance
20
10
ReFusion
ReFusion left-to-right
8B
-20
-10
Qwen3
4 Throughput (tokens/sce)
Dream B-Instruct---- -10
Dream DZF
1.7B
-20-
LLaDA D2F
-30
LLaDA-8BB-Instrue
Our extensive experiments on seven benchmarks spanning math, code gen- eration, and general-purpose under- standing and reasoning demonstrate that REFUSION decisively establishes new offteoofthh-aat for MDMs.
LLaDA w/ Fast-dLLM
Dream w/ LLL
Compared to LLaDA (Nie et al. 2025)
Figure I: cefformannc-throoughe trade-off on MBPP. We plot pass@1 1 (%) against throughput (tokens/sec), with both metrics calculated relative to the Qwen3-8B baseline the origin. The "REFUSION left-to-right" ablation forces serial decoding using the REFUSION model.
and Dream Ye et al. (025). RE-
FUSION achieves an average perfor- mance gain of 34% while being over 18x faster in throughput (tokens/sec). More strikingly, REFUSION consis- tently challenges and often surpasses strong ARMs. For instance, it outperforms Qwen3-8B (Yang al. 2025) on GSM8K (Cobbe et al. 2021) and MBPP (Austin et al. 202T) by 3.68 absolute points while being 2.33x faster on average. This dual advantage in both performance and speed is further illustrated in Figure [] where REFUSION (the red point) pushes the performance-efficiene boundary significantly towards the top-right quadrant. It significantly outperforms both the Qwen3-series (the blue line) and prior MDM- based methods, which are situated in the bottom-left or bottom-right regions, indicating that they lag behind in throughput, performance, both dimensions. Furthermore,
2

---

Published as a conference paper at ICLR 2026
controlled experiments confirm that these gains are driven by architectural and training innovations, rather than initialization or data advantages.
Our contributions are summarized as follows:
We propose REFUSION, a generative model integrating inter-slot parallel decoding with intra-slot autoregressive decoding, combining the strengths of autoregREssive and CEFFUION-basse modeling.
To the best of our knowledge, REFUSION is the first MDM that achieves full KV cache reuse of every decoded token, while maintaining global generation flexibility and tractable learning complexity.
Extensive experiments on seven diverse benchmarks show that REFUSION not only overwhelmingly surpasses all prior MDMs in both performance and speed, but also bridges the performance gap to ARMs while maintaining the efficiency advantage.
2 RELATED WORK
MDMs promise to outperform traditional ARMs by offering faster inference through parallel decoding and potentially superior solutions via flexible generation orders (Kim et al.) (025).. Recent MDMs such LLaDA (Nie et al.] 2025), the first open-source MDM trained from scratch, and Dream (Ye) et al. 2025). initialized from an ARM, have delivered performance on par with ARMs of equivalent scale across diverse tasks, establishing MDMs as a viable research direction.
Architectural Designs for Efficient MDMs. Standard MDMs' reliance on bidirectional attention precludes the use of KV caching. Recent work alleviates this bottleneck through three main strategies. The first strategy approximates KV cache reuse while retaining bidirectional attention. For instance. dLLM-Cache (Liu et al., 2025) reuses slow-changing KV states, while sparse-dLLM (Song et al. 2025) dynamically prunes non-critical KV states. The second strategy mixes bidirectional attention and causal attention. Models like BD3-LMs (Arriola et al. 2025) and Fast-dLLM (Wu et al. 2025a) partition the sequence into consecutive blocks, enforeing a left- roight order between blocks to enable KV cache reuse, while retaining parallel, bidirectional generation within each block. D2F (Wang) al., (025) further parallelizes the generation of succeeding blocks, although performance limited by the lack of inter-block lookahead attention. The final strategy leverages only causal attention, enabling an exact KV cache. Eso-LMs (Sahoo et al. 2025), for instance, dynamically reposition newly generated tokens ahead of masked ones at each step facilitate caching. However, this strategy introduces an intractable learning objective at a token-level permutation space, which hinders training and leads significant performance drops.
Decoding Strategies in MDMs. A crucial aspect of MDM inference is the strategy used to select which tokens to decode in parallel at each step. Existing approaches generally fall into two categories. The first class leverages confidence heuristics derived from the model's own distribution, such as top token probability (Nie et al., 2025), low entropy (Ben-Hamu et al.) 2025). and probability margins between top candidates (Kim et al. 2025). Some methods further retine these heuristics with position-aware weights and frequency- based calibration (Huang et al. 2025). While simple, these methods rely on the often-unreliable assumption that the model's 's confidence scores are perfectly calibrated (Wu et al. (025a).. The second class employs external models for verification, e.g., using small ARM to validate and extend the longest acceptable prefix (Hu et al.) 2025 Israel et al., 2025). or using dedicated reward models to guide generation (Gwak et al. 2025). Although effective, these approaches introduce the overhead of maintaining and querying a separate model. Unlike these methods, REFUSIOn adopts unified inference framework that benefits from the parallel efficiency of MDMs without sacrificing the quality assurance of ARMs, all within a single architecture.
Comparison with Block-based Diffusion. Notably, while REFUSION shares the concept of grouped processing units with block-based diffusion methods (Amiol et al., 2025), our slot design is funda- mentally distinct from the block across the following dimensions:
Motivation: Slots are designed to reduce learning complexity to ensure coherent generation, whereas blocks are primarily introduced to enable KV cache reuse.
Operation: Slots are parallel inter-slot and serial intra-slot. Blocks are exactly the opposite: serial inter-block and parallel intra-block.
3

---

Published as a conference paper at ICLR 2026
Characteristies: Slots in REFUSION support full KV cache reuse while maintaining global generation flexibility. Block-base methods, however, sacrifice this flexibility for a left-to-right inter- block schedule, and intra-block bidirectional attention precludes KV caching and risks incoherence.
Compatibility: Slots are hierarchically compatible with blocks, as they can be nested within each block, positioning REFUSION more generalizable framework.
PRELIMINARY
Autoregressive Models. ARMs are a prominent class of generative models that factorize the joint probability of a sequence x = (I1...., zz) by enforcing strict left-to-right conditional dependency using a causal attention mask. This structure leads next-token prediction objective, where the model parameters are optimized by minimizing the negative log- log-likelihood: - >E log Po (a d<i)). During inference, generation is inherently sequential process requiring T forward passes to produce a sequence of length T, resulting in latency that scales with the sequence length.
Masked Diffusion Models. MDMs represent another class of generative models that operate on
a maask-andddenoiss principle. During training, each sample zo = (#)) x1. ... .x6') is corrupted to xe by masking each token with a special token " [[MASK]" " with probability
( 1). The
model learns to reconstruct the original sequence by minimizing the objective: -f - 1(2) =
[MASK]) log Po (xr) 1z)) MDM inference proceeds by progressively generating tokens from fully masked sequence. It requires fewer forward passes than an ARM thanks to parallel decoding, but each pass is drastically more expensive due to its incompatibility with KV caching.
METHODOLOGY
4.1 SEQURNCE REORGANIZATION
To address the aforementioned limitations of MDMs, we propose two methods of sequence reorgani- zation: token reorder to simultaneously support full KV caching and globally flexible decoding; and slot partition to reduce learning complexity for more manageable objective.
Token Reorder for Full
Decode: Step 1-
Step 2
KV Cache Reuse. RE- FUSION adopts a standard causal attention similar
Position ID
3
Docode 1,3
3
Sequence
Step 3-
Result
traditional ARMs, while performing global position- flexible decoding during inference as in MDMs. Specifically, REFUSION moves newly decoded tokens the front of the remaining masked tokens while preserving their relative internal order (Figure Consequently, before each decoding step, all decoded tokens appear contiguously at the beginning, followed by the remaining masked positions. This layout enables the use of full KV cache every decoding step.
Decodo
Order
Attention:
Suep
Step 2
Skep
Clean
Masked
Decode
KV-enched
Figure 2: Illustration of full KV cache reuse via token reordering.
However, this strategy introduces a mismatch between the input positions of decoded tokens and their original positions in the correct sequence, which may adversely affect attention computation and semantic modeling. To address this, we compute attention using position IDs that correspond to each token's i original index, ensuring that attention scores remain invariant token reorder during decoding.
Taking the widely used Rotary Position Embedding (RoPE) (Su et al., 202T) as an example, the attention computation between query qm with position ID m and key ks with position ID is formulated as:
(Rukin) f(gm; ka) (R)
q Rr Ru-mkin knz
(1)
4

---

Published as a conference paper at ICLR 2026
Inference
Training
Peompt
Respotae
Proppi
Respone
slat leyat
linificationn
Stage Dilfesian based Siot Selection
Permutaaion of Claan
Slats
Prodiction
X
Stage II: Autoragrasivv Slat Infillig
Reoreer
Reerder
Pa
Fintficatioon
Caan
Stot (Mulli-token Sogment)
Token
Clean
KV-cachod
Donft
Masked
Rearder
Forward Pass
Figure 3: Overview REFUSION, Left (Inference): An iterative slot-level "select-and-infill" loop. diffusion stage selects and drafts slots, followed by parallel autoregressive verify-and-r preedcct infilling. Reordering generated slots before masked ones enables full KV cache reuse, while position IDs correspond their ground-truth indices, invariant reordering. Right (Training): Mirrors inference, optimizing a hybrid objective of autoregressive loss (LARM) on permuted clean slots and denoising loss (CMDM) on masked slots.
where the term Ra-m- which governs relative distance perception, remains invariant to token reorder. The aforementioned mechanism allows REFUSION achieve full KV cache reuse, significantly boosting generation efficiency while maintaining flexibility in the global decoding order.
Slot Partition for Reducing Learning Complexity, We further elevate modeling granularity by partitioning the token sequence into continuous, non-overlapping slots. The design of both inter- and intra-slot decoding orders is a key consideration. Prior studies (Luxembourg et al. 2025) suggest that conditional independence assumptions break down most severely among nearby tokens. Our pilot dependency analysis (Appendix A.2) further confirms that dependency strength decays rapidly with relative distance, particularly in denser contexts with lower masking ratios. This motivates hybrid decoding design: slots are generated in diffusion-like manner with global flexibility, while tokens within them are decoded autoregressively to capture strong local dependencies.
This design naturally supports full KV cache reuse. For inter-slot decoding, after each decoding step, newly decoded slots are moved in front of remaining masked slots while preserving their internal order, corresponding to applying the previously described reordering process at the slot level. Within each slot, standard autoregressive KV caching is directly utilized during token-level decoding.
We intentionally use the term slot to distinguish design from the well-known block partitioning in the literature. contrast to our slot dynamics, block-based methods operate with inverted logic: they employ inter-block left-to-right autoregressive decoding and intra-block parallel diffusion decoding. For a detailed discussion on other fundamental distinctions, please refer to the paragraph "Comparison with Block-based Diffusion" in
4.2 INFERENCE
Built upon the proposed sequence reorganization strategies, the inference process iteratively generates a final response ño given prompt Po. starting from an initial sequence 7 composed of masked tokens. During inference, the response is partitioned into K consecutive slots of length Each decoding iteration comprises two stages: diffusion-based slot selection and autoregressive slot infilling.
Stage Diffusion- based Slot Selection. This stage selects the next slots to decoded diffusion-Il like manner. At timestep t, defined the ratio of remaining masked slots, construct the input sequence St by concatenating already- -reeddd slots (Srlean, in their generation order) with the masked slots (Smaskedd in their original positional order).
The model then computes a certainty score for each masked slot based on its predictive distribution. While various scoring strategies exist, adopt simple yet effective one: the probability of the
5

---

Published as a conference paper at ICLR 2026
most likely token the slot's first position. An ablation of different scoring strategies is provided in Appendix C.2.
Finally, slots with scores exceeding threshold Tslor are selected for infilling. This strategy identifies slots that are highly predictable and suitable for parallel decoding given the current context. Further- more, inspired by speculative decoding (Leviathan et al., 2023), sample a corresponding draft Stfaat for these selected slots from the predictive distribution to accelerate the subsequent infilling stage.
Stage II: Autoregressive Slot Infilling. The second stage begins by efficiently verifying the draft slots Sdraat Specifically, all draft slots are concatenated into a sequence according their original positional order. The model performs single forward pass to compute token probabilities conditioned on the prompt and the already-decodee slots. We then identify the longest continuous prefix of this concatenated sequence where every token's 's probability exceeds a threshold Tioken If this prefix covers one or more complete slots, these slots are accepted in their entirety. The remaining unverified or partially verified draft slots are re- re-masked, and the process immediately proceeds to the next selection iteration.
Otherwise, we revert to parallel iterative completion process to handle the draft slots independently, following the spirit speculative decoding again. Each iteration consists of two steps: (i) Verification: we independently identify the longest valid prefix for each slot where conditional probabilities exceed Tioken: and (ii) Prediction: we retain these valid prefixes, re-mask the remaining suffixes, and concurrently predict the masked tokens using the model's MDM capability, conditioned the context and accepted prefixes. This cycle repeats until all selected slots are fully decoded.
Finally, the newly completed slots moved to the front of the remaining masked slots, following the reorganization in $4.1 Their KV caches are directly concatenated for future iterations. This serves valid approximation, as the lack of inter-slot conditioning during parallel generation has minimal impact on performance (see (C.3).
The select-and-infill cycle continues until all slots are completed, after which the response is con- structed by restoring the original slot order. This process is formalized in Appendix(A.3 (A.3]and illustrated in Figure 3 (left). For further clarity, step-by-step visualization is provided in Appendix D.2
4.3 TRAINING
The training procedure for REFUSION is carefully designed to mirror the dynamics of our two-stage decoding algorithm. This requires a data construction strategy that simulates the non-sequential, partially-decodes states encountered during generation, and a hybrid training objective that jointly optimizes the model's s selecting and infilling capabilities.
Training Data Construction. To simulate the partially decoded states encountered during iterative generation, we introduce a three-step strategy to construct training data from each prompt-response pair (Pa, ro). The response ro is first partitioned into sequence of K slots, S. [S] ......., SK each of size k. Then, a corrupted version S, is constructed given a masking ratio f ~ UO, 1) as follows: (1) Random slot masking. Analogous to token-level masking in traditional MDMs, we randomly select and mask |tK] slots from the original sequence Sy. Each selected slot is replaced with block of [MASK]" tokens. (2) Permutation of clean slots. Since the generation order of slots is dynamically determined, the model must learn to process context in any arbitrary permutation. To achieve this, we randomly permute the unmasked (clean) slots to form Selean, while keeping the original relative positions of the masked slots to form Stnasked (3) Reorder. The final training instance S, is assembled by concatenating the permuted clean slots followed by the masked slots.
Hybrid Training Objective. To empower our model with the dual capabilities of global selecting and local infilling, we propose a hybrid training objective that learns from every token in the sequence.
On one hand, the clean slots Stleam are trained with standard ARM loss for next token prediction:
|
k
)
1
gelean log Pe (15² Po. St
LARM = (pro (0.1)
(2)
(k 1) Selenn
/=2

---

Published as a conference paper at ICLR 2026
Table I: Zero-shot performance and throughput (TPS) comparison multiple benchmarks. Each model displays accuracy/pass I (top row) and throughput (TPS, bottom row). Within the MDM category, we highlight the best performance results in bold and underline the second best, An italic score in the ARM category signifies that it surpasses the best- best-performing MDM.
Model
MMLLUPrr ARC-C
GSM8K
MATH
GPQA
HvmanEval
MBPP
Avg.
Autoregressive Model
35.23
82.76
75.13
25.48
29.46
46.34
53.00
49.63
Alama-3-8B-Instruu
32.07
44.12
42.81
19.73
42.00
42.26
41.68
37.81
Qwen3-8B
90.36
81.96
83.28
87.80
63.80
73.36
31.42
42.78
31.20
30.11
30.43
30.95
30.07
32.42
Masked Diffusion Model
85,58
76.35
38.78
32.37
45.12
25.60
LLaDA-8B-Instruct
18.21
0.03
27.35
23.93
12.42
2.97
12.41
35.02
82.85
76.27
38.58
28.35
37.80
24.80
46.24
LLaDA w/ Fast-dLLM
39.81
0.86
73.07
52.23
17.54
62.52
37.19
40.46
22.84
84.13
39.04
23.68
31.25
36.59
35.20
38.96
LLaDA D2F
44.54
3.70
82.59
59.48
23.84
96.90
53.85
53.
40.05
88.31
76.42
46,60
30.36
56.71
50.40
55.55
Dream-7B-Instruet
15.98
0.06
20.30
18.99
1.81
3.51
1,23
8.84
40.36
86.86
75.82
36.76
31.25
56.10
10.60
48.25
Dream Fast-dLLM
47.18
1.42
61.49
58.24
22.96
49.73
19.55
37,22
23.82
85.92
41.62
29.98
30.13
50.00
$1.60
44,72
Dream w/ D2F
56.43
59.08
79.20
84.14
49.96
69.15
65.59
66.22
89.76
84.91
54.22
78.66
68.20
65.31
REFUSION
52.74
32.46
81.24
81.77
64.11
103.90
92.09
72.62
where o is the j-th token in the i-th clean slot, Seleann (,<()
is the prefix of the token in Sylean
On the other hand, the masked slots Smasked are trained with an MDM objective for denoising!
LMDM
(po.rr) tr,1(
log (o
po, Seleem Smasked ((E(()))
|
Smaskedd
where of the ground-truth token from original response corresponding to the j-th token the i-th slot of 5 5imasked The final training objective summation of the two losses, balanced by A:
C CarM + XLMDM-
(4)
Notably, this approach improves data efficiency, which contrasts with traditional MDMs where clean tokens only serve as context and provide no direct supervision.
Following prior work (Gong al. 2025 Ye al. 2025). initialize Po with off-the-shelf ARM backhone. Crucially, all tokens retain their original positional indices from ro throughout the training process. This allows the model to maintain awareness of the relative positions among all tokens, ensuring sequence coherence despite the shuffled input order. Figure (right) illustrates the training process.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Implementation Details. We initialize REFUSION from the Qwen3-8B checkpoint (Yang et al. 2025) and fine-tune for 4 epochs on a diverse 3.7M-sample dataset (~~1.22 tokens) covering mathematics, coding, and general mnstruction-follovinnn compatibility with block-based diffusion methods discussed $2 we define the block size as b.
For inference, leveraging the hierarchical
Implementation and hyperparameter details are provided in Appendix B.1 and B.2 respectively.
Our per-token normalization, E.Sppaaa implicitly includes the & weighting since Smmakkdd A tK, where K is the total number of slots.
7

---

Published as a conference paper at ICLR 2026
Evaluation Benchmarks and Metrics. To comprehensively evaluate REFUSION, test its performance on diverse benchmarks spanning: (1) purprose understanding and reasoning: MMLU-Pro (Wang et al. 2024) and ARC-C (Clark et al. 2018); (2) Mathematical and scientific problem-sol ving: GSM8K (Cobbe et al. 2021). MATH (Hendrycks et al. 2021), and GPQA (Rein
et 2024;; (3) Code generation: Human Eval (Chen et al. 2021) and MBPP (Austin et al. 2021 We
use pass for code generation and accuracy for the others. We further assess inference throughput terms of tokens decoded per second (TPS) with single A100 GPU and batch size of
Baselines. We evaluate REFUSION against three categories of baselines (1) ARMs: Llama-3-8B- Instruct @T Meta 2024) and Qwen3-8B (Yang et al. 2025). (2) MDMs: LLaDA-SB-Insttun (Nie)
et al. (025), ) and Dream-7BB IB-Instt (Ye et al. 2025). (3) Statc-of-the-ae MDM acceleration methods:
Fast-dLLM (Wu et al. 2025b) and D2F (Wang et 2025). We implement the baseline methods
based on their official hyperparameters
Table 2: Controlled comparison of models initialized from Qwen3-8B and trained on 120K subset.
Model
MMLU-Pro
ARC-C
GSM8K
MATH
GPQA
HumanEval
MBPP
Avg.
54.22
90.53
88.17
66.94
30.36
63.72
66.00
65.71
Qwen3-8B (Retrained)
31.16
29,46
30.69
31.58
28.49
30.52
30.62
30.36
38.27
86.43
80.06
44.94
0.67
42.07
39.40
47.41
LLaDA (Refrained)
1.89
0.02
9.73
13.92
0.21
2.57
1.31
4.24
26.11
78.24
83.55
47.32
29.02
59.15
46.80
52.88
BD3-LMs (Retrained)
16.82
2.19
16.35
16.90
2.82
15.59
14.09
12.11
42.14
84.81
80.74
51.78
31.70
70.12
58.20
59.93
REFUSION (Retrained)
40.56
29.01
53.96
77.42
46.65
58.40
67.35
53.34
Table Controlled comparison with Dream-7B IB-Istruct on its native Qwen2.5-7B backbone.
Model
MMLU-Pro
ARC-C
GSM8K
MATH
GPQA
HumanEval
MBPP
Avg.
40.05
88.31
76.42
46.60
30.36
$6.71
50.40
55.55
Dream-7B-Inster
15.98
0.06
20.30
18.99
1.81
3.51
1.23
8.84
35.25
83.11
80.21
46.36
29.02
68.90
61.60
57.78
REFUSION (Retrained)
76.02
53.38
107.83
139.55
102.29
106.89
98.04
97.71
5.2
MAIN RESULTS
The main results in highllgt two key findings: (1) REFUSION dominates all MDM baselines. RE.FUSIO consistently outperforms all MDM baselines in both performance and throughput (TPS) across all seven benchmarks, often by substantial margin. For instance, HumanEval, it achieves 78.66% pass@1 1, surpassing the next-best MDM -rream-BBIInstree While acceleration methods like Fast-dLLM and D2F improve throughput significant performance cost, they still fall short of REFUSION'S 's efficiency. Notably, on MBPP, REFUSION reaches 92.09 TPS, which is 1.4x faster than the next-fastest MDM (Dream w/ D2F). REFUSION thus delivers both state-of-the-arr performance and superior efficiency, establishing a new frontier for MDM (2) REFUSION challenges strong A RMs. More remarkably, REFUSION challenges and often surpasses strong ARMs. It delivers an average speedup of 2.33x over Qwen3- -BB across all tasks while exhibiting superior performance on several benchmarks. For instance, on GSM8K and MBPP, it outperforms Qwen3-8B by 3.68 absolute points. This demonstrates that our non-autoregressive approach can break the long-standing trade-off between the speed of MDMs and the quality of ARMs (Feng et al. 2025).
by nearly 22 absolute points.
5.3 CONTROLLED COMPARISON
To isolate the benefits of REFUSION from data or backbone advantages, we adopt a controlled approach: retraining reproducible baselines using unified settings, while retraining REFUSION to match non-open-source baselines.
We omit BD3-LMs due its limited scale (0.2B), but compare scaled-up version (8B) 5.3 "Appendix C.1 compares a broader set of models.
"Appendix analyzes model trade-off fronticrs.
8

---

Published as a conference paper at ICLR 2026
60
2.5
-61
40
50
A pass@i
20
0.0
A Throughour (TPS)
1-
pass
-2.5-
à puss& (%)
Thronghput (TPS)
A Throughput (TP5)
59
-5.0
0.6
0.9
1.0
0.5
1.0
16
Threshold Talet
Threshold Traken
Slot Size
Figure 4: Impact of key hyperparameters on MBPP (0-shot). The plots show the change in pass (%) and throughput (TPS) of REFUSION relative to Qwen3-8B (dashed lines at zero). When one parameter is varied, others are held at their default values (Tsto 0.9, Troken 0.3, k = 32). Yellow regions highlight the "sweet spot" where REFUSION surpasses the baseline in both metrics.
Reproducible Baselines. We conduct a controlled comparison using a 120K data subset randomly sampled from the full 3.7M dataset due to resource constraints. Specifically, we fine-tune Qwen3-8B, LLaDA, BD3-LM. and REFUSION for 10 epochs using their respective original objectives, with all models initialized from Qwen3-8B. This setup ensures that observed differences are attributable solely to the algorithm design. Appendix C.5 discusses the scaling properties of REFUSION regarding data size.
Results in Table confirm the architectural superiority of REFUSION. LLaDA suffers catastrophic performance collapse, and BD3-LMs lags behind REFUSION in both performance and speed. We note that the already highly-optimized Qwen3-8B baseline understandably degrades when retrained on our smaller, open-source dataset. However, under this controlled setting (with data advantages eliminated), REFUSION still outperforms it by ~~6 points on HumanEval while being 1.9x faster. This result robustly validates that REFUSION'S architectural innovations are the primary driver of its success, enabling effective learning even from limited data where standard MDMs fail.
Non-Open-Sourred Baseline. Furthermore, we conduct controlled comparison with Dream- In-truttut.. Since training code and data processing details are not open-sourced, we cannot retrain it Qwen3-8B. Instead, we train REFUSION variant on Dream's original Qwen2.5-7B backbone. It is crucial to note the significant disparity training resources: Dream benefits from massive pre-training (580B tokens, 146.5M samples) followed by SFT (1.8M samples), whereas our REFUSION variant is exclusively fine-tuned (3.7M samples) without any pre- to-trrnninng. Despite this disadvantage, Table shows that REFUSION still achieves 2.23% average performance gain and massive 11.05 speedup over Dream. REFUSION significantly excels on reasoning and coding tasks (GSM8K, HumanEval, MBPP). Its lower performance knowledge-intensive tasks (MMLU-PP U-oo,, ARC-C) is expected, as it skips the pre-training stage that Dream utilizes for knowledge injection. These results confirm that REFUSION'S architectural advantages are robust across different base models and training setups.
5.4 ANALYSIS OF HYPERPARAMETERS
We examine the key hyperparameters governing the performance-efficiencyy trade-off in REFUSION: the slot selection threshold Taket, the token acceptance threshold Tikens and the slot size k. The threshold Tslos controls the confidence for slot selection, and Tikenn governs the confidence for draft acceptance (infilling), while & defines the granularity of the generation unit. An analysis other hyperparameters is shown in Appendix C.7
As illustrated in Figure 4. these hyperparameters create a predictable trade-off. (1) Slot selection threshold 7slot Increasing Talor improves performance due to higher token reliability. However, throughput (TPS) exhibits a non-monotonic trend. Although a higher threshold reduces slot par- allelism, processing fewer slots mitigates synchronization overhead during the parallel iterative completion phase, which can potentially boost throughput.
(2) Token acceptance threshold Tiken Increasing Tooken reduces the number of accepted draft tokens per step, thereby lowering throughput.
"The block size is to 8, which the minimum block size used in the REFUSIOn experiments.

---

Published as a conference paper at ICLR 2026
Performance also follows non-monotonic trend. While higher thresholds enforce stricter verifi- cation, excessive strictness causes frequent truncation and regeneration. This can trap the model in local optima and degrade final performance. (3) Slot size k: A larger slot size enhances local coherence and allows more draft tokens to be accepted wholesale during verification. This leads simultaneous gains in both performance and speed. Collectively, these analyses reveal robust and wide "sweet spot," highlighted by the yellow shaded regions in Figure 4 where REFUSION consistently surpasses the Qwen3-8B baseline in both performance and throughput (TPS). This superior operating zone corresponds to slot selection threshold Tstor [0.5, 1.0], token acceptance threshold Tioken E [0.1, 0.9), and a slot size {8, 32).
5.5 CASE STUDY
Figure 3 provides qualitative under- standing of how REFUSION solves programming problem from the
Problem: Write a function to sum all amicable numbers from speeified number.
ReFusion:
def icab .uubers
MBPP benchmark, highlighting two key capabilities: (1) High degree of parallelism. The model frequently generates multiple slots concurrently. For instance, at iteration 8, it simulta- neously generates four separate slots, demonstrating its ability to exploit parallel decoding opportunities. Si- multaneously, within each slot, the model leverages speculative decoding to accept four tokens in single pass, significantly accelerating the genera- tion process. (2) Non-linear genera- tion order. The generation process markedly non-linear. For exam- ple, the model constructs the central "for" loop structure (iteration 5) be- fore initializing a local variable "sum =1" (iteration 6). This ability to se- lect and execute in parallel, non- monotonic fashion allows REFUSION to construct complex, structured code in a manner that both efficient and conceptually closer human problem-solving. Appendix D.I shows the results of baseline models on the same problem.
def
sun dsv
isors
(nun
1:
For i
in renge
nun
sum
SUI
nun
for
range
sun
sun
isars
sun
1=
and
sun
div Ssors
(sun
1
an
icable
SUT
return
icable
3n [n__edd
Figure 5: A case study of REFUSION generating a Python
function for MBPP problem (k 4, Taket 0.6, Tinken 0.3, = 16). The number in the top-left corner of each slot indicates the generation order, while the color intensity within each slot represents the generation time (darker indi- cates earlier).
6 CONCLUSION
In this work, we present REFUSION, a novel generative model that leverages sequence reorganization with a causal attention framework. By synergizing the strengths of diffusion-based selection and autoregressive infilling, it effectively addresses the long-standing efficiency and coherence challenges in traditional MDMs. This unique design enables full KV cache reuse alongside flexible generation orders, while making the training objective tractable by simplifying the combinatorial complexity of the generation space. Extensive evaluations across seven benchmarks show that REFUSIOn establishes new state of the art for MDMs. More strikingly, it bridges the performance gap to strong ARMs, often outperforming them while being significantly faster. Our work demonstrates that by structuring the parallel generation process, it is possible to achieve the throughput potential of MDMs without sacrificing generation quality. Future directions include further scaling of the model and data size, well as leveraging reinforcement learning to optimize the model's 's planning policy for complex, multi-step reasoning tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Beijing Major Science and Technology Project under Contract no. Z251 100008425002; the National Natural Science Foundation China (Nos. 62522609, 92470118); the Beijing Natural Science Foundation (No. L247030); Ant Group Research Intern Program; and the fund for building world-class universities (disciplines) of Renmin University of China.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our experimental results, we have open-sourced our training and inference code. The specific settings for training and testing are detailed in $5.1 and Appendix
REFERENCES
A Meta. Llama model card. 2024. URL
wot.:/t..ommmmmmeeeeeeee 1aw.iaaaaaaaammmm 1aaiaaaalamaaaa 1lama3/
b blob/ma woob/main//nna i//OOE,,, BORRODEE_AAAD......
CARD
md
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíčck, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL httph:tt///aaatt http://awxxvvv..gaab// abss 2502 02737
/
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. UtLhttps: /twp/vaaivvv/v.////////a.//a.ava org/ abs 2503. 09573
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint (00v::2008.07332. 2021.
Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. arXiv preprint arXiv:2505.24857. 2025.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptise Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (.X.v:22077033777 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint
2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10.:22100.444 14168, 2021.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 2023.
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Ming-Wei
Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang. and Di He. Theoretical benefit and limitation of diffusion language model. arXiv preprint Axii:2552.006222 2025.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi. Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025. URL
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXivv2407.21783, 2024.
Daehoon Gwak, Minseo Jung, Junwoo Park, Minho Park, ChaeHun Park, Junha Hyung, and Jaegul Choo. Reward-weighted sampling: Enhancing non-autoregressive characteristics in masked diffusion llms. arXiv preprint wwXv:2500.00777,, 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint ar 033.03874, 2021.
Zhanqiu Hu, Jian Meng. Yash Akhauri, Mohamed S Abdelfattah, Jae-sun Sco, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided 2467, 2025.
diffusion. arXiv preprint
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, and Minlic Huang. On the learning of non non-autoregressive transformers. In International conference machine learning, pp. 9356-9376. PMLR, 2022.
Pengcheng Huang, Shuhao Liu, Zhenghao Liu, Yukun Yan, Shuo Wang. Zulong Chen, and Tong Xiao. Pc-sampler: Position-aware calibration of decoding bias in masked diffusion models. arXiv preprint arXiv:2508. 1...::2508.132222 13021, 2025.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models. 2024. URL httpppppp//p.....//pppptttta/...
04905
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion 1lms via adaptive parallel decoding. arXiv preprini arXiv:2506.0000 004306 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 2.N.:224/2.1672000 2024.
Jacyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768. 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML 23.. JML..org, 2023.
12

---

Published as a conference paper at ICLR 2026
Jia-Nan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, and Rui Yan. Streamingdialogue: Prolonged dialogue learning via long context compression with minimal losses. Advances in Neural Information Processing Systems, 11886774-810011 2024.
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. Diffusion models for non-autoregressive text generation: A survey, arXiv preprint arXiv:2303.0657/ 2023.
Hunter Lightman. Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint
arXiv:2305.200055 2023.
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang. and Linfeng Zhang. dilm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprini NXXY::2006002995. 2025.
Omer Luxembourg, Haim Permuter, and Eliya Nachmani. Plan for speed-dilated scheduling for masked diffusion language models. arXiv preprint arXiv:2506. 19037, 2025.
Christopher Manning and Hinrich Schutze. Foundations of statistical natural language processing. MIT press, 1999.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprini arXiv:2502.09992 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: graduate-leve google-pree proof q&a benchmark. In First Conference Language Modeling, 2024.
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models. arXiv preprint arXiv:2506.00 000.02220000000288, 2025.
Yuerong Song. Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction. arXiv preprint urXiv:2508.02555. 2025.
Jianlin Su, Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. 2017. URL https.//pproeeednns.s. htwp://cweeeiinn..eeeee......... www.:nuuuiu..cppppeepc
fww.ffilepppoo ftles//ppper/2117
Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion llms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.0919 09192, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 2024.
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 01124424-248333 2022.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion Ulm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.226115 2025a.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion 1lm by enabling kv cache and parallel decoding, 2025b.
org/ /tbp//55555 22618
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue. Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang. Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL / /ttp::///aavvv..v/a org abs 2505 09388
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao. Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508, 15487, 2025.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, arXiv preprint arXiv:2309.05653, 2023.
Lingzhe Zhang, Liancheng Fang. Chiming Duan, Minghua He, Leyi Pan, Pei Xiao, Shiyu Huang, Yunpeng Zhai, Xuming Hu, Philip Yu, et al. survey on parallel text generation: From parallel decoding to diffusion language models. arXiv preprint arXiv:2508.08771: 2025.
14

---

Published as a conference paper at ICLR 2026
Table 4: Comparison between REFUSION and existing MDMs. I. denotes the generation length and k denotes the block or slot size.
Generation
Attention Mechanism
Generation Order
Full KV
Number of Distinct Masking Patterns
Model
Scope
Cache Reuse
LLaDA
Full Sequence
Bidirectional
Any-Order
X
Ein (1) = 24
Intra-block
Bidirectional
Any-Order
BD3-LMs
X
2 k
Inter-block
Causal
Leighto-Right
Intra-slot
Causal
Left-to-Right
Ete ('14t) it = (( ee -1
REFUSION
Inter-slot
Causal
Any-Order
(1(2)) -cl * 24 for large A:)
A METHODOLOGICAL DETAILS
A.1 COMPARISON BETWEEN REFUSION AND REPRESENTATIVE MDMS
Table 4 provides a detailed, side-by-side comparison of the architectural and methodological designs of REFUSION against two representative MDMs, LLaDA (Nie et al. 2025) and BD3-LMs (Arriola al. 2025). This comparison highlights how REFUSION uniquely addresses the fundamental trade-off's offf between generation flexibility, computational efficiency, and learning complexity.
(1) LLaDA, as conventional MDM, operates the entire sequence with bidirectional attention mechanism. This grants it maximum flexibility, allowing for fully unconstrained, any-order genera- tion process. However. this design choice incurs two significant penalties. First, the bidirectional attention is fundamentally incompatible with KV caching, resulting in substantial computational overhead at each decoding step. Second, it must learn dependencies across an exponential space of possible masking patterns. For sequence of length L, any given training or inference state is defined by a subset tokens that remain masked. Since each of the L positions can be either masked or unmasked, the model must, in principle, handle any of the 24 possible subsets of visible contex 6 This combinatorial space of approximately 2 distinct masking patterns presents an intractable objective, the model may not be sufficiently trained on the specific patterns encountered during inference, leading to incoherent parallel generation.
(2) BD3-LMs attempts to mitigate these issues with hybrid, block-based approach. It enforces rigid, left-to-right generation order between blocks, which enables KV cache reuse across block boundaries. However, within each block, it retains bidirectional attention and any-order token generation. This design makes a critical compromise. It sacrifices global generation flexibility, hindering the discovery of optimal generation strategies, which is key theoretical advantage MDMs. Furthermore, it still faces the challenges of token-level incoherence and the inability to utilize KV caching for decoding.
(3) REFUSION introduces a more elegant and unified solution. Generation is structured at the slot level. Within each slot (intra-slot), generation is autoregressive (left-to-right) under a causal attention mask, directly addressing the strong local dependencies between adjacent tokens. Between slots (inter-slot), the model retains the flexibility of any-order generation, enabling to discover better, non-linear generation paths than the left-to-right order. Crucially, by reordering generated slots to always precede masked ones in the input sequence, REFUSION enables full KV cache reuse for every decoded token, feature unique among these models. This design simultancously achieves two critical goals: combines global generation flexibility with universal computational efficiency, and drastically reduces the learning complexity from exponential token-level permutation space to far more manageable slot-level one ( e] 1). For a typical sequence length of L = 4, 096, a
slot size of just l: 8 is sufficient to ensure L( ! e] - < <4.
Nottbbyy due to the hidirectional attention, the model is invariant to the order in which clean tokens are revealed. Therefore, the learning complexity is permutations (T.I).
15

---

Published as a conference paper at ICLR 2026
In summary, while prior models are forced trade flexibility for efficiency or vice REFUSION's innovative clot-based framework is the only approach that concurrently offers global any-order generation, full KV cache reuse, and a tractable training objective.
A.2 LOCALITY OF INTER-TOKER DEPENDENCY
cornerstone REFUSION is grouping contiguous tokens into slots to enable inter-slot parallelism and intra-slot serialization. This design is motivated by the critical insight that the conditional independence assumption is most prone to failure for nearby tokens, frequently leading to semantic incoherence (Luxembourg et al. 2025). To formalize this insight and guide our design, conduct pilot study to quantitatively investigate how dependency strength between two tokens correlates with their relative distance.
Formally, we define the dependency strength between two tokens, zá and zi in given context Ta. the degree to which the presence of zy influences the model's prediction of z In practice, approximate this measurement in a pilot study on the GSM8K test set (Cobbe et al. (021). For a corrupted sequence 2t, we first reveal the ground-truth token zij at a randomly selected masked position j. and then quantify the influence of this reveal on the prediction any other masked position i through the Jensen-Shannon (IS) divergence (Manning & Schutze 1999) between the distributions
before and after this reveal, i.c., p(= IT) and p(z)) Iz, T6). A higher divergence implies stronger dependency, with zero divergence indicating conditional independence. Using both LLaDA (Nie) et al. 2025) and Dream (Ye et al. 2025), we plot the averaged JS divergence against the relative distance between positions i and j in Figure 6 The average JS divergence consistently decays as the relative distance increases, and this decay is more rapid in denser contexts (i.e., lower masking ratios t).
A.3 INFERENCE FORMALIZATION
In this section, formalize the two-stage decod- ing iteration as follows. Crucially, standard causal attention is consistently applied throughout the de- coding process, and the position IDs for all tokens always correspond to their indices in the original rect sequence, remaining invariant regardless of this reordering. By using these consistent position IDs with RoPE (Su et al., 2021), the model correctly per- ceives relative positions even when the input buffer is reordered.
LLaDA-BB
0.
0.00
Iend
60T-
2
Stage I: Diffusion-basse Slot Selection. The first stage leverages the model's MDM capability to select the next decoding slots. At timestep (defined as the ratio of remaining masked slots), construct the input S. for enabling KV cache by concatenat- ing already-decoded clean slots (Stelean in generation order) with the remaining masked slots (Stmasked, in their original positional order). The selection process then generates a draft for all masked slots. This draft serves a dual purpose: providing basis for scoring each slot for selection, and acting as a speculative guess for the subsequent infilling stage (Leviathan et al. 2002).. Specifically, for each position j in the i-th slot of Šuasked a draft token de is sampled from the model's marginal distribution, conditioned the leading context:
Figure 6: The locality of inter-token depen- dency MDMs, with the sign on the x-axis denoting the direction from the revealed token (positive for rightward, negative for leftward).
2.
Po(. Po. Sileen Smasked (t.)))
(5)
Smasked denote the tokens before the position of the target token. This yields a draft version where (.()(
of the masked slots, denoted as Syiral = {d.."), }. We then quantify the model's certainty score i-th slot St in Smasked as the model's predicted probability of its first token d.
C($)) (ott Po, St
Selean (.()))))
(6)
16

---

Published as a conference paper at ICLR 2026
The model then selects a batch of slots with scores exceeding threshold Tstor for subsequent infilling. no slot meets this criterion, the single slot with the globally highest score is selected instead. This strategy identifies slots that are strongly constrained by the existing context and weakly interdependent (e.g., distinct function definitions in code generation); making them suitable to parallelize.
Stage II: Autoregressive Slot Infilling. The second stage verifies and completes the selected draft slots by leveraging the model's 's autoregressive modeling capability. To accelerate infilling, we employ a speculative decoding strategy powered by the model's MDM capability. To achieve this, we first concatenate the slots in their original left-to-right order. The model then calculates the conditional probability of each token, conditioned on all preceding tokens within the newly formed sequence:
Po(di 1.. (o(d)
Pos Selean Smasked
Po: Sclean 5dma L.6(5))) i
P(
j
(,<().))
Next, we verify the draft by identifying the longest prefix of the concatenated sequence, denoted as length I, where every token's probability exceeds the threshold Tioken If this prefix spans at least one complete slot (i.e., 1 > k), we accept the first [1/k] slots in their entirety. The remaining unverified or partially verified slots are then re-masked, allowing the model to immediately initiate a new select-and-infill iteration while bypassing the costly suffix completion.
Otherwise, if the accepted prefix fails to cover least one complete slot (i.e., < k), we revert to pro- cessing the draft slots independently via Parallel Iterative Completion process. We independently 1 refine the draft S {d Jj=1 for each slot I in parallel, where I denotes the set of indices for the selected slots. The completion process iterates through two steps until all selected slots are fully completed. In each iteration m!
(i) Verification: For each slot E I. we identify the longest valid prefix length 1!Th) where the conditional probability of every token in the prefix exceeds Tiokens conditioned on the prompt, the Arraad--ecodee clean slots, and the preceding tokens within slot i:
(m))
max {n Vj S 7, Po(
po. St. Splean J:31) > Ttoken J
(8)
where d. denotes the prefix within the i-th slot.
(ii) Prediction: We retain the valid prefix di g.145") ra-mask the remaining suffix, and predict the masked tokens by sampling from the model's distribution:
H Pol: po. St Selean
Smasked
Vj E (L,m)) k).
After infilling each selected slot, the completed slots are moved from Smaskod to Selomm For the subsequent iteration, the KV caches from these parallel-generated slots are concatenated. While this parallel generation forgoes inter-slot conditioning, we observe in our experiments that this has minimal impact on final performance (see $C.3). This select-and-infill iteration repeats with an updated timestep t until no masks remain (t = 0). at which point the final response Fu is formed by sorting Splean back into its original sequence order.
B EXPERIMENTAL DETAILS
B. IMPLEMENTATION DETAILS
Our training data comprises 3.7M samples from MAmmoTH (Yue et al. 2023), OpenMathInstruet- 2 (Toshniwal et al., 2024), OpenCoder (Huang et al.] (022). SmolLM (Allal et al. 2025), and Tulu
3 (Lambert et al.| 2024). For OpenMathInstruutt.. we use its IM open-source version and remove questions longer than 1,024 tokens as instructed. We use global batch size of 512, maximum sequence length of 4,096, and a learning rate of 2e-5. The training is conducted on 16 nodes, each with 8 H20 GPUs, and is accelerated using DeepSpeed ZeRO-2 (Rajbhandari et al. 2020) and Flash-attention-2 2 (Dao 2023). The total training cost is approximately 10.68K H20 GPU GoU-hours. We set X in Eq. 4 to 1. For each training sample, randomly select slot size from (4, 8, 16, 32).
17

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameter settings for different tasks.
Benchmark
Generation Length
Slot Selection Threshold Tix
Token Acceptance Threshold "saen
Slot Size
Block Size
MMLU-Pro
0.9
16
ARC-C
512
0.8
0.1
8
GSMSK
512
0.9
0.4
32
128
MATH
512
0.8
0.5
32
GPQA
512
0.8
0.2
8
HumanEval
0.9
32
MRPP
512
0.9
0,3
32
128
Table 6: Performance comparison of different models on HumanEval and MBPP
Model
Total Params
A Activated Params
System Optinization
Throughput (TPS)
HumanExal (# Shots)
MBPP ( Shots)
Auterograssiv Model
Nova Micro
79.30 (0)
65:40 (3)
GPT 4o Mini
59
$8.00 (0)
74,60 (3)
Gemial 2.0 Flash Lite
201
90.00 (0)
75.00 (3)
Markee Diffusioo Model
Merenry Coder Mini
1,109
88.00
27.10 (3)
Mereary Coder Small
737
90.00 (0)
76.60 (3)
Gomini Diffusion Diffasion LLaDA-MoE
1,479
89.60 (0)
76.00 (3)
82,80
REFUSION
8B
3B
78.66 (0) 68.20 (0)
Existing MDMs decode sequences to predetermined length. Even when end-of-sequence (EOS) token appears early, the model still expends decoding time on all tokens with higher position IDs. To address this issue, introduce mechanism for efficient variable-length generation. Specifically, during training, we pad shorter sequences a mini-batch with padding tokens and exclude these tokens from the loss computation. During inference, upon generating EOS token, dynamically truncate the target length to that token's position. This prevents the decoding of any tokens with higher position ID, thereby reducing redundant computation.
B.2 HYPERPARAMETER SETTING
During the REFUSION inference process, four hyperparameters can be adjusted: the slot selection threshold Tslat the token acceptance threshold Tioken, the slot size k; and the block size b. Table shows the specific settings used in our evaluation.
C EXPERIMENT RESULTS
C.I COMPARISON WITH CL.OED--SOURGEE AND DIVERSE ARCHITECTURES
To comprehensively demonstrate our advantages, we further compare REFUSION against closed- source and diverse ARM and MDM architectures. As shown in Table 6 REFUSION outperforms several highly optimized ARMs either speed or performance. Specifically, it achieves 1.66x the throughput of GPT-4o Mini and surpasses Nova Miero 2.8 points MBPP. Furthermore, REFUSIOn yields results approaching those of larger and more powerful closed-source MDMs.
C.2 ABLATION ON CERTAINTY SCORING
key design choice in our slot selection stage the metric used to compute the certainty score, which determines which slots are selected for parallel generation. In our default implementation, as described in Section A.3 we use the probability the most likely token at the slot's first position. This choice is specifically motivated by our two-stage decoding process: the diffusioo--base stage aims to identify valid anchors for parallel generation, while the subsequent autoregressive infilling stage ensures local coherence by completing the slot conditioned this initial token. Therefore, the confidence the first token serves as an efficient and effective proxy for the viability initiating a slot's generation.
18

---

Published as a conference paper at ICLR 2026
Table Comparison of certainty scoring strategies on zero-shot performance. 1 (4) indicates that higher (lower) value represents greater certainty. "Prob. of First Token" " denotes our default method (used in Table T while "Mean Prob. of Slot" and "Mean Entropy of Slot" serve as alternatives. The highly comparable results across metrics validate our design choice.
Method
MMLU-Pro
ARC-C
GSM8K
MATH
GPQA
HvmanEval
MBPP
Prob. of First Token
45.94
89.76
84.91
54.22
35.49
78.66
68.20
Mean Prob. Slot
44.96
89.76
82.71
54.18
34.60
78.66
68.00
Mean Entropy Slot +
41.95
89.76
83.09
54.22
33.71
77.44
66.80
Table Ablation regarding our KV cache reuse mechanism. We compare our default REFUSION, which efficiently reuses KV caches by concatenating them after parallel generation, against a variant (w/ KV Re-computation) that recomputes caches for full contextualization at a higher cost.
Model
MMLU-Pro
ARC-C
GSM8K
MATH
GPQA
HumanEval
MBPP
45.56
89.76
84.38
54.18
35.49
77.44
68.20
REFUSION w/ KV Re-computation
42.80
28.03
69.42
69.20
48.51
78.00
74.45
45.94
89.76
84.91
54.22
35.49
78.66
68.20
REFUSION
52.74
32.46
81.24
81.77
64.11
103.90
92.09
Alternatively, more intuitive approaches might involve using the mean probability or average entropy of the most likely tokens across all positions within draft slot. To evaluate these alternatives, conduct comparative experiment. As shown in Table 7 the performance of all three methods is highly comparable across seven benchmarks. This consistency suggests that these metrics are robust and likely identify a significantly overlapping set of high- high-confidence slots. Given this performance parity and the lower computational overhead of the first-token probability, we retain as our default method.
C.3 ABLATION ON KV CACHE REUSE
60-
ReFusion
To maximize efficiency, REFUSION directly concatenates the KV caches of costly forward pass that would otherwise be needed to
pass (%)
LLaDA-BB-Instruc
slots, bypassing a
Dream Instruct
contextualize them. To quantify the impact of this ap- proximation, conduct ablation study comparing our default model against variant, "REFUSION w/ KV Re- computation," " which performs this extra forward pass ensure full contextualization at the cost of speed.
2
4
Average #TPF
As shown in Table 3 our default approach is consistently 1.36x faster across all benchmarks. Surprisingly, this significat speedup comes at virtually no cost to perfor- SION and baseline MDMS over TPF. mance; in fact, accuracy remains stable and even slightly improves on several benchmarks. We hypothesize this counter-intuitive benefit arises from form of implicit regularization: by avoiding ever-conditioning method mitigates error propagation. This result validates our KV cache reuse strategy not merely speed-accurag trade-off, but as design choice that simultaneously enhances efficiency and robustness.
Figure 7: Pass@) on MBPP for REFU-
potentially flawed parallel drafts, our
C.4 TRADE OFF FRONTIER ANALYSIS
To further distinguish REFUSION from prior MDMs, we investigate the trade-off frontiers of different models. Figure/]ssoww that both LLaDA and Dream suffer a sharp performance decline as parallelism (tokens generated per forward pass, TPF) increases indicating failure to uphold the conditional independence assumption when selecting tokens for parallel decoding. In contrast, REFUSION's
Wee use TPF here, rather than TPS, isolate the algorithmic trade-off from any system-level overheads.
19

---

Published as a conference paper at ICLR 2026
curve is substantially flatter, validating that its training and decoding strategies more reliably identify conditionally independent tokens.
C.5 SCALING WITH DATA SIZE
To understand the scaling properties of our model, we investigate the impact of training data size on REFUSION's performance and efficiency. To this end, we collect additional data of potentially lower quality to expand our training set to 14M samples. Figure Figure@iilustrates illustrates the results this analysis on GSM8K and MBPP, where we train REFUSION for one epoch on datasets of varying sizes (from 120K to 14M samples) and evaluate it using the same hyperparameters as in Table 5
The results reveal clear and positive scaling trend for both key metrics. Specifically, throughput (TPS, dashed lines) improves generally consistently as the training data size increases. For instance, on MBPP, throughput rises from approximately 51 TPS with 120K samples to over 81 TPS with 14M samples. This indicates that as the model is exposed to more diverse data, its internal generation process becomes more efficient, likely due to higher confidence and thus higher acceptance rate of its parallel drafts, leading to fewer decoding iterations.
Interestingly, the performance scaling (solid lines) is not strictly monotonic, common phenomenon when training with a fixed epoch count. On GSM8K, accuracy peaks at 2M samples before slightly decreasing 3.7M. This behavior highlights trade-off between data breadth and training depth under a constrained computational budget: with a fixed one-epoch schedule, training on a larger dataset potentially leads under-convert convergence relative to the dataset's complexity.
Nevertheless, the consistent rise in throughput coupled with the substantial performance uplift from the 120K baseline suggests that with an increased computational budget (i.c., more training epochs on the larger datasets), performance would likely continue to improve, further unlocking the full potential of our approach.
C.6 ANALYSIS OF GENERATION LENGTH AND LATENCY
To ensure that REFUSION's superior perfor- mance is not merely an artifact of generating more tokens than baselines, rect comparison of the average generated token length and total inference latency for represen- tative tasks. The results, shown in Table | ad- dress the hypothesis that quality gains might stem from quantity.
GSMBK (0-shot)
-75
82
MRJP (0-sbot)
82
80
present a di-
30-
-000
60
Paod8 (55)
70
Accensry
Thronghren
-6-
Thouphy (TPS)
Thoogggu
70
120K IM
2M
3.7
14M
120K
1M
254
3.7M
34M
The data clearly demonstrates that REFUSION's generated outputs are consistently and signifi- cantly shorter than those of the ARM baseline (Qwen3-8B) and are either shorter compara- ble in length to other MDMs. For instance, on MMLU-Pro, REFUSION generates only 124 to- kens, roughly 5x fewer than Qwen3-8B, while achieving superior performance. This directly refutes the hypothesis that model's quality gains are achieved by generating longer sequences.
Data Size
Datu Staee
Figure Scaling properties of REFUSION on GSM8K and MBPP We plot performance (Aceu- racy/Pass@@l 1, %) and inference throughput (TPS) function of training data size.
Furthermore, the table highlights REFUSION's dramatic efficiency advantage, with measured la- tency being substantially lower across all tasks. These results confirm that REFUSION's superior performance-efficienngg profile is direct result of its methodological innovations, enabling it to produce concise and high-quality responses with minimal latency.
C.7 ANALYSIS OF BLOCK SIZE
Our inference strategy is compatible with block-based diffusion methods (Arriola et al. 2025). Specifically, during inference, the target sequence is partitioned into consecutive blocks of size b. These blocks are decoded sequentially, with our decoding algorithm applied within each block. Notably, the constraint b > & must be satisfied, where & is the size of a slot, the fundamental unit for parallel decoding in our method.
20

---

Published as a conference paper at ICLR 2026
Table 9: Comparison of average generated length and total latency (in seconds) across key benchmarks. Each model displays generated length (top row) and the measured latency (bottom row).
Model
MMLU-PPr
ARC-C
GSM8K
MATH
GPQA
HumanEval
MBPP
654
300
778
810
444
Qwen3-8B
20.82s
1.46s
9.63s
25.84s
26.59s
14.34s
1.74s
251
247
242
9
334
82
LLaDA-SB-Instruct
13.81s
32.86s
9.02s
10.13s
4.71s
26.85s
27.65s
211
223
206
10
121
45
Dream-7B-Instruct
13.23s
33.74s
10,99s
10.87s
5.39s
34.36s
36.26s
124
141
177
9
65
45
REFUSION
2.358
0.07s
1.74s
2.17s
0.14s
0.63s
0.48s
Figure iillsttaate the impact of block size on our method's performance and throughput (TPS). The figure reveals that both metries exhibit non-monotonic trends as b increases. The trend in performance arises from trade-off between the global selection horizon and modeling complexity. Specifically, while larger 6 allows the model cover longer semantic units in single selection to enhance coherence, it simultancously imposes a greater challenge in generating larger, more complex blocks in arbitrary order. Similarly, the non-monotonic trend in throughput (TPS) is attributable to computational overhead. Although a larger b provides more opportunities for parallelism, forces the model process longer sequence containing many "padded" (i.e., tet-t--be-generatte positions. This significantly increases the latency of each decoding step, which eventually diminishes and then reverses the throughput gains observed with larger block sizes.
Although performance fluctuates slightly with larger block sizes, REFUSION's pass@ decreases by only approxi- mately 0.2% relative to its peak as b increases from 32 to 512. This robustness block size highlights the model's ability to leverage strong diffusion- based selection to se- lect the most appropriate slots for decoding across wide range. Collectively, these analyses reveal a robust and wide "sweet spot," highlighted by the yellow shaded re- gions in Figures and where REFUSION consistently surpasses the Qwen3-8B baseline in both performance and throughput (TPS). This superior operating cor- responds to a block size E [32, 128).
60
Thooughpat (TPS)
pass (%)
(%)
58
A Throughut
56
64
128
256
512
Block Size
Figure Relative change in REFU- SION'S pass@1 (%) and throughput (tokens/sec) compared to Qwen3-8B (horizontal dashed lines at zero) block size b varies. The yellow shaded region highlights the range of b where REFU- SION surpasses Qwen3-8B.
D CASE STUDY
D.1 CODE GENERATION CASE STUDY
Table 10 showcases responses to the same MBPP problem
Figure from different models. REFUSION's ability globally sketch the high-level structure via a diffusion- based process while locally infilling details autoregressively results in better-structured high-quality code, demonstrating superiority over existing MDMs.
D.2 STEP-BY- BY-STEP VISUALIZATION OF INFERENCE
In order to facilitate the understanding of our inference method, we show a step-by-step decoding process in Figure I0 Specifically, our inference process progressively generates the response through an iterative "select-and-ing
mechanism.
The model maintains standard token-wise causal attention throughout the entire process. Each decoding iteration operates as a two-stage cycle: First, the selection stage predicts drafts for all masked slots in parallel and selects subset of high-quality slots based confidence scores. Second, the infilling stage treats the selected slots as a batch and completes them autoregressively to ensure local coherence.
21

---

Published as a conference paper at ICLR 2026
Table 10:
Model responses the same MBPP problem as Figure
Problem
You are an expert Python programmer. Your task is to write a single Python function to solve the problem deseribed below, and here your task: Write a function sum amicable numbers from specified number. Your code should pass these
assert amicable, amiccabl_onubbessuu. numbers_aum (999)
assert mwwicable_uumbesscco (9999) -316266 assert
sum (99) --0
LLDDA-BB-Instru
def amicable_numbet (1imit):
def La_perfect_aqpatt (n)
root int (m**0.5) return root root
def sum dum_eivieerstt) in):
total for i in range(1, n): ir a - 0: total return total
def 1s Le_amicalle return sum_divisors (a) == b and sum_divisors (b) ==
ami caiceble set
for i in range 1imit 1): if _s_perfect_squaree
amicable.ado
for j in range + limit
if is ie_amicable (1, }):
amicable.
return sum (amicable)
Dream-7B-Inse BB-Insruct X
def ami cawll_aanbbbrrssaeccooacoaaoaaee
numbers (n)
def is issanicael (num) divisors
for i in range (2, num) if num & i -- 0:
divisors +-
!- divisors +- num //
return divisors num
total
for i in range (2, n): if is_smicable(i) total +- 1 return total
REFUSION
def amicable wawrablleennbbereeeeeee (n) def sum_divisors (num)
sum
range (2, int (mum+ 1)
if num $ i -- 0: aum +- 1
i != num //
//
return Sum
ami caall__b_sau
for i in range (2, 1): sum sum_divisorsidl if sum amicable_st += return amicablo_ssm
sum divisors 1:
To enable full KV cache reuse, the newly decoded slots moved to the front of the remaining masked slots after infilling. Crucially, while the tokens' positions within the input sequence may change due to this reordering, their position IDs remain invariant, always corresponding to their
22

---

Published as a conference paper at ICLR 2026
indices in the original correct sequence. By utilizing these consistent position IDs with RoPE (Su et al. 202I), the model accurately perceives the relative positions all tokens, even when the imput buffer is reordered.
E LIMITATIONS
A primary limitation of our current framework is the immutability of generated slots. Once the tokens within a slot are generated via diffusion-based selection and autoregressive infilling, they considered final and cannot be remasked refined in future iterations. This design choice, while simplifying the process, precludes the model from correcting potential errors made within completed slot.
promising direction for future work would be to introduce a re-masking mechanism at the sub-slot level. For instance, after infilling slot, the model could verify the generated tokens and preserve only a high-confident prefix, while re-masking the lower-quality suffix. This would allow for iterative refinement but would necessitate more complex inference logic, potentially involving dynamic adjustments of slot sizes to handle these newly masked, smaller segments. Developing an efficient strategy for such dynamic, fine- ane-grained refinement remains a key challenge for future research.
F THE USE OF LARGE LANGUAGE MODELS
In the interest of complete transparency, we wish clarify the use AI assistance in the preparation of this manuscript. The core research ideas, including the conception of the REFUSION model, the design of the training and inference algorithms, all experimental setups, and the analysis the results were developed exclusively by the human authors. We utilized Large Language Model for the limited purpose linguistic refinement. This involved polishing certain sentences and paragraphs to improve grammatical correctness, clarity, and overall flow. This usage restricted to editing and did not extend to research ideation, content generation, or experimental analysis.
23

---

Published as a conference paper at ICLR 2026
High--qulity
promus KV cache fixed at the sequence start (no shilling and it cmmited from later steps for visual clarity.
Masked token
KVV-aached token
Forward pass
Rule-based high high-quality slot selection
Prompt
the proverb early bird
Answer
XXXX
XXX 15
1=0
Position
10
17
Diffusion-basee Slot Selection
bird
Suscleo
werm
in
8
10
13
16
17
II, Autoregressive Slot Infilling
(1) Globel Veriiiatiion
Slot Length
Answer
Landes
the wam
XXX
Position
7
15
16
17
Step
Diffision-base based Slot Selection
janche
10
14
Autoregressive Infilling
(1) Global Verification
Length
Secho
Jurm
The
bed
work
stat
10
11
12
13
14
Completion
worm Swceld
D2
PVOLES
Kweeld
Martt
your
10 1I
wore
enty
Verteeatoe
kucch ttz worm
chesld sten
your
10
11
Answer
Skould
Position
13
Step
Diflision-bssed Slot Selection
Slould
stan
your
work
Autoregressive Infilling
(1) Global Verification
Slot Length
which roerss
you
powitte
Answer Fanchae
The
caryy
bind
your
Position ID
10
11
12
13
14
15
16
17
Answer
The
zarts
kinl
Ihc
wom
which
meeess
y0u
6scal
NOIS
t=1
Position
9
10
11
12
13
14
15
16
17
Figure 10: Visualization of the REFUSION inference mechanism.
24


Reviews:
[{"review_id": "PPAzgKbBQp", "reviewer": "Reviewer_ZGiW", "summary": "This paper focuses on two intrinsic drawbacks of masked diffusion models (MDMs): the infeasibility of KV caching and missing token dependencies.\nTo alleviate these issues, the authors propose to combine the AR and MDM paradigm, called the ReFusion Model.\nAlthough this idea is not very novel, the authors discuss in detail the practical design of the ``plan-and-infilling'' process.\nExtensive experiments demonstrate the generation speedup benefiting from the KV caching and the improved accuracy gain.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors propose a practical ``plan-and-infilling'' process that works fairly well, although the basic idea of ReFusion might not be very novel.\n- The pilot study in Section 4.1 is very interesting and clearly exhibits how the distance affects the correlation.\n- The experiments, including the ablation studies, provide a very comprehensive understanding of how ReFusion works.", "weaknesses": "- Similar ideas have been discussed in previous works. E.g., BD3-LM (https://arxiv.org/pdf/2503.09573) utilizes the block diffusion, EDLM (https://arxiv.org/abs/2410.21357) utilize AR models to model the correlations. The authors should clearly state the differences and their unique contribution.\n- I find the two-step inference method in Section 4.2 somewhat obscure. I suggest the authors reorganize the desciption into mathematical equations and add more details (e.g., how to perform positional encoding for $S_t^{clean}$ and $S_t^{mask}$).\n- The authors should discuss more detailedly how why the use the slot's first position to compute the certainty score. For example, the mean of the probs on all positions in a slot seems to be a more appropriate method.\n- As an intrinsic drawback of the ReFusion method, there are many critical hyper-parameters to be tuned. e.g, $k$ and $\\tau$ and $\\lambda$. These parameters may depends on the data sets, and can affect the broader application of ReFusion.\n- Why is the weight $1/t$ missing from MDM objective in equation (2)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qm5McXd3zX", "reviewer": "Reviewer_VXks", "summary": "This paper proposes ReFusion, a masked-diffusion LLM that performs plan-and-infill decoding at a slot level: a diffusion planner selects weakly dependent fixed-length spans (“slots”), and an AR stage that fills multiple slots in parallel. By reordering into a causal sequence during infill, the method reuses exact KV cache, narrowing the cache-efficiency gap between diffusion LMs and AR models. \n\nAcross various benchmarks and two dLLM architectures (LLaDA, Dream), ReFusion reports consistent >10× speedup over prior MDMs, and about 1.4x faster than SOTA AR models.", "questions": "1. The authors mention 3.7M samples; approximately how many tokens does this correspond to, and how long does training take on 16x8 H20 GPUs (total GPU-hours)?\n\n2. How orthogonal is ReFusion to other dLLM acceleration techniques (e.g. confidence-aware parallelism)? Could the slot planner or infill step compose with the techniques, and if so, what additive speed/quality gains should we expect? \n\n3. Do the authors have preliminary results on how does scaling the training data beyond 3.7M samples shift the speed–quality frontier? \n\n4. In Table 1, why is D2F significantly worse on ARC-C than its performance on other tasks? any intuitions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The slot abstraction plus causal infill provides an intuitive route to exact KV-cache reuse, reducing efficiency gap with AR decoding. \n\n2. The paper provides comparison with competitive baselines, and it shows consistent wins on most tasks with both LLaDA and Dream across many tasks, supporting generality. \n\n3. The paper probes slot thresholds and provides qualitative evidence that aligns with the design intuition.", "weaknesses": "1. ReFusion introduces extra data preparation and training cost, adding pipeline complexity to realize its gains.\n\n2. While ReFusion is much faster than prior MDMs, its throughput versus AR models is not significant better, and ReFusion may not be orthogonal to existing tricks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u8de8I986a", "reviewer": "Reviewer_dCJA", "summary": "This paper introduces a training method to achieve inter-slot parallel decoding with intra-slot autoregressive decoding for dLLMs. The core idea is that conditional independence assumption is most prone to failure for nearby tokens, and thus the authors propose to perform serial decoding within a slot, in contrast to prior method, e.g., block diffusion. The method consists of two parts. For inference,  it would first decide which blocks to decode in parallel, and then reorder the slots to do parallel decoding while still have autoregressive decoding in each slot. For training, the authors propose to adopt two objectives in training, one MDM loss for masked tokens and another AR loss for clean tokens inside the slots.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors try to challenge one common belief in current dLLM literature, that we need to perform intra-block auto-regressive decoding  instead of intra-block parallel decoding. The analysis is interesting, and it's also very interesting to see the challenge of common belief.\n2. The proposed method is soundness, both for the training and the inference side.", "weaknesses": "My primary concern lies in the unfair, insufficient, and potentially incorrect experimental evaluations, which lead to several overstated claims in the paper.\n\n* **Missing Comparison with Block Diffusion**\n\nOne of the key hypotheses this paper wants to show is that intra-block autoregressive decoding + inter-block parallel decoding is superior to intra-block parallel decoding + inter-block autoregressive decoding. However, no experiments are provided to substantiate this claim. A direct comparison with Block Diffusion would be necessary to validate the proposed design.  Without empirical comparison, it is unclear whether the proposed method offers any real advantage over Block Diffusion in this aspect.\n\n* **Unfair Comparison with Prior MDMs/dLLMs**\n\nThe authors claim that Refusion overwhelmingly surpasses all prior MDMs in both performance and speed. **However, this performance gain comes mostly from a better base model.** The comparisons with previous dLLMs are not fair, as the proposed model adapts from a much stronger base model, Qwen3. While I acknowledge that the proposed method achieves the best overall results and appreciate this contribution, fair evaluation requires experiments starting from the same base model as prior works. Training from scratch may be costly, but at least one experiment comparing the proposed approach with DREAM, using the same initialization checkpoints, would be essential for a fair comparison.\n\n* **Significant Performance Drop Compared to Qwen3, the base model of Refusion**\n\nThe adapted model exhibits an 8.5% average performance drop relative to Qwen3. This is a substantial degradation. The paper claims that the model “challenges strong ARMs,” but given such a drop in accuracy alongside a 1.34× speedup, the claim seems overstated. Besides, have you conducted experiments to evaluate whether the acceleration ratio still holds when the batch size is larger than 1?\n\n* **Confusing Results in Table 2**\n \n(1) For Qwen3, training with the original objective (presumably the NTP loss) results in a 10.9% performance drop, which seems unusual and needs clarification.  \n(2) For the GPQA benchmark (a 4-choice classification task), the reported performance is 0, which is implausible. Previous works using lm-eval-harness constrain predictions to 4 choices, meaning even random guessing should yield roughly 25% accuracy.  \n(3) The reported TPS (tokens per second) for LLaDA is suspiciously low. While prediction accuracy may differ, TPS should mainly depend on the model structure and sequence length. The discrepancy between Table 1 and Table 2 needs more explanation.  \n(4) The paper claims that the proposed model outperforms Qwen3-8B on HumanEval by ∼9 points while being 1.3× faster. However, Qwen3-8B without the authors’ training already achieves 87.8, which is actually **25.6%** higher than Refusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r383eZY7ni", "reviewer": "Reviewer_wAnP", "summary": "The authors propose ReFusion, a hybrid generative model that integrates inter-slot parallel decoding with intra-slot autoregressive decoding. ReFusion is the first Masked Diffusion Model (MDM) to achieve full KV cache reuse for every decoded token, a significant efficiency gain, while maintaining global generation flexibility and a tractable training process. Based on their experiments, the authors assert that ReFusion dominates all MDM baselines and is competitive with autoregressive models (ARMs).", "questions": "1. In your discussion of Eso-LM, you state: “dynamically reposition newly generated tokens ahead of masked ones at each step to facilitate caching. However, this strategy introduces an intractable learning objective at a token-level permutation space” (lines 130-132). Can you please clarify what this \"intractable learning objective\" is and why it is intractable? What makes ReFusion different?\n\n2. For the experimental benchmarks, how many tokens are being generated for each task? Is it possible that ReFusion achieves better quality simply by generating more tokens than the baselines?\n\n3. What are the measured latencies for ReFusion and the baseline models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The paper combines the strengths of diffusion and autoregressive generation, helping to maximize parallelism and quality.\n\n* The idea of achieving full KV caching for every token within a parallel generation framework is significant and worthwhile.\n\n* The paper provides an interesting analysis regarding the locality of inter-token dependency.\n\n* The reported experimental results are strong, showing performance that Pareto-dominates existing baselines.", "weaknesses": "* The proposed method appears conceptually similar to Eso-LM [1]. At a high level, both models seem to perform diffusion and autoregressive infilling. The authors must do more to clearly and substantially differentiate ReFusion from this existing work.\n\n* The paper lacks clarity regarding the training and architectural choices. It is not at all clear how the model achieves \"full KV cache reuse of every decoded token\" (lines 103-104). Diffusion LLMs typically require bidirectional attention to support flexible, any-order generation, which inherently prevents KV caching. However, Table 4 claims ReFusion supports any-order generation while using a causal mask. It is very difficult to understand how the model can simultaneously support parallel sampling and use a causal mask for KV caching. This mechanism would be made more clear with a detailed example of a training input and its corresponding attention mask.\n\n* The illustrative example provided in Section 5.5 is very confusing and fails to clarify the generation process. It is not clear which text segments were generated via diffusion and which were generated autoregressively.\n\n* The manuscript reads as heavily LLM influenced, but no disclosure is made. \n\n[1] Sahoo, Subham Sekhar, et al. \"Esoteric Language Models.\" arXiv preprint arXiv:2506.01928 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)