AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Size
Authors: Guanxi Lu
Hao Mark Chen
Yuto Karashima
Zhican Wang
Daichi Fujiki
Hongxiang Fan
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Large Language Models
Non-Autoregressive Decoding
Efficient ML
Primary area: generative models
TL;DR: A lightweight scheduler that enhances sampling quality for Diffusion-based LLMs.

Abstract: Diffusion-based large language models (dLLMs) are gaining attention for their inherent capacity for parallel decoding, offering a compelling alternative to autoregressive LLMs. Among various decoding strategies, block-wise semi-autoregressive (semi-AR) approaches are widely adopted due to their support for KV caching and their favorable accuracy–speed trade-off.
However, this paper identifies two fundamental limitations in the conventional semi-AR decoding approach that applies a fixed block size: i) late decoding overhead, where the unmasking of high-confidence tokens outside the current block is unnecessarily delayed, and ii) premature decoding error, where low-confidence tokens inside the current block are committed too early, leading to incorrect tokens. This paper presents the first systematic investigation challenging the fixed block size setting in semi-AR decoding. Through a statistical analysis of confidence dynamics during the denoising process, we identify a volatility band (VB) region during dLLM decoding, which encodes local semantic structure and can be used to guide adaptive block sizing.
Leveraging these insights, we introduce AdaBlock-dLLM, a training-free, plug-and-play scheduler that adaptively aligns block boundaries with semantic steps by adjusting block size during runtime. Extensive experiments across diverse benchmarks show that AdaBlock-dLLM achieves up to 5.3% accuracy improvement under the same throughput budget. Beyond inference-time optimization, we hope our semantics-aware adaptive scheduling approach and confidence-based analysis will inspire future training strategies for dLLMs. Our code is available at https://github.com/lgxi24/AdaBlock-dLLM.

Published as a conference paper at ICLR 2026
AdaBlock-dLLM: SEMANTIC-AWARE DIFFUSION LLM INFERENCE VIA ADAPTIVE BLOCK SIZE
Guanxi Lu' Hao (Mark) Chen'
Yuto Karashima
Zhican Wang' Daichi Fujiki²
Hongxiang Fan'
'Imperial College London, UK
Instittte of Science Tokyo, Japan
{guanxi.lu2) lu22, hao.chen200 @ immperialac.ukk uk. karashima. yuto @nto@@rriiciiii arttic.irisct.acc.p wangzhican @ gmail.com d'uuiki @ www.ti.iiiisst.acc. hongxiang.fra imperial.ac.uk
ABSTRACT
Diffusion-based large language models (dLLMs) are gaining attention for their inherent capacity for parallel decoding, offering a compelling alternative to autoregressive LLMs. Among various decoding strategies, block-wise semi- autoregressive (semi-AR) approaches are widely adopted due to their support for KV caching and their favorable accuracy-speed trade-off. However, this paper identifies two fundamental limitations in the conventional semi-AR decoding ap- proach that applies fixed block size: late decoding overhead, where the un- masking of high-confidence tokens outside the current block is unnecessarily de- layed, and ií) premature decoding error, where low-confidence tokens inside the current block are committed too early, leading to incorrect tokens. This paper presents the first systematic investigation challenging the fixed block size set- ting in semi-AR decoding. Through statistical analysis of confidence dynam- ics during the denoising process, we identify a volatility band (VB) region during dLLM decoding, which encodes local semantic structure and can be used to guide adaptive block sizing. Leveraging these insights, we introduce MlaBllock-LLLM training-free, plug-and-play scheduler that adaptively aligns block boundaries with semantic steps by adjusting block size during runtime. Extensive experi- ments across diverse benchmarks show that AdaBlock-dLLM achieves up to 5.3% accuracy improvement under the same throughput budget. Beyond inference- adaptive scheduling approach
time optimization, we hope our semantics-awan and confidence based analysis will inspire future training strategies for dLLMs. Our code is available at //ttts://gihuu..oom// wot..////.mmom////aalaoooooook
INTRODUCTION
Diffusion-based large language models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering parallel decoding, improved controllability, and greater data effi- ciency in low-resource settings (Zhang et al.] 20025] Prabhudesai et al.) (025). Open-source dLLMs such LLaDA (Nie al. 2025 Zhu et al. 2025) and Dream (Ye et al.) 2025 have demonstrated comparable performance to autoregressive models of similar scale. Notably, in structured generation tasks such as coding, proprictary models including Seed Diffusion (Song al. 2025b) and Gemini Diffusion (Gemini Diffusion) 2025) have achieved throughput exceeding 1,400 tokens per second. These advances highlight the potential of dLLMs to deliver efficient inference while maintaining competitive algorithmic performance.
Recent works have widely adopted semi autoregressive (semi-AR) decoding paradigm that com- bines block-wise KV caching (Wu et al., [2025] Chen et al., (2025] [Sng et al. (222a) and confidence- based dynamic sampling (Wang et al.) 2025d] | Wu et al. 2025) Wei et al.) (025) to improve inference efficiency. However, semi-AR decoding enforces levek causality: the current block must be finalized before decoding the next block. We identify two fundamental issues introduced by con- ventional semi-AR decoding with fixed block size: i) Late Decoding Overhead. As shown in the upper-left of Figure [ semi-AR decoding delays the unmasking of high-confidence tokens outside the current block. For instance, the second and third blocks decoded in separate iterations, in- curring unnecessary computational overhead to generate a simple complete sentence. ii) Premature Decoding Error. As shown in the lower-left of Figure 1 the autoregressiveness across blocks forces

---

Published as a conference paper at ICLR 2026
Prompt: Alice has trince apples, and Bob has four. How many anarges de they have in tohal?
Conventional
(Fixed size)
Approach (Adeptive block size) deme
Sepaace
(KASS) (ARER) (P300)
Seap sequarcc
Juttel
(spua]] (eeeec [SDAR]
Denobe
AAic bed thrs sailE
BG Dae
they tuta
Denoir
tol meree opale and Beb toa four
They They hotel
Opples
Somple AAce her three srplee
Somple Aloc toa mree opála and lct for four
Adaptive block size
Adaptive block size
High confidence tokens not decoded
Denwre Alce lei theer exple end Kay hei forr
Tetal
Issue 1: Late Decoding Overhead
confidlncce decoded
Semple Alia
Imergn
Seople Aloe Hire Expler
Decade Low- Low-Cenfidence Taken
Ooffdence
Corfidence
Issue-2: 2: Premoture Dacading Error
Figure I: Illustrative examples of two fundamental issues (left) and how AdaBlock-dl..N addresses them (right). Appendix A.1 presents a case study from a real inference scenario.
early commitment to low-confidence tokens within each block, and this suboptimal sampling often yields incorrect token predictions (Figure particularly in reasoning tasks.
To address these limitations, we begin by investigating how confidence scores evolve across decoding steps. Our statistical analysis identifies a volatility band (VB), a re- gion in which confidence fluctuates markedly over time. VB regions encode local semantic structure, which can be exploited dynamically adapt block size during runtime.
Accuracy vs Throughout
Madiod
80
Accuracy (8)
To address these limitations, we analyze how confidence scores evolve across decoding steps. Our statistical anal- ysis identifies the volatility band (VB), a region in which confidence exhibits high temporal variance over time. We find that VBs often coincide with local semantic units, pro- viding a signal that can be leveraged adapt block sizes at inference time. Motivated by this observation, we propose AdaBlock-dl a training-free, plug-and-play adaptive block-size scheduler for semi-AR dLLM inference that ad- justs block boundaries semantics-avee manner (Fig- ure D right). Specifically, AdaBlock-dLLM aligns block size with semantic boundaries, indicated by semantic delimiter tokens (e.g.. periods and (n), thereby improving sampling quality. Based various benchmarks,
$.3
76
74-
50
70
90
110
130
150
170
190
Throughput (takensts)
Figure 2:
Performance improvement over Fast-dI.LM (Wu et al. 2025).
on comprehensive experiments proves accuracy by up to 5.3% while achieving throughput comparable to prior dI.LM acceleration methods (Figure 2) Gains especially pronounced under KV caching, where fixed block sizes further compromise semantic consistency. Our results motivate exploring semantics-awart training objectives for diffusion language models that better preserve contextual coherence.
demonstrate that AdaBlock-dLLM im-
In summary, our contributions are threefold:
We systematically analyze the semi- cutreeetoreee decoding paradigm, and identify the inaceu- racy and inefficiency behind fixed block size settings (Section 4.1 and Section 4.2).
We propose AdaBlock-dLLMM, a training-free, plug-and-play technique that enhances the exist- ing semi-autoregressive decoding paradigm, which dynamically adjusts block sizes based on the confidence of semantic delimiter tokens (Section 4.3).
We conduct extensive experiments demonstrating that AdaBlock- LLLM improves accuracy by up to 5.3% over state-of of-the-ar methods under the same speed budget (Section
2 RELATED WORK
2.1
DIFFUSION LANGUAGE MODELS
Diffusion models (Sohl-Dickstein et al.) 2015 Ho et al. 2020; Song et al. 2021 Karras et al,
2022) have achieved high-fidelity generation across continuous data domains, including images and
2

---

Published as a conference paper at ICLR 2026
video (Peebles & Xie 2023 Ho et al. 2022). Motivated by this success, growing line of work adapts diffusion to NLP tasks, giving rise to masked diffusion models (MDMs) that iteratively de- noise an initially masked token sequence into coherent output (Austin et al. 2021a Hoogeboom et al. 2025
2021: Lou et al. 2024). Recent efforts have scaled MDMs up to 8B parameters (Nie et al., Ye et al. 2025). highlighting their robustness and scalability. Current diffusion language mod- els can be categorized into i) models trained from scratch (Nie et al., [2225] Yang et al., 2025), ii) models adapted from autoregressive (AR) models (Ye al., 2025). and iii block-diffusion mod- els (Cheng et al. 2025 Wang et al. (025d), which combine the training efficiency of AR models with the sampling efficiency of diffusion models.
2.2 INFERENCE-TING OPTIMIZATIONS FOR DIFFUSION LLMS
Recent work investigates inference-tine optimization for diffusion LLMs, focusing on inference ac- celeration and generation quality improvement. For acceleration, existing methods largely fall into two categories:
approximate caching and parallel decoding. Caching approaches reuse key-value (KV) states across denoising steps to reduce redundant computation, including delayed caching (Mal et al. 2025), similarity similarity-guided guided caching (Liu et al.] (025)), and block-wise caching (Wu et al. 2025)
tailored to semi-AR decoding. Parallel decoding methods improve the speed-quality trade-off via
confidence-based dynamic unmasking (Wu et al. 2025 Wang al. 2025e Wei et al. 2025
masking schedule design (Luxembourg et al. 2025). and guided diffusion (Hu et al. 2025 Israel
al. 2025). Beyond speed, test-time strategies such voting (Wang al. 2025b), early commit
decoding (Li et al. 2025a), and remasking- based refinement (Hong et al. 2025 Wang et al. 2025a
He et 2025) have also been explored to improve generation quality.
2.3 BLOCK-WISE SEMI-AUTOREGGRESSVVE DECODING
Seri-autoregreesivve (semi-AR) decoding partitions the sequence into blocks. Decoding is autore- gressive over blocks, but non-autoregressive within each block, enabling tokens in the same block to be decoded in arbitrary order. Block Diffusion (Arriola al. 2025) first introduced this paradigm for dLLMs by interpolating between token-level autoregression and fully non-autoregressive diffu- sion via a block-causee attention mask. Semi-AR decoding has been commonly adopted by dLLMs, including LLaDA (Nie et al.) 2025) and MMaDA (Yang et al. 2025). In prior work, semi-AR de- coding typically uses fixed block size. In contrast, this paper takes the first attempt explore adaptive block-size decoding with a semantic-aware, training-free method.
PRELIMINARIES
The decoding process of diffusion-based LLMS initializes from a fully masked sequence. Each decoding step alternates between two operations: denoise, which predicts token distributions con- ditioned on the current partially masked sequence, and sample, which uses these distributions to unmask masked positions. This denoise-sample cycle is repeated until all positions are unmasked. Using LLaDA (Nie et al. 2023 as an example, we formalize the decoding process below.
Setup. Let V denote the vocabulary, which includes special mask token [MASK] E V. Given prompt q (go, ... QL,-1) 44, and a generation budget L, define the index set & (0, 1,.., Lp+L-11 Let T be the total number of denoise-sample iterations.
At stept t (T 1,....... 0}. the sequence state is y (w/)eS E pLoth, The initial state of the sequence is y (go.... QT,-11 MASK], ...,, [MASK]).
Denoise. A mask predictor predicts sequence Y' E (V) [MASK)) tr using greedy decoding:
gf = arg max Po (u ( i J.
(1)
NE
Sample. Define the masked- masked-position set:
Mr a (ie J: vf [MAS]] }.
(2)
3

---

Published as a conference paper at ICLR 2026
A sampler selects St S M₁ unmask. For all i J, update
M/ [MASK],
i St (unmask),
i Mr \ S: (stay masked),
(3)
i i Mr (already unmasked; keep).
Then, the sequence state becomes
y y-11= (yi""))e E Vlath.
This process repeats for ( = T,7,-,,,,...... and terminates when M, = B.A
Samplers differ primarily in their unmasking schedules and position-selection rules. LLaDD (Nie et al. 2025) proposes linear-schedule sampler that unmasks a fixed number of tokens L/T at each step, selecting positions either uniformly random or by confidence. LaViDa (Li et al. 2025b) applies a shifting schedule that unmasks a variable number of [MASK] positions at each step. Fast- dLLM (Wu et al.
2025) and Dimple (Yu et al. 2025) implement dynamic sampling by introducing
confidence threshold т: each step t. the model unmasks all positions with confidence e' & Pe(" y 2)) that ct 2 т. Compared to the linear-schedule or the shifting schedule, dynamic sampling adapts token-level uncertainty and improves the seuurcy-throughpu balance.
METHODOLOGGY
This work focuses on a widely adopted inference paradigm for diffusion-based LLMs: semi- autoregressive (semi-AR) decoding with dynamic sampling (Wu et al. 2025 This paradigm forces block-causal dependencies, enabling block-wise KV caching and allowing multiple tokens to be decoded within a single denoise-sample iteration. it governed by two key hyperparameters: the confidence threshold 7 and the block size B. The threshold 7 controls the speed-quality trade-off, whereas the impact of B is yet be explored. We systematically analyze the drawbacks of fixed block sizes and propose lightweight sampler that improves sampling quality for dLLMs.
we analyze the confidence dynamics that govern dynamic sampling. Using these patterns, we characterize the inaccuracies and inefficiencies caused by the misalignment between fixed block size and the model's inductive decoding preferences in Section [42]] This motivates an adaptive block-size scheduler introduced in Section 4.3 that minimizes this mismatch.
In Section 4.1
4.1 ANALYSIS OF CONFIDENCE DYNAMICS
Confidence Dynamics. Confidence scores are key metric in dynamic sampling of dLLM.. A high confidence score indicates that the model is more certain about the prediction (Wei et al. 2025). Figure 3 visualizes confidence dynamics at carly, middle, and late decoding stages of LLaDA-8B. Base inference on GSM8K Benchmark. Based on these statistical analyses, we summarize the following observations and patterns for different decoding stages:
For stages, a high-confidence region emerges near the decoded tokens. This indicates that masked positions adjacent to decoded tokens more likely to attain high confidence and thus to be decoded. We attribute this to confidence locality: dLLMs exhibit higher confidence in regions where local semantic meaning is complete.
Although the model may generate in arbitrary order, its decoding trace shows a global autore- gressive tendency. This pattern is consistent with chain-like progression driven by semantic dependencies, where subsequent predictions rely on prior semantics.
As the decoding process unfolds, confidence for decoded tokens remains high, and the high- confidence region extends toward adjacent positions. In contrast, positions outside this region maintain consistently low confidence.
4

---

Published as a conference paper at ICLR 2026
(0 tokens decoded)
164 to dec odded
1128 tolees decoded)
(192 tokens deroded)
1256 tokens decoded)
Contaancc
Confidlance
358214
190226254
Postion
Postion
Postdon
Posidan
Postdon
Modiun lan
25th Percentee Interval
w.w
Decoded Position
Figure 3: Confidence scores across sequence positions for LLaDA-8B-Bass evaluated on 100 sam- ples from the GSM8K benchmark. The high confidence plateau expands as decoding progresses, while positions adjacent to the decoded prefix exhibit high variance.
25
Sample
Sample
Sample 3
22.0%
Percestage (%)
20-
14.5%
11.0%
9.3%
7.7%
12
LONTAE 204
15-23 1.0
- 32)
18 - 201
18
-
321
Doc Posiian
licadary
Premeture Deceding Emror
Lake Decoding Overheed
Figure 4: Illustationn of the high confidence plateau, the volatility band (VB), and the low confidence floor across three samples. Within VB, the distribution of confidence scores and the width of the band vary across samples.
Figure 5: Proportion of sampling steps affected by late decoding overhead and premature decoding error on GSM8K and HumanEval for fixed block sizes.
Volatility Band and Local Stochasticity. Building on these observations, partition the position-wise confidence landscape into three regions: i) a high-confidence plateau with consis- tently high scores, ii) volatility band (VB) characterized by unstable, variable scores, and ii) low-confidence floor with persistently low scores. Among these regions, the VB is the key region where the current decoding steps take place. As illustrated in Figure scores within the VB fluc- tuate over decoding steps (time) and across positions (space), yet remain clearly separated from the low-confidence floor. Additionally, the width of the VB varies from case case. In contrast the left-to-right expansion of the high-confidence plateau that is driven by global autoregressive causal- ity, the decoding order within the VB is locally stochastic: the positional preference diminishes, and the sampling choice depends more heavily on the immediate semantic context.
4.2 MOTIVATION FOR ADAPTIVE BLOCK SIZF.
Despite the local stochasticity exhibited during decoding, prevailing semi-AR decoding strategies use manually set, fixed block sizes that often fail to capture this stochasticity. This misalignment leads to Late Decoding Overhead and Premature Decoding Error, as illustrated in Figure D thereby reducing accuracy and efficiency. This motivates incorporating an adaptive block size.
fixed block size poses a hard constraint on the number of tokens that can be sampled in single step. By fixing the block size, the sampler must exclude nearby higher- confidence positions outside the block, especially when a small block size (Wang et al. 2025d] Cheng et al.) 2025) is used. Deferred high-confidence positions undergo additional denoising in later iterations, incurring unnecessary computation overhead and resulting in degraded throughput.
Late Decoding Overhead.
Premature Decoding Error. With fixed block sizes, the sampler (Algorithm 3) must decode all masked positions in the current block before proceeding to the next block. Under this constraint, the sampler can be forced decode low-confidence positions within the block rather than deferring them in favor of higher-confidence positions in the consequent blocks. As result, the sampler can commit to incorrect tokens early, inducing systematic errors: premature decoding increases token- level error rates, propagates mistakes to subsequent steps by conditioning later blocks these tokens via block-level autoregressive dependencies, and shifts generation toward poorly calibrated regions of the confidence landscape. This usually contributes to degraded accuracy.
5

---

Published as a conference paper at ICLR 2026
Algorithm Semantic-Aware Block Size Determination
predicted sequence y: confidences generation budget L; default block size Bo: delimiter set DD delimiter threshold rp; current position g.
Inputs:
Output: block size B
1: function COMPUTEBLOCKKLENCHHI c, L, Bo, D, TD. g)
2: Sampling window boundary
3: start, remaining 5- 9. I.
4: w min( max(11, [0.25 9]); remaining)
5:
W < {start, .... start + w
window token indices
6:
Find highest-confiience delimiter
7:
I E-= EiE W
D}
8:
I + then
9:
pos < arg m Ci
Select position with max delimiter token confidence
10:
Cmax + Cpos
else
12:
Cmax + -0
end if
14:
Determine block size
if Cmax 2 TD then
16:
B + (pos- start + 1)
D inclusive length to the delimiter token
17:
else
18:
B - min remaining)
19:
end
20: return B
21:
end function
4.3
SEMANTIC-AWARE ADAPTIVE BLOCK-SIZA SCHEDULER
Challenge for predicting block size. As explained in Section the VB delineates the span of positions undergoing active decoding: positions preceding the VB are decoded and exhibit con- sistently high, stable confidence, whereas positions within the VB show pronounced step-to-step fluctuations. Additionally, Figure 4 shows that the denoised predictions the semantic context. In contrast, positions in the low-confidence floor repeatedly predicted as non-content tokens, such as placeholders or formatting symbols.
VB typically relate to
Although the VB region inspires scheduling block-sizes adaptively, it often spans many posi- tions, failing to provide actionable guidance for scheduling. As shown in Figure the token "GB" repeatedly appears with confidence scores mostly between 0.1 and 0.3, placing inside the VB. While such tokens may be contextually related, their significance to the current context is weak, thus offering limited guidance for the current sampling step. Consequently, VB regions alone are often insufficient for local scheduling decisions, motivating finer finer-grained grained segmentation to identify tokens that are most tightly coupled the current decoding step.
Aligning Block Size With Semantic Steps. To obtain fine- fnn-ggained segmentation that reflects the context of the current decoding step, we partition the sequence into semantic steps, which are contiguous spans whose provisional tokens exhibit local semantic coherence. We then couple the scheduler to the semantic step, setting the block size guided by the length of the current semantic step. This allows for finalizing higher-confidence positions within the step while deferring lower- confidence positions until the semantic step is ready to close. Across semantic steps, dependencies are enforced by the semi-AR paradigm, since each downstream step conditions on completed prede- cessors. This alignment curbs premature commitments outside the active step and prevents splitting a step across iterations, thereby reducing both error propagation and computational overhead.
Comantic-Award Block Size Scheduling. To facilitate the dynamic scheduling of block size B. we insert an additional block-size determination procedure (Algorithm []] before sampling the first token of each block. This procedure predicts the block size that aligns with the length of semantic steps. Given the current predicted sequence ý and confidence scores c, we collect indices whose predicted tokens fall in the delimiter set D (line 7). Tokens in the delimiter indicate the end of

---

Published as a conference paper at ICLR 2026
semantic unit and demonstrate sharp confidence drops (Appendix A.4). We choose the delimiter token Mmax with the highest confidence Cmax- If Cmax 2 TD. we set to the position of Ūmax (lines 15-16), indicating that the model has reliable preference for the end of the current semantic step. If no delimiters appear in y within W (lines 11-12); or all delimiter predictions are low- confidence (Cmax < TD. lines 17-18), fall back to the default block size. Additionally, an index window W that masks distant positions to avoid decoding the <EOS> token in the early stage, cause for severe performance drop (Nie et al., 2025). This procedure yields step-aware blocks when evidence is strong, while remaining conservative in ambiguous regions.
apply
EXPERIMENT
5.1 EXPERIMENTAL SETUP
Implementation Details. We evaluate AdaBlock-dI..M representative diffusion LLMs: LLaDA-8B-Instruu (Nie et al. (025). LLaDA-1.5 (Zhu et al. (025). and Dream-v0-Base-78 (Ye
al. 2025). Unless otherwise noted, all experiments run on NVIDIA H100 GPUs.
Hyperparameter Settings. We the generation budget I 512 for all benchmarks. For dynamic sampling, we use confidence threshold = 0.9. We sweep default block sizes B {16, 32, 64). AdaBlock-dLLM introduces two hyperparameters: the delimiter
and the delimiter confidence threshold Jp- We set 'D = } which commonly marks the end of reasoning steps in test-time search (Snell al. 2024), and frequently causes a significant confidence drop (Appendix A.4). We use Tn 0.3 for LLaDA models and Tn 0.5 in Dream models. This is tuned on a small subset of the GSMSK benchmark, the selection discussed in Section 5.3
Benchmarks and Metrics. We evaluate AdaBlock-dLLM on standard LLM benchmarks. For math reasoning, we use GSM8K (5-shot) (Cobbe et al. 2021) and MATH (4-shot) (Hendrycks) et al. 202T). For code generation, we use Hvma (0-shot) (Chen et al. 2021 and MBPP (3-shot) (Austin et al. 2021b). Generation quality is measured by accuracy: pass @ for code gen- cration and answer accuracy for math reasoning. We compare five sampling methods: Vanilla (top-1 confidence sampling); Dynamic (threshold-based dynamic sampling) and +Cache (Dynamic method with DualCache), both following Fast-dLLM (Wu et al., (205);; and two variants enhanced with AdaBlock-dLLM, +Ada (Dynamic method with adaptive block sizing) and +Ada+Cache (+Cache method with adaptive block sizing).
5.2 MAIN RESULTS
Generation Quality Across Models. Table I reports accuracy to quantify generation quality. AdaBlock-dLLM achieves accuracy gains across most model and dataset pairs. Notably, on GSM8K with LLaDA accuracy improves by 3.0% without caching and by 5.3% with caching.
GSMBK
MATH
HumanEval
MBPP
82
37.6
50
Parato Optimal
Accurscy
19
37.0
46
76-
35.74
74
Throughput (takensns)
(hakennsr))
(tokennds
(tolensgr))
Vanilla
Dynamic
Dynamic + Cache
Dynamic + AdsBlook
Dynamic
Adtallock- Cathe
Pareto Fronber
Figure 6: Accuracy and throughput different sampling methods evaluated LLaDA-Instruct In- tegrating AdaBlock-dLLM, into Fast-dLLM (Wu et al. 2025) yields accuracy gains across all bench- marks while maintaining little throughput overhead. Notably, Fast-dLLM with AdaBlock-dil dLLM is Pareto-optimal on the GSM8K and MATH datasets.
7

---

Published as a conference paper at ICLR 2026
Table I: Accuracy (%) across sampling methods, evaluated on LLaDA-1.5, LLaDA-Instruc,, and Fream-Base under default block sizes Bu {16, 32, 64). Differences are shown in gray. Compar- isons are reported relative Dynamic and +Cache (Wu al. 2025).
LLaDA-Instruce
LLaDA-1.5
Dream-Bant
Method
Bo
16
Bg
32
Bo
64
Bo
16
Bg
32
Bo 64
Bo 16
Bg
32
Bg 64
GSM8K
Vanilla
78.8
76.7
76.8
82.3
82.3
80.4
76.3
76.4
75.1
Dynamic
79.1
77.6
77.3
82.6
82.2
80.7
75.5
75.5
75.6
+Ada
80.6+1155
80.6 +3.0
79.5 +2.2
83.0
82,4+0.2
80.3-0.4
75.7 +0.2
75.7 +0.2
75.9 +0.3
+Cache
78.0
74.5
75.4
80.7
80.2
80.0
75.6
74.5
74.6
AAda+Cache
80.0
78.5
80.7
81.3
81.7
79.7
76.5 +0.9
75.1 40.6
74.6+0.0
EvmanEval
Vanilla
43.9
43,9
42.7
39.0
36.6
38.4
53.7
52.4
54.3
Dynamic
42.7
43,9
43.3
36.6
37.8
36.6
53.0
51.2
52.4
+Ada
43.3 40.6
43.3 -0.6
43.9 +0.6
37.8
38.4
38.4+1.8
53.7 +0.7
51.2
53.7 +1.
+Cache
45.1
46.3
47.0
33.5
36.0
34.1
50.0
53.0
56.1
CAda+Cache
49.4+4.x
46.3 +0.0
48.2+1.2
36.0+2.5
39.0 +3.0
36.0
52.4 +2.4
53.0 +0.0
57.3 +1.2
MATH
Vanilla
36.7
36.9
37.3
36.3
37,0
34.4
39.8
40.2
40.1
Dynamic
37.0
36.9
37.1
36.3
36.7
34.4
39.7
39.9
39,9
+Ada
36.5 -0.5
37.3 +0.4
37.4+00.
36.8 +0.5
36.7 +0.0
34.1 -0.3
39.6 -0.1
39.9
39.9 +0.0
Cache
35.4
35.8
36.0
34.9
33,2
32.1
38.0
38.5
38,8
Ada+Cache
35.8 40.4
35.3 -0.
35.6
35.2 +0.3
33.9
32.4
37.8 -02
38.4
38.4
MBPP
Vanilla
39.2
38.6
37.0
38.2
37.0
23.2
12.4
12.4
12.8
Dynamic
39.8
38.8
36.8
38.2
37.0
24.6
12.6
12.4
12.2
+Ada
40.2
39.8 +1.0
36.4 -.4
39,4+1.2 2
37.6
29.8
12.8 +0.2
14.2+1.8
12.4
+Cache
35.6
37.8
36.4
38.0
34.8
19.8
12.8
+Ada+Cache
39.4 +3.8
38.0
+0.2
36.8 +0.4
36.6-1.4
36.4
36.6+6.8
12.8 +0.0
11.6
12.4 +2.8
Pronounced Accuracy Improvement With Cache. We observe that accuracy gains are particu- larly pronounced when KV caching applied. Unlike autoregressive decoding, where eaching effectively lossless, block-wise KV caching in dLLMMs is an approximation because key and value tensors vary across time steps, and the decoding order within each block is non-sequential. This approximation results in notable accuracy degradation at large block sizes.
Table 3 reports accuracy gains with both Pache and DualCache (Wu al. 2025). Improve- ments are twofold: i) For large default block sizes Bg. the resulting average block size B is smaller, reducing the approximation error of KV Caching. ii) By enhancing semantic locality within each block, inter-block dependencies
reduced, making decoding less sensitive to stale cached tensors. The second effect is more impactful: on GSM8K, B for Bg 64 is 33.98, yet accuracy still exceeds that of Bo 32 without AdaBlock-dLLMM by 1.90% (no cache) and 6.20% (DualCache). Given that block-wise KV caching is core advantage of semi-AR decoding, these results indicate that the method integrates seamlessly with existing techniques that improve inference efficiency.
Discussion Difference Between Models. The gains from AdaBlock-dLLM vary across models. In particular, AdaBlock-dLLAN yields larger improvements on the LLaDA family. By predicting block size informed by local semanties, AdaBlock-dLLMM groups tokens that constitute semantic step into the same block and focuses on refining the local context. This effect strongest for dLLMs trained from scratch, which exhibit greater local stochasticity. In contrast, dLLMs adapted from AR models display global autoregressive order and a high degree of local autoregressiveness (Gong 2025). In such scenario, the improvements from AdaBlock-dI.AA are correspondingly smaller. Additionally, the generation quality is also limited by the model's denoising quality, while our work focuses on mitigating the performance loss of sampling within fixed-sizes blocks.
Discussion Throughput Overhead. Table reports the accuracy, throughput (measured in tokens per second, TPS), and the average number of function evaluations (NFE). The product of throughput and NEE remains stable across methods and block sizes, indicating approximately
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison across default block sizes Bo under generation budget of
L = 512. The product of throughput and NFE is nearly identical across methods and block sizes, indicating an approximately inverse relationship between these quantities when no block-wise KV caching is applied. AdaBlock-dLLM yields throughput gains for Bo E {4,8). Boldface indicates superior performance; this convention applies to all tables unless noted otherwise.
Acc. (%)
TPS
Avg. NFE
TPSxNFE
Ace. (%)
TPS
Avg. NFE
TFSxNFE (10*)
Acc. (%)
TPS
Avg. NFE
TFSx (10")
(10")
Method
Bo 4
Bo =
Bo = 16
Vanilla
80.9
16.1
512.0
8.2
80.5
16.1
512.0
8.2
78.8
16.1
512.0
8.2
Dynamic
81.2
43.0
189.4
8.2
80.6
60.0
135.5
8.1
79.1
74.7
109.2
8.2
+Ada
81.6
51.3
159.8
81.8
63.9
128.3
8.2
80.6
73.9
102.4
8.2
Method
Bo
32
Bg
Bo = 128
Vanilla
76.8
16.1
512.0
8.2
76.8
16.1
512.0
8.2
71.0
16.0
512.0
8.2
Dynamic
77.6
85,0
94.9
8.1
77.3
89.4
91.6
8.2
70.7
81.2
101.2
8.2
+Ada
80.6
83.5
98.5
8.2
79.5
87.9
93.4
8.2
72.8
80.6
101.6
8.2
Table 3: Accuracy (%) on GSM8K for LLaDA-
Table 4: Accuracy (%) on GSM8K for LLaDA- Instruct under different generation budgets.
Instruct across caching methods with L 512.
Method
Bo 16
Bo = 32
Bg 61
Method
= 256
= 512
L 1024
+PrefixCache
78.2
76.9
75.0
Dynamic
78.1
77.6
77.5
Prd+PrrefiCaach
81.4
79.8
77.6
+Ada
78.5
80.6
79.3
+DualCache 78.0
74.5
75.4
+Cache
77.4
74.5
75.8
DuacahhalCachhe
80.0
78.5
80.7
+Ada+Cache
79.2
78.5
78.1
inverse relationship between these metrics. This observation suggests that throughput can be im- proved primarily by reducing NFE, for example, by increasing the parallelism capacity of the de- noiser (Wang et al. 2025c) or by improving the sampling efficiency of the sampler.
We observe that vanilla decoding, which fixes NFE to the generation budget, maintains almost iden- tical throughput across block sizes. In contrast, dynamic sampling with either fixed or adaptive block sizes benefits from increasing Bg from 64, but throughput drops when Bo is increased further. We attribute this initial positive correlation between block size and throughput to the mitigation of late decoding overhead. The subsequent degradation in throughput arises from less reliable denoiser predictions under large blocks: as Bo grows, within-block semantic coherence weakens, fewer to- kens meet the sampling criterion per step, and dynamic sampling therefore decodes fewer tokens per iteration, increasing NFE. Since throughput is approximately inversely proportional to NFE when per-iteration cost nearly constant, the higher NFE reduces throughput. This effect consistent with the accuracy trend, where large block size (Bo = 128) exhibits marked performance drop.
With AdaBlock-dl.LM throughput increases for small default block sizes (By E (4, 8)) and de- creases slightly for larger defaults (Bo {16,32 32, 64, 128)). AdaBlock-dLLM attempts to align the block size with the length of semantic steps, which yields B Bo on average when the default is small and B < Bg on average when the default is large. For small By, AdaBlock-dLLM effectively reduces late decoding overhead. As By increases, selecting B < By strengthens within-block co- herence and typically improves sampling quality, at the cost of modest throughput reduction. Nev- ertheless, Dynamic+Ada consistently achieves higher accuracy than Dynamic while maintaining comparable throughput. Finugg 6 presents the trade-off between accuracy and throughput, demon- strating the improvements induced by our method.
5.3 ABLATION STUDIES
Performance Across Different Generation Budgets. We evaluate AdaBlock-dI.LMM under three generation budgets: I. E (256, 512, 1024), as shown in Table 4 Across all generation budgets and decoding settings, AdaBlock-dLLM improves generation quality. These results motivate semantics-aware block-size scheduling design for dLLMs.
Effect on Delimiter Threshold Tp. We evaluate three delimiter thresholds for each model family, shown in Table We observe that TD 0.3 yields the best performance in most cases for

---

Published as a conference paper at ICLR 2026
Table 5: Accuracy (%) on GSM8K for LLaDA and Dream across delimiter thresholds E {0.3, 0.5, 0.7) with Bo 32. A smaller To provides sufficient semantic guidance for dLLMs trained from scratch (e.g., LLaDA).
Table 7: Accuracy (%) on GSM8K
across eight different delimiter sets with By 32. Results show that using the newline token ('n) as the delimiter accounts for most of the accuracy gains, while addition-
Model
TD = 0.3
TD = 0.5
TD = 0.7
alDA--Istruct
80.59
79.08
77.94
ally including the comma and pe-
Dream-Base
75.66
75.74
75.74
riod further improves performance.
Delimiter Set
Acc. (%)
Table 6: Accuracy (%) on IFEval for LLaDA-1.5 across sam- pling methods. AdaBlock-dLLM also improves performance.
None (+Cache)
74.5
[[n
78.5
Method
Bo 16
Bp = 32
Bo = 64
[[,]]
75.1
Vanilla
69.1
66.7
61.3
[[.]]
74.5
Dynamic
69.0
66.7
61.2
[[,,, [.]]
75.1
+AAa
68.4.0.6
67.5
64,4+3.2
[[)p), []]
78.5
+Cache
67.5
64.6
59.4
[[)n), [.]]
78.3
Ada+Cache
68.9 +1.4
66.4
62.7
[[)n [], [-1]
78.7
LLaDA, whereas higher 7D 0.5 is optimal for Dream. We attribute this difference the distinct confidence distributions of the two models. L.LaDA is trained purely from scratch and exhibits lower variance within the volatility band, whereas Dream is adapted from autoregressive models and shows substantially higher variance (Figure 8). Consequently, different thresholds are required to track the boundary of a semantic step. Additionally, overly high thresholds (e.g., TD 0.9) often cause the scheduler to revert to its default behavior, reducing its effectiveness.
Selection of Delimiter Set D. We apply more delimiter sets DD to include additional tokens (comma and period), which often mark the termination of local semantic context. Table 7 shows that although accuracy improvements vary, the inclusion of the comma and period achieves higher accuracy than the dynamic sampling baseline. These results highlight the importance of aligning block size with semantic steps semi-AR decoding. We provide further analysis in A.4
Performance on Non-Reasoning Benchmarks. We further evaluate the performance of AdaBlock-dLLM on IFEval (Zhou et al., 2023). a benchmark that examines the instruction-follloinng capability of LLMs. As shown Table with the delimiter set D [[)n, [,,, [.]), [.),, AdaBlock-dLLLM yields accuracy improvements, especially when integrated with block-wise KV caching. These results suggest that aligning block sizes with semantic steps effectively enhances sampling quality, leading to improved performance tasks other than math reasoning or coding.
Limitations. AdaBlock-dlLLM effectively augments the existing semi-AR decoding paradigm by aligning block sizes with semantic steps. However, the proposed block scheduler may be less effec- tive when the generation budget is small (e.g.. multiple-choice questions), where semi-AR decoding not particularly beneficial. Moreover, dLLM decoding alternates denoising and sampling, and AdaBlock-dil dLLM primarily improves sampling quality. When the denoiser's 's predicted token distri- butions are unreliable, adaptive block sizing cannot recover quality, and the advantage of AdaBlock- dLLM shrinks. Future work also includes automating the choice of delimiter tokens and their asso- ciated delimiter threshold, rather than relying on fixed heuristics.
CONCLUSION
This work proposes AdaBlock-dl.LM, training-free, plug-and-play scheduler that enhances the existing semi-autoregressive decoding paradigm. We identify two fundamental limitations of con- ventional semi-AR decoding (late decoding overhead and premature decoding error) that motivate adaptive block-size scheduling. Building adaptively adjusts the block size at runtime, aligning block sizes with semantic steps. Extensive ex- periments across benchmarks demonstrate improvements in generation quality of up 5.3% under a comparable throughput budget. We hope that our semantic-awe adaptive approach and statistical analysis will inspire future training and inference strategies for dI dL..Ms.
analysis of confidence dynamics, AdaBlock-dLLM
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no research involving human subjects or animals conducted. All datasets used, including GSM8K, MATH, HumanEval, and MBPP, were obtained in compliance with relevant usage guidelines, ensuring no violations of privacy. We took care to avoid the bias and discriminatory outcomes throughout our research process. No person- ally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We ensure that all reported results reproducible. The code repository is open-sourced to facilitate replication and verification. The experimental setup, including model configurations and hardware details, is described in Section[5.1 We also provide full description of our methodology. including detailed pseudocode in Algorithm D to support understanding and reproduction. Additionally, the open-source models used in this work (e.g., LLaDA and Dream) are publicly available, enabling consistent and reproducible evaluation.
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. In The Thirteenth International Conference Learning Representations, 2025. URL https:://// mtte://axxiii arxiv. org/ abs / 2503. .05773
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Struc- tured denoising diffusion models discrete state-spaces. In NeurIPS, 2021a. URL https:
//arxiv.orr/abs//21ll 63006)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint Stt::2100.0073322 2021b.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03::
2021.
Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Hellen" Li, and Yiran Chen. Dpad: Efficient diffusion language models with suffix dropout, 2025. URL. luttps://arxxt org.
www./a///////oooooo/////////aa.a abss 2508 14148
Shuang Cheng, Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Linfeng Zhang, Wenghai Wang, Qipeng Guo, Kai Chen, Biging Qi*, and Bowen Zhou. Sdar: A synergistic dif- austor-aatooregrsssionn paradigm for scalable sequence generation, 2025. URL https://
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 11Xv:22110.1416. 2021.
Gemini Diffusion. Gemini diffusion: state-of-the-arr, experimmttl text diffusion model. https: dddepmmmi....oogge//moeee////end
2025. Accessed: 2025-09-22.
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu. Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code gen- eration. arXiv preprint arXiv:2506. 20639, 2025.
Haoyu He, Katrin Renz, Yong Cao, and Andreas Geiger. Mdpo:
Overcoming the divide of masked diffusion language models. arXiv preprint arXiv:2508. 13148, 2025.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 34th Con- ference on Neural Information Processi, 2020. URL httpss//aa......rrr/a//////////a
11239
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633- 8646, 2022.
Feng Hong, Geng Yu, Yushi Ye, Haicheng Huang, Huangjic Zheng, Ya Zhang, Yanfeng Wang, and Jiangchao Yao. Wide-in, narrow-out: Revokable decoding for efficient and effective dllms. arXiv preprint 2tti::2507..857777 2025.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. arXiv preprint arXiv:2102.05379,
2021. BRLL:tps::/ http///axivv... /ttp///xwwv...//////////...... 05379
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. arXiv preprint arXiv: 25t5::25552224000 1467, 2025.
Daniel Israel, Guy Van den Brocck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding. arXiv preprint arXiv:2506.0044 2025.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-
based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan, Li Shen, Yi Liang, Soroush Vosoughi, and Shiwei Liu. Diffusion language models know the answer before decoding. arXiv preprint arXiv:2508. 19982, 2025a.
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: large diffusion language model for multimodal understanding. arXiv preprint arXiv:2505.16833 2025b.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprini arXiv:2506.0622 2025.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffussion modeling by estimating the ratios of the data distribution, 2024. URL pattps://ee ::ta://axiiii wwtp/rrg//.aroeeeeo.oeeeeeeeeeee 2310. 16834
Omer Luxembourg, Haim Permuter, and Eliya Nachmani. Plan for speed-dilated scheduling for masked diffusion language models. arXiv preprint 2iXi:22506619037., 2025.
Xinyin Ma, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint 2XXi:2505.157811 2025.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint 25X.:22502.09992. 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2023.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffu- sion beats autoregressive data-constrained settings. arXiv preprint arXiv:2507.15857, 2025.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033/4 2024.
12

---

Published as a conference paper at ICLR 2026
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. ICML, 2015. URL https:
/atxii..rr//ab//550333 03585
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion Ilms with dynamic cache eviction. arXiv preprint ar0iv:2508. X11.:2508880558,, 2025a.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuchang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.02 02 193, 2025b.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference- scaling. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025a.
Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang. Hao Chen, and Chunhua Shen. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint arXiv:2508.09132 2025b.
Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion Ilms can do faster-than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09/92, 2025c.
Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, Ke Shen, and Mengdi Wang. Revolutionizing inforcement learning framework for diffusion large language models, 2025d. URL https:
06949
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang. Accelerating dif- fusion large language models with slowfast: The three golden principles. arXiv preprint arXiv:2506. 10848. 2025.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training- acceleration of diffusion 1lm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.2221 226118, 2025.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. /5487, 2025.
Runpeng Yu, Xinyin Ma, and Xinehao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding, 2025. URL
Lingzhe Zhang, Liancheng Fang, Chiming Duan, Minghua He, Leyi Pan, Pei Xiao, Shiyu Huang, Yumpeng Zhai, Xuming Hu, Philip Yu, et al. A survey on parallel text generation: From parallel decoding diffusion language models. arXiv preprint arXiv:2508.08712 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooiing evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Fenggi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint 2rXiv 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 CASE STUDY FOR THE Two FUNDAMENTAL ISSUES IN SEMI-AUTOREGRESST DECODING
Prompt:
Carla downlooding 200 68 file. Normelly she con downlood 2 68/minute, but 40% of the way through the downlood Windows forces restort instail updates, which takes 20 minutes. Then Canla has restart the download from the beginning. How load doas take to downlood the file?
Decoded Tokens
Decoded Tokens
Firet, let's determine how long would take Carls to download the entire
First. let' s determine how long it would take Corlo to download the entire 200 68 file without any interruptions.
200 6B file without any interruptions.
Carla's downlood speed 69 per minuts. download 200 68, she would need:
Carla's 's download speed is minute. To downlood 200 68, would need: M kfrac(200 Mtext( GB]K2 Meexti 6B/mincte]) 100 Vtext( minutes) \]
\fraae
(56 decoded)
40% wey through download. beginning
has restort the restart. (120 tokens decoded)
Current Block (Block No. 8) Denoised Sequence
Current Block (Block No. 16)
Denoised Sequence
Block Start 2(1 201.000 000000) 01.000) M0.988) text(0.980) ((0,933) 68(0.977)
Block Start The(1.000) Sownlood:0.677 time(0.992)) for(0.992) the(10000)
(M(0.984)
first 977) spacex(0.973) 4(0.559) Black End tokens) Taken
Block End tokens)
More tokens can be decoded without on additional denoising step
low confidence decoded, sub- sub-optimal sampling
2(0.988) V.0,800) (.99) (10 (4.949) 63(0.965) (m44( (te(0.703)
1/0 6406 of(0.555) the(0.531) file(0.203) (s(0.277)
(](.969) +(0 (0.957) (apecio 949) 100 (388938)0000 dio 941) (4.944) MG.949)
:00.399) Ve. (.244)
tExt(0.934) ((0.945) minutes(( (30000033)))) XC.871) (008115 (40) Ve(0.
Optimal Block End (15 tokens)
(22).922)
Optimal Block (30 tokens)
"GB" tokens the Volatility Band,
significance ([8.188) ((0346) 0(0.124) 0(0.177) 68/01162 1866).
(v(0.240), (1040)_004490 (0.492) (33) need(0.152) (o55)) the00.140)
M.0.777) 18.184)
the 0.209) the(0.16) download 0.10)) the(0.087)
downlocd 204)
68/0.200)
680
2400)
j0 102) dowelose 089) (2 sppcee(O.0 093) oro (x0)097) 097) 0/0 104)
6A(0.299)
68.0.386)
G8/0 (65) 68:0.136) 68(0.142)
010.116) 0.0.112) 0(0.111) 000.108) 0.0 010.111) 0(0.096) 0:0.104)
68/0 68 M0.180) VO 131) { 143) minutes/O.1300
103) 111 0(0.108) 0(0.112) 1(3.123) 114,144) 0.03009 000.108)
NO 162) (1.)14) 12.) 0.18198) An(D 198) AejO 143) AnjO 0913 An(0.075)
\w(0.666) \n(0.062) sspcce minutes (0.07e)(0.067))
Late Decoding Overhead
Premature Decoding Error
Figure 7: A case study of the two fundamental issues (Late Decoding Overhead and Premature Decoding Error). The configuration uses dynamic sampling with a generation budget of = 512 and block size of B = 8.
A.2 SEMI-AR DECODING WITH ADAPTIVE BLOCK- -IZE SCHEDULING ALGORITHM
Algorithm Auxiliary: Denoiser
Inputs: mask predictor pe: vocabulary V: index J: current sequence y. Outputs: predicted sequence ý; confidences e
1: function DENOISER V, J, y)
2: for i E I do
0 < arg max,ey po y, 4: G <- max Po(u i)
3:
5: end for
6: return y,
7:
end function
14

---

Published as a conference paper at ICLR 2026
Algorithm Auxiliary: Threshold Brrese Dynamic Sampling
Inputs: current sequence y; predicted sequence y; confidences c; unmasking threshold 7;
index set.
Output: sampled index set S
1: function THRRESHOLD-SAMPLEE,, y, C, T; J)
2:
S+
3: Jmusk {ie E 3 [MASK]
Select all masked positions in the block
4: Select index dop with highest confidence qi
5:
S
SU hop
6:
for i E I do
7:
if
A C 2 r then S + JU {i}
Tmask
8:
end 10: end for
9:
11: return
12: end function
Algorithm& Semi-AR Decoding with Adaptive Block-Size Scheduling
Inputs: mask predictor po: vocabulary V; initial sequence (T) with index J: generation budget L; default block size Bg; delimiter set D; delimiter threshold Tp; unmasking threshold
Output: decoded sequence
1: generated length g O, timestep & t- T
2: while <L t do
3:
4:
First denoising to obtain predicted sequence and confidence at step I.
5:
(y(0, c(0) DENOISER (po, V, J, (1))
6:
7:
Compute block size
8:
B + COMPUTIBLOCKLENCTHII c(t), L, Bo, D, TD. 9)
9:
Joik < gt, .e.........--1- B+B-1}
10:
11:
First sample
S THRESHOOLDSAMMPEEYY. (9,
(0, y(1), c(1), Tr Joik)
13:
for i E S do
Vi 0f
sample tokens with high confidence
15:
end for
17:
Vj (1-1) + Vj (1) Vj $ S, t+t-1
> Copy other tokens
18:
Jmask EiE Jolk yi [MASK]
19:
20:
In-block denoise- cycles
21:
while Juask + 9 t do
22:
(t))), c (
DENOISER(PE. V, Jmask y¹
23:
S THRESHOLL-SAMPLEEYY 5(1), cl1) T, Imask)
24:
for i E do
(1-1)
0. (1)
25:
Vi
26:
end for
(4) Vj $ 1+1-1
27:
Vj
+ 1j
28:
Imask
(i E Joik
!! [MASK]
(!)
29:
end while
31:
4- + B
32:
end while
33:
34: return y'
15

---

Published as a conference paper at ICLR 2026
A.3
CONFIDENCE DYNAMICS FOR DREAM-BAL BASE
(0 tokens decoded)
164 tolens decoded)
1128 tokers deccded)
(192 tokens decoded)
1256 tokers decoded)
Confoanca So0e
Conforennce Score
Contorni
Conforrcc
0.3
22 64
90
20
306
201255
60
32
64
148
L00
00 LB 302 20230926
0.
35
04
05 12 12221220
Figure 8: Confidence scores samples from the GSM8K benchmark. Adapted models such as Dream exhibit a similar degree of global autoregressiveness LLaDA, but with higher variance in the volatility band. This increased variance motivates the use of a higher delimiter threshold Tn to provide stronger semantic guidance.
sequence positions for Dream-v0- -0---s--BBBBBB evaluated on 100
A.4 FURTHER ANALYSIS OF THE DELIMITER SET
The choice of in as a delimiter token is supported by statistics of confidence drops between consec- utive positions. We measure a confidence drop as the decrease in token-level confidence from one position to the next. Large drops within the volatility band are correlated with semantic transitions and thus provide heuristic for segmenting semantic steps. In sample of 100 GSM8K examples, An is the token that most frequently leads to large confidence drops, consistent with its role as boundary marker. Further, Table lists additional frequenry tokens associated with large con- fidence drops (e.g., commas and periods); incorporating these tokens into the delimiter set improves sampling accuracy our ablations.
Large
Medium (0.1-0.2)
Simal (<00.1
frequercy Confiiinnce
OR
201
Token
Figure 9: The frequency of confidence drops between consecutive tokens.
A.5
THE USE OF LARGE LANGUAGE MODELS
LLMS were used for polishing the manuscript. Specifically, we used an LLM to assist in correcting grammar errors improve readability. important to note that the L.IM was not involved in the ideation, research methodology, experimental design. All intellectual contributions and scientific ideas developed in this work originated from the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis. The authors take full responsibility for the content of the manuscript, including any text polished by the LLM. We have ensured that the LLM- -ooishedd text adheres to ethical guidelines and does not contribute to plagiarism or scientific misconduct.
16


Reviews:
[{"review_id": "MIp4G97tqX", "reviewer": "Reviewer_THAi", "summary": "This paper presents AdaBlock-dLLM, a semantic-aware adaptive block-size decoding method for diffusion-based large language models (dLLMs). By analyzing confidence dynamics during denoising, the authors propose AdaBlock-dLLM, which is a training-free, plug-and-play scheduler that dynamically adjusts block boundaries according to semantic step length. Extensive experiments on benchmarks show that the proposed approach maintains compatibility with caching mechanisms and improves both efficiency and semantic consistency in diffusion LLM inference.", "questions": "Please refer to weakness for more details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes an adaptive, training-free approach that integrates seamlessly with existing diffusion LLM frameworks and improves accuracy without retraining.\n\n2. This paper provides a clear and well-motivated analysis of confidence dynamics and their connection to semantic structures in dLLM decoding, then uses the confidence score of delimiter to decide the block size adaptively.", "weaknesses": "1. The heuristic nature of the delimiter-based semantic segmentation might not generalize well to less structured text other than math or coding problems. The authors should evaluate on more diverse test sets. For example, what if the output text is expected to be a long section and there's no '\\n' in the output?\n\n2.  Figure 6 can hardly be viewed as a pareto-frontier, as the throughput nearly does not change as the accuracy is increasing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sQUlurkjnd", "reviewer": "Reviewer_c59M", "summary": "This paper identifies that fixed block sizes in semi-autoregressive (semi-AR) dLLM decoding cause \"late decoding overhead\" and \"premature decoding error\". It proposes AdaBlock-dLLM, a training-free scheduler that adaptively aligns block boundaries with semantic steps based on delimiter token confidence. This method improves accuracy by up to 5.3% without sacrificing throughput.", "questions": "Given the marginal benefit of adding more delimiters (Table 6), did you consider a more automated or learned method to identify semantic boundaries instead of a predefined list?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper systematically analyze and address the limitations of the fixed block size assumption in semi-AR decoding.\n\n- The core problems (late overhead, premature error) are clearly identified and illustrated. The proposed solution is simple, intuitive, and well-supported by experiments.\n\n- The method is practical (training-free, plug-and-play) and shows clear accuracy improvements on the accuracy-throughput frontier, especially when combined with KV caching.", "weaknesses": "- The method introduces new hyperparameters that require model-specific tuning, slightly weakening the \"plug-and-play\" claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7v1A9hwxs3", "reviewer": "Reviewer_e6SK", "summary": "This paper introduces a method to adaptively select the left and right boundaries for the block in the semi-AR setting for dLLM. The main idea come from the analysis of the confidence dynamics, where inside a volatility band area, the confidence fluctuates dynamically, and the VB regions exhibit semantics structure. Then the author propose to collect indices whose predicted tokens fall in the delimiter set to determine the block size. Experiments show that the performance exceeds the one in DualCache.", "questions": "1. See Weaknesses.\n2. The way of selecting the delimiter set can be explored more. For traditional NLP tasks, there are a lot of ways (semantic parsing) to know different types to separate the semantics of a sentence, not only by the rough way of selecting \\n, [,], and [.].", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a new method to solve the problem of how to adaptively decide the block size and also its position in dLLMs.\n2. The paper is well-written with sufficient analysis for the motivation.\n3. The proposed method is effective compared to the baseline.", "weaknesses": "1. Adaptive block size is important, but the authors didn't show any other baseline for different ways to adaptively deciding the block size. They are other ways that can also achieve the adaptive block size, like some naive and straightforward ways you can just expanding the block size by using a sliding window and track the latest position of tokens that are not decoded as the left index. I think adding more analysis and comparisons with different ways to achieve adaptive block is needed.\n2. Why the performance for MBPP on Dream-Base is only 12. The results from the official report is ~55%. It's a large discrepancy.\n3. I am confusing about the result in Table 2. In Table 2, the TPS first increases and then decreases as you choose a larger bock size. In the analysis, the authors mentioned that it may because the increase of NFE, but the thing matters is the ratio between NFE and Block size. Can you provide more analysis on this.\n4. I think the contribution of this submission is not that meet the standard of ICLR. I fully agree that adaptive block is needed, but the contribution of the method is only to choose the block size by the delimiter set, and they are a lot of papers discussing the confidence dynamics in dllm decoding. The idea is interesting, but it's too simple and straightforward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihVGh2vEjb", "reviewer": "Reviewer_caBK", "summary": "This paper investigates the limitations of fixed block-size semi-autoregressive decoding in diffusion-based large language models (dLLMs). The authors identify two inefficiencies — Late Decoding Overhead and Premature Decoding Error — that arise when fixed-size blocks fail to align with semantic structure during decoding. To address this, they propose AdaBlock-dLLM, a training-free, plug-and-play adaptive scheduler that dynamically adjusts block size according to semantic cues and confidence scores during inference. The method leverages a novel concept called the Volatility Band (VB) — regions of fluctuating confidence that correspond to evolving semantic steps — to determine when to expand or contract blocks.", "questions": "Technical Concerns/Questions and Points to Address in Rebuttal:\n\n- Line 53 lacks evidence and most be supported through experiments or citations to prior work.\n\n- How is this work different from [1] ?\n\n- The concept of the Volatility Band (VB) appears intuitive—tokens near already decoded regions naturally exhibit higher confidence—so unless counterexamples or non-trivial cases are shown, the finding risks seeming self-evident rather than novel.\n\n\nReferences:\n\n[1] Wang, Xu, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. \"Diffusion llms can do faster-than-ar inference via discrete diffusion forcing.\" arXiv preprint arXiv:2508.09192 (2025).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well motivated. The paper convincingly articulates the inefficiencies of fixed block-size decoding in diffusion LLMs.\n\n- The adaptive block-size scheduler is conceptually elegant, lightweight, and compatible with existing architectures.\n\n- The authors provide extensive results across multiple models and datasets. The method yields consistent accuracy improvements, particularly when combined with caching.", "weaknesses": "- The experiments focus mainly on math  and code generation. Broader text generation tasks (e.g., summarization or translation) and harder math reasoning tasks such as AIME would strengthen claims of generality.\n\n- At larger block sizes, AdaBlock-dLLM shows reduced throughput compared to the baseline, raising concerns about its scalability for high-speed inference.\n\n- The delimiter threshold (τ_D) and delimiter set (D) are manually tuned per model family. A more principled or automated way to select these would increase robustness.\n\n- The paper lacks a discussion of situations when adaptive block size might fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)