TEST-TIME SCALING IN DIFFUSION LLMS VIA HIDDEN SEMI-AUTOREGRESSIVE EXPERTS
Authors: Jihoon Lee
Hoyeon Moon
Kevin Zhai
Arun Kumar Chithanar
Anit Kumar Sahu
Soummya Kar
Chul Lee
Souradip Chakraborty
Amrit Singh Bedi
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Large Language Models
reasoning
inference time
Primary area: generative models

Abstract: Diffusion-based large language models (dLLMs) are trained to model extreme flexibility/dependence in the data-distribution; however, how to best utilize this at inference time remains an open problem. In this work, we uncover an interesting property of these models: dLLMs {trained on textual data} implicitly learn a mixture of semi-autoregressive experts, where different generation orders reveal different specialized behaviors. We show that committing to any single, fixed inference time schedule, a common practice, collapses performance by failing to leverage this latent ensemble. To address this, we introduce HEX (Hidden semi-autoregressive EXperts for test-time scaling), a training-free inference method that ensembles across heterogeneous block schedules. By doing a majority vote over diverse block-sized generation paths, HEX robustly avoids failure modes associated with any single fixed schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to 3.56× (from 24.72\% to 88.10\%), outperforming top-K margin inference and specialized fine-tuned methods like GRPO, without additional training. HEX even yields significant gains on MATH benchmark from 16.40\% to 40.00\%, scientific reasoning on ARC-C from 54.18\% to 87.80\%, and TruthfulQA from 28.36\% to 57.46\%. Our results establish test-time scaling as a powerful principle for dLLMs, showing that the sequence in which masking is done can play a significant role in test-time scaling/inferencing of dLLMs.

Published as a conference paper at ICLR 2026
TEST-TIME SCALING IN DIFFUSION LLMs VIA HIDDEN AEMO-AUUOREGRESSIVVE EXPERTS
Jihoon Lee', Hoyeon Moon', Kevin Zhai Arun Kumar Chithanar, Anit Kumar Sahu², Soummya Kar Chul Lee, Souradip Chakraborty Amrit Singh Bedi
Yonsei University, 2Oracle, Caanngiee Mellon University, University of Maryland, University of Central Florida
ABSTRACT
Diffusion-bast large language models (dLLMs) are trained to model extreme fexibblityydeeeedennce in the data-distribution; however, how to best utilize this inference time remains an open problem. In this work, we uncover an interest- ing property of these models: dLLMs trained on textual data implicitly learn a mixture of remi-autoregress different specialized behaviors. We show that committing to any single, fixed in- ference time schedule, common practice, collapses performance by failing to leverage this latent ensemble. To address this, we introduce HEX (Hidden semi- autoregressive EXperts for test-time scaling), a training- free inference method that ensembles across heterogeneous block schedules. By doing a majority vote over diverse block-sized generation paths, HEX robustly avoids failure modes associ- ated with any single fixed schedule. On reasoning benchmarks such as GSM8K, it boosts accuracy by up to 3.56x (from 24.72% 88.1 10%), outperforming top-K margin inference and specialized fine-tuned methods like GRPO, without addi- tional training. HEX even yields significant gains on MATH benchmark from 16.40% to 40.00%, scientific reasoning on ARC-C from 54.18% to 87.80% and TruthfulQA from 28.36% to 57.46%. Our results establish test-time scaling a powerful principle for dLLMs, showing that the sequence in which masking is done can play significant role in test-time scaling/inferencing of dLLMs.!
experts, where different generation orders reveal
INTRODUCTION
Diffusion-basee large language models (dLLMs) are rapidly emerging as promising alternative to traditional autoregressive LLMs generalizing beyond the next token prediction (Nie et al., 2025b). Unlike autoregressive models, dLLMs generate text via an iterative mask-and-unmask process, al- lowing them to decode tokens in essentially arbitrary order (Kim et al., 2025). This fundamental change in the generation mechanism during training grants dLLMs remarkable flexibility at infer- ence time. In fact, recent dLLMss have already demonstrated competitive (and sometimes superior) performance compared to their autoregressive counterparts These early successes indicate that the masking strategy during inference plays a crucial role.
similar scale (Zhao et al., 2025).
Gaps in our understanding about dLLMs. The freedom choose the generation order, the mask- ing strategy, is the central advantage of dLLMs. Recent works (Kim et al., 2025; Nie et al., 2025b) have tried harness this flexibility by relying on prediction confidence, progressively unmasking high-confidence tokens (top-K margin in Figure 1). However, such an approach often leads to inher- ently biased solutions, overlooks the crucial sequential structure in the language training data, thereby inducing implicit biases in the learned masking strategies. As result, these methods might perform worse than random unmasking, as shown Figure 1. Why this happens and what we can do to avoid such a failure remains an open question, which we address in this work.
Our key finding: hidden semi-autoregressive experts. We uncover new dimension of test-time scaling for diffusion LLMs, centered on the masking strategy. We observe that the central degree
Joint supervision.
'The source code is available httth:t//gttteeee http:///////ggthh...o////ooesiir
Scal 1ing.

---

Published as a conference paper at ICLR 2026
prompt
masks
generations
80.00
71.97
Average Accuracy (%)
d1 (GRPO)
M o
66.56
Aggregation
60.00
Hid den Expert
45.91
Hidden Expert 2
40.00
31.77
29.00
30.00
itideen Expert B
output
final
20.00
Top-K
Top-K Random margin
Figure Overview of our proposed HEX framework. Left: HEX leverages multiple semi- autoregressive hidden experts, guided by different masking schedules, to produce concatenated out- puts and final answer. Right: НEX outperforms Top-K, Top-K margin (Kim et al., 2025) and Random expert selection strategies (Nie et al., 2025b) on reasoning tasks (GSM8K, MATH, ARC- C), surpassing the training-ban GRPO baseline (d1) (Zhao et al., 2025).
of freedom during inference lies in choosing the masking strategy, which is unique to dLLMs and critically shapes the generation distribution. The sequential nature language data causes dl dM. implicitly learn a mixture of comi-autoregressive experts during training. Each of these "experts" is naturally biased towards distinct masking distributions at test time, with a natural preference toward eenr-eetoreeeeesivve generation. For the first time, we demonstrate that this latent mixture can be deliberately accessed during inference. By varying the block size used in semi-autoregressive decoding, we can activate different experts, mirroring the conditions the model saw during training. This insight unlocks novel method for test-time scaling. By marginalizing across these block schedules, we can exploit the latent ensemble of experts, resulting significantly more robust and optimal inference.
Hence, we propose HEX (Hidden semi-autoregressive EXperts), a training-free inference method that uncovers new dimension of test-time scaling for dLLMs. HEX marginalizes across block schedules, treating block size and order as latent variables that define an additional scaling dimen- sion, and aggregates predictions via majority voting. In doing it robustly avoids the pitfalls of committing to any single decoding path, turing di dLLMs' hidden flexibility into a principled mech- anism for test-time scaling. We summarize contributions follows.
ii) New dimension of test-time scaling in dLLMs. We show that dLLMs implicitly learn a mix- ture of cemi-autoregressivv experts, and that block scheduling helps uncover this latent structure. (Section 3).
(ii) HEX: Hidden semi-autoreeressive EXperts for test-time scaling. We introduce HEX, training-fire free inference algorithm ensembling over semi-autoregressive schedules with majority-vote aggregation (Algorithm 2), turning ordering into reliable test-time scaling dimension. (Section 4).
(iii) Comprehensive experimental analysis: matching GRPO-llevl performance. HEX achieves GRPO-level results on GSM8K. MATH, ARC-C, and TruthfulQA, without retraining, establishing test-time scaling as powerful new paradigm for diffusion LLMs. HEX outperforms existing state- of-the-art inference methods (Kim al., 2025) on reasoning tasks, boosting accuracy by up to 3.56x (from 24.72% to 88. 10%.) HEX even produces massive gains more challenging tasks, including MATH (Lightman et al., 2023) (from 16.40% 40.00%) ) ARC-C (Clark al., 2018) (from 54. 18% to 87.80%), and TruthfulQA (Lin et al., 2021) (from 38336% 57.46%). (Section 5).
1.1 RELATED WORK
Diffusion Large Language Models. Diffusion models have achieved state-of-the-art performance in image generation (Ho et al., 2020: Song et al., 2020), and recent advances extend them to the discrete domain of language. The carly approaches applied continuous diffusion to latent text rep- resentations (Austin et al., 2021; Li et al., 2022: Dieleman et al., 2022), but faced challenges with tractable dis-
scalability and discretization. A masked diffusion paradigm soon emerged
2

---

Published as a conference paper at ICLR 2026
crete alternative (Nie et al., 2025a), with large-scale implementations such DiffuLLaMA (Gong et al., 2024) and LLaDA (Nie et al., 2025b) demonstrating that diffusion LLMs (dLLMs) can rival similarly sized autoregressive models, 2025). This potential extends even to multimodal understanding (You et al., 2025; Wen et al., 2025).
complex reasoning (Zhao et al., 2025; Tang et al.,
Inference-Tines Methods for dLLMs. In autoregressive models, inference-time scaling has been extensively studied, ranging from chain-of-thought prompting (Wei et al., 2022) and self- consistency (Wang al., 2022) scaling the allocation of test-time compute (Snell et al., 2024). In contrast, inference-time methods for dLLMs remain sparse. Most gains in dLLM performance so far have come from training-ting -tiee improvements, such as applying GRPO (Zhao et al.. 2025; Tang et al., 2025) or other post- wost-training methods (Wang et al., 2025; Asano et al., 2026; Zhai et al., 2026).
2 PROBLEM FORMULATION
Masked Diffusion Language Models (MDM). Let x (x1,,..,,,,, ..)) E V be length-n token sequence over vocabulary V. A masked diffusion large language model (dLLM) specifies condi- tional denoiser po [x[[]]]] x =) for any mask M S [n], where M [n] \ M. The notation x r[M] defined as the sequence of tokens from x on the indices of M, i.c. z|M] = (xz) In the forward (corruption) process, random subset of tokens M S [n] is masked (replaced by special symbol [MASK], and model pe is tasked with recovering the original tokens M given the unmasked tokens in the complement M =- [n]] \ M. Formally, for random mask pattern M, the model produces a conditional distribution po (z M] x x((M) on the masked tokens. Here, x[M] denotes the masked tokens in z. The training objective is to maximize the likelihood of ground-truth tokens in these masked positions. Hence, the training problem can be written as
0* E arg min Lmaak (f) == EN (in-D-Reeeun((i)))MMMMM
Σ REM
log (o xxMM))
= =1-
1
1 EEND log cc[M))
M
SEM
(M)
where is the data distribution, l' Unif((n]) uniformly sampled number of masked tokens É E n.....,n} and M C [n] is the randomly selected subset of length |M f. The summation in (1) runs over all masked token positions E .,, and the loss on each such position is the negative log- log-likelihood of the true token z; given the remaining context (unmasked) 2: x[M'). The objective in (1) trains the model to predict randomly masked-out tokens, and can be viewed as averag- ing next-token losses over all token permutations, ic., anyerrder objective (Nie et al., 2025b).
Algorithm 1 Vanilla MDM Inference
Input: prompt prompt output length L., steps T; mask schedule [M,) model pe(-I-).
Inference as the Core Challenge. After learn- ing 0* from (1), generation happens step by step. for t= 1, do
Initialize: (0) [MASK] *I;
For instance, for a given prompt xprompt it starts by selecting the number of tokens to be gener- ated (say L), and then requires choosing how to reveal tokens. Let decoding trajectory be quence of masks T = (M,...,,Ae My) that parti- tion [n] (such that _ Mf [n]).. with per-step
Predict all masked tokens simultaneously via ~ pe (- [Zprompt. ((t-1)))
x1(() Fill with predicted tokens
Fix tokens at location i E Mf
Mask tokens at location i E M, (U
Output: TT
sizes & = [Me]. At step t, the model predicts all
masked tokens conditioned on the currently revealed context 25 [ Mg]. For a fixed trajectory t
and prompt ppompe. functional of natural log-likelihood is
6-1 U s=1
I [T: xprompr) Σ
log Pa-
I; Eprompt
I
M.
3

---

Published as a conference paper at ICLR 2026
We summarize the vanilla inference procedure in Algorithm 1. The ideal (but empirically in- tractable") goal is to choose T such that we obtain a sample which maximizes [[; 0 Iprompp).
Because (I) trains all mask patterns, many training subproblems intrinsically ill ill-posed (c.g., extremely large masks with scant context) due to the implicit sequential bias in the language train- ing distribution. For example, some conditionals are rarely observed provide little meaningful context, making them effectively unsolvable. As a consequence, the model ends up learning only subset of subproblems or conditionals, while others remain poorly learned ignored. A uni- form masking objective forces the model to put equal weight on every subproblem, including those that confound the masking sequence, resulting in suboptimal masking strategy overall. This mis- match creates gap at inference time: the model's behavior becomes highly sensitive to the masking schedule, and strong performance depends on selecting strategies that align with the sequential bi- ases implicitly learned during training.
Key Open Question. Thus, the central question becomes: how can we design an inference strat- egy that faithfully reflects what the model has learned during training, given that dLLMs leave the process fundamentally under-specifeet and require us decide the optimal masking trajectory.
LIMITATIONS OF SOTA AND OUR KEY INSIGHT
Failure of existing inference methods for reasoning tasks. The existing dLLMs rely on heuristic inference-time strategies to choose which tokens unmask at each denoising step. The common ap- proaches include random sampling (randomly picking masked positions to predict) and confidence- based selection (choosing the token positions with the high model confidence or probability). For example, Kim et al. (2025) showed that for Sudoku puzzles simple confidence-basee top-K margin method can boost accuracy from 7% nearly 90%. Intuitively, one might expect similar benefit for reasoning tasks. Surprisingly. we find the opposite in reasoning benchmarks. In our experi- ments (see Figure on GSM8K math problems, random unmasking far outperforms top-K K margin (high-confidence) decoding. Instead of guiding generation, high confidence derails the unmasking trajectory into producing degenerate outputs. Figure 2, the top-K margin strategy consistently unmasked the [AfterEoT]] ((endoftext)) token at all positions, proceeding backward from the end to the front (See A.2 for detailed explanation).
This leads to degenerate outputs (shown in red text) in Figure 2. This surprising result challenges our intuition built through studying prior work.
GSMBK
Random (block 256)
Top-K margin (block
Random
50.87
Tep-K margin
Unioskiin
45
127
255
255
30
pred: 16.0 gt: 16.0
pred: None gt: 16,0
Let's aacume that Dorothy TE Fstabook frierdss 2. Charle
<(eot]d endoftext >-x p jendof beat ppnds/toodtp--]
25
24.72
Therefore, James Facebook friends, </masonings lboxedi 16)
-
> >=(endoffeett andoftiet|s> (eeddffeedd-- <jon doftext] lendattext] b-c lendohext) beeet//--eeeddeoeoe =]eedddortttt]-- (jendoftest endoffoxt|s>
20
est idje-d [djo-//endotooos
Jendoftext
blue: Mask
red: Aitertol
white: Text
green: Eoll
Figure 2: Random vs. Top-K margin inference on GSMSK. Left: Random decoding achieves 50.87% accuracy, while Right: Top-K margin only 24.72%. For each method, the text box shows the result at the last unmasking step. Top-K margin generates output tokens in reverse, from the end toward the beginning, and exhibits a catastrophic collapse in which all tokens are [AfterEoT] (shown in red). Over 55.5% of top-K margin runs suffered this collapse, yielding very low accuracy. These failures cast doubt on methods that rely solely on token confidence.
Because simply making consecutive greedy choices does not guarantee that the overall J is maximized, and performing global search would require evaluating all possible trajectories, which is practically infeasible.
4

---

Published as a conference paper at ICLR 2026
Unexpected reversal of intuition. Our findings highlight key limitation of relying common heuristics from autoregressive models and prior work (Kim et al., 2025). While one would expect that "follow the model''s 's own methods relying solely on token confidence are not sufficient for strong performance in complex reasoning tasks. Our results thus raise basic questions: Why does random sampling outperform top-K margin? Why does the model overconfidently pick the [AfterEoT] so early? To answer these questions, we propose a new perspective on the dLLM's internal structure. Our key insight that the dLLM can be viewed as an implicit mixture of experts, which allows us to mitigate the risk of overconfidently predicting [AfterEoT] tokens too early. By aggregating predictions from experts conditioned on different subsets of tokens, effectively marginalizing over contexts, our approach avoids prematurely committing to high-confidence tokens like the [AfterEoT]] token.
tokens" is reliable strategy, our results show that
From failure to mechanism. The surprising failure of confidence- based schedules suggests that local token probabilities are unreliable under many masked contexts induced inference. Because the dLLM objective (in (1)) averages over wide variety of maskings, including ill-posed ones, some conditionals are poorly learned and disproportionately favor special tokens such as [AfterEoT]. Our view is to treat each semi-AR block schedule querying different conditional expert, then marginalize across experts to recover robust predictions. This replaces brittle confidence fnnllwinc-followingg with consensus-seeking and is the rationale behind HEX.
Our Key Insight: dLLM is an implicit mixture of experts. From (2), it is evident that the diffusion LLM training leads model with family of masked token conditionals
{ (x; prrmpp x [[[]]
[n]] U S [m)] {
which can view as implicit "experts" indexed by the visible/unmasked set of tokens U. For a fixed target index i at test-time, the ideal goal of inference is to aggregate the relevant experts {po(2; (2y, [Zprempt, x (rc into a single prediction rule. A principled aggregation strategy is the mixture-of-experts predictor given by
Pmix (I: a (Pprospp)
Σ
TT (ppromps) pa(a; [Tprompe 2 (U]]),
where (U prompp) denotes the weight or trust placed on expert U for prompt prrompt-
A Toy Example. In our toy example (Figure 3), we examine the model's 's answer to the prompt "Who invented telephone?" (ground truth: "The in-
Token Confidence Distributions Over Different Mask Conditions
ventor was Bell."). We treat each latent condi- tional context (or expert)
0.6
Confudrce
as separate setting that produces a distribution to predict masked tokens. Figure plots these dis- tributions for particular position, = 4 (the token 'Bell'). As the plot makes clear, most experts concen- trate probability on "Bell" (the correct token), but a few contexts produce flat or incorrect distributions that never predict "Bell".
0.3
0.1
0.3
Edison
Resinder
Bel
Scoitlin
iveeted
teleshone
Graham
unkhown
{1, 3), The inventor was [M].
U (3), [M] [M] was [M]
Who invenced telephone?
<2),
[M]
{1. 3), The [M] wes [M]
U (1), The [M) [M] [M].
The inventor Bell.
(1. 2), The inventor [M] [M).
U (2)) [M] [M] [M] [M].
Figure 3:
The distribution of the 4th token Bell' in the output sequence changes significantly depending on the 2 masking condi- tions applied to the previous three tokens: The', 'inventor' was'. The star mark indicates the highest confidence for each distribution generated by U. Some masking conditions (violet and green) produce collapsed distribution: "Bell Bell was invented. (ungrammatical sentence), "The telephone was invented." (missing target information), respectively.
Correct-Answer Contexts (Experts): The majority of conditionals yield a distri- bution that peaked at the correct answer token Bell. These contexts effectively act "experts" on this question.
Non- Expert Contexts: Some conditionals do not produce a clear peak in 'Bell'. These contexts fail predict the correct answer.
5

---

Published as a conference paper at ICLR 2026
According to our insight (3), dLLMs learn unmask uniformly random masked patterns during training. As result of this randomized training, the accuracy varies across conditional contexts (or experts). Ideally, we would like to construct Pmix that maximizes accuracy by accounting for the importance of each expert. However, directly estimating Pmix is challenging. This is because doing would require likelihood estimation experts), and the hidden gating distribution (U)) that governs how likely each expert yields the correct answer is unknown and not directly observable. Meanwhile, the toy example (Figure 3) intuitively suggests that it may be possible to directly pursue the final objective of selecting the most accurate token without explicitly constructing Pmix- When each masking condition is regarded as an individual intrinsic expert, Figure 3 shows that mujority of experts "know" the answer.
all (2 for the N-th token) conditional contexts (or
4 HIDDEN SEMI-AUTOREGRESSIVE EXPERTS FOR TEST-TIME SCALING
In addition to the exponentially large number of experts required for estimation, prior work (Tang et al., 2025; Zhao et al., 2025; Nie et al., 2025b) has reported difficultics in likelihood estimation in dLLMs. Ideally, we would compute the Bayes-optimal conditional probability Pmix (2; = a (prrspp) by fully marginalizing over all possible sequences of tokens in U. To sidestep this, we approximate the ideal mixture by ensemble-averaging over small B of "semi-autoregressvve" each defined by a particular block-schedule b E B. Concretely, we sample several semi-AR decoding schedules each of which queries exactly one expert Us, and then averaging:
Pmix (x;;
pprompe)
Et [po(x; = a prompes x (U%])]
(4)
The final prediction can be made using the Bayes rule:
arg max Pmix (T; = a (pproppi).
(5)
This simple Monte Carlo approximation can be further simplified via majority voting: draw one sample âs from each queried expert Un. and return the mode of the sampled values. Thus, either by mixture averaging or majority vote, test-time aggregation amplifies the correct prediction by combining the specific conditionals (experts). The effectiveness of this approximation is intuitively illustrated in Figure 4. Based on this insight, we summarize our proposed approach Algorithm 2.
For each schedule, we first convert the final generated token sequence (T) into natural language string representation. We then apply numeric parsing to remove LaTeX formatting, whitespace, and commas. This yields a parsed output (x((T) for each schedule, which we store in a list. The final answer chosen as the value that occurs most frequently across schedules. If there is tie (i.e., two or more values appear with the same highest frequency), we choose the value generated from the schedule with the smallest block size.
Algorithm 2 Hidden semi autoregressive EXperts for test-time scaling
Input: prompt Pprompt, output length L, output parser f, steps I', experts B; semi-AR mask schedule ((Ur,)(-1) model po (-I-)-
Initialize outputs + for b = 1.2..... B do
Initialize: (0) + [MASK] for t 1,2 do
Predict all masked tokens simultaneously via (- [ppoomps (t-1
(1 <- Fill with predicted tokens
Fix tokens at location E USD,D
Mask again tokens location UL. (U
(T) - (T) as a natural language string Append parsed string output ((x((T) to outputs Output: max key lambda x: (outputs.coum) putputss.inexx(f)

---

Published as a conference paper at ICLR 2026
Token Confidence Distributions Over Different Block Sizes
0.8
Confidence
0.4
0.2
Andrea Timothy
John Alexander
David
Matthew
James
Anthony
Michael
Marc
Peter
Andrew
Robert
Cynthia
Anna
Simon
Aaron
Alan
Arthur
the
Alex
Jonathan
Lawrence
Eric
Adrian Adam Allen
prompt:
Please explain briefly who received the 2024 Turing Award.
output:
The 2024 Turing Award was awarded to Andrew G. Barto and Richard S. Sutton, and the content is their development of the conceptual and algorithmic foundations of reinfercement learning, through series of foundational papers since the 1988s that established key ideas, nath nathematical constructs, and algorithns pivotal to modern AI.
Block Size
Block Size 10
Block Size 18
Block Size
Block Size
Block Size Block Size 12
Block Size 20
Block Size 28
Block Size 64
Block Size Block Size 14
Block Size 22
Block Size
Block Size 128
Block Size Block Size 16 Block Size 24
Figure 4: When asked about the 2024 Turing Award winners, names other than the actual recipients (such as Michael David) might be generated due to different block sizes, which in turn risks producing incorrect information in the subsequent token sequence. However, if we generate outputs with various block sizes and then select the most frequently produced answer, that answer is more likely to be correct, since it was probably derived through a valid reasoning (Andrew) during the inference process.
Why semi-autoregressive? Diffusion LLMs allow all trajectories to reveal masked tokens, but uniformly random orders are suboptimal for language: they create unnatural partial contexts that the model was never intended to generate at test time. A practical restriction is semi- autoregressive left-to-right decoding (semi- AR): fix block size {1...., Buax) (where Buax n) and partition [n]] into consecutive blocks
Table 1: Semi-AR based decoding eliminates [Af-
terBoT] collapse and improves accuracy.
Dataset Collapsed (L. %)
Accuracy (f. %)
Baseline (non semi-AR)
GSM8K
55.80
22.52
MATH
29.80
16.60
Mr = (t-(((t))) (11--166-) I,... min n)},
Semi-AR
GSM8K
0.00
76.27
MATH
0.00
32.80
for -= 1, 1,..,, T(b) [n/bb], revealing blocks left-to-right while denoising within each block
via diffusion. This preserves a strong prefix structure (natural for language), yet allows parallel de- noising inside block. Empirically, we find (see Table 1) that semi-AR decoding avoids pathologies seen in fully parallel decoding. In particular, when using single large block (i.e., non-semi-AR par- allel decode), we often observe an [AfterEoT] collapse: the model erroneously floods the tail with [AfterEoT] tokens or repeats (Figure 8). By contrast, constraining to moderate block sizes (decoding left-to-right) eliminates this collapse and dramatically improves accuracy. (See Table 1: semi-AR has 0% collapse and much higher accuracy than non-semi-Ar decoding.) Intuitively, focusing first on the left part of the output prevents the model from prematurely committing to a length or drifting with high-confidence tail tokens.

---

Published as a conference paper at ICLR 2026
GSM8K
MATH500
ARC-C
TruthfulQA
100.0
87,80
90.0
88.10
40.0
40.00
62.60
60
57.46
79.8
(GRPO)
37.2
35.0
60.00
Accuracy (%)
50
70.0
70.05
60.0
70.00
42.40
25.0
40
50.0
40.0
54.18
30
29.82
28.36
30,0
22.52
24.72
15.0
50.00
20.0
10.0
20
10.0
5.0
Top-K
Top-K margin
Random
Ours
Figure 5; HEX improves reasoning accuracy. On LLaDA-8B-Insttt HEX outperforms training- free baselines (Random, Top-k, Top-k-margin) on GSM8K, MATH, ARC-C, and TruthfulQA. In GSM8K, MATH, ARC-C, it even outperforms the model trained with GRPO without any training.
EXPERIMENTS
In this section, we empirically validate our claims. (i) Effectiveness: We first demonstrate that HEX significantly outperforms existing training-free and fine-tuned methods on suite of reason- ing benchmarks. (ii) Scaling behavior: We then analyze the performance-computee
trade-off, showing how accuracy scales with more diverse generation paths. (iii) Working mechanism: Finally, through a series of ablations and qualitative examples, we explore the mechanisms behind HEX's success, confirming that its gain comes from ensembling a latent mixture semi-AR experts rather than relying on heuristics like model confidence.
5.1 SETUP
Datasets and Metrics. We follow standard reasoning benchmarks: GSM8K (Cobbe et al., 2021) consisting of high-quality problems with diverse linguistic expressions, MATH (Lightman et al., 2023) is more challenging math benchmark that includes competition-level math problems, ARC- C (Clark al., 2018) is the Challenge Set from A12's ARC dataset, consisting of science knowledge- based questions that are difficult to solve with simple keyword matching or retrieval, and Truth- fulQA (Lin al., 2021) which evaluates the tendency of language models to generate false infor- mation by following human misconceptions or false beliefs. Primary metric is task accuracy.
Models and Baselines. All experiments with inference methods were performed using the LLaDA- 8B- SB-Instruct model (Nie et al., 2025b), and the application of dl (GPRO) (Zhao et al., 2025) is subsequently based on this model. For all methods, when the output length 256 tokens, the number of unmasking steps is 128. At each step, two masked tokens are unmasked, and this process is repeated until all tokens are revealed. Random unmasks two randomly chosen masked tokens per step. Top-k margin unmasks, at each step, the two masked tokens with the highest margin defined as (top-1 confidence - top-2 confidence) at their positions. dI (GRPO) row uses the reported best value (Zhao al., 2025) for GSM8K and MATH, and for ARC-C we report value reproduced after epoch of training. TruthfulQA trained on di (GRPO) is excluded because there is no training data available, and neither were checkpoints released. HEX draws five samples temperature 0.9 for each of the block sizes [8, 16, 32, 64, 128), yielding 25 samples in total. If a tie occurs for the most frequent value, the value generated with the smallest block size selected (Algorithm 2).
5.2 MAIN RESULTS: HEX ESTABLISHES A NEW STATE-OF-THE-ART
Overall performance. Figure shows that HEX achieves the strongest results across all four rea- soning benchmarks, outperforming both training- -free and fine-tuned baselines. Compared to ex- isting decoding strategies (Nie et al., 2025b; Kim et al., 2025), HEX delivers large and consistent
We official evaluation scripts; numeric parsing strips LeTeX wwapers/whiicppuc//commss
8

---

Published as a conference paper at ICLR 2026
gains. In GSM8K, for example, HEX reaches 88.10% accuracy, far higher than Random decod- ing (50.87%) and Top-k margin (24.72%). These results show that confidence- based heuristics are unreliable in diffusion LLMs, whereas consensus-based voting in HEX is robust (Figure 7).
Comparison with GRPO fine-tuned models. Perhaps most strikingly, HEX also surpasses (GRPO), which requires costly reinforcement learning fine-tuning. On GSM8K (88.10% 79.80%), MATH (40.00% vs. 37.20%), and ARC-C (87.80% vs. 82.68%), HEX sets a new state of the without updating model parameters.
Intuitively, fixed inference scheduled in existing techniques sometimes asks the model to guess hard tokens early, which leads mistakes. In contrast, HEX tries several semi-autoregressive schedules and then picks the answer that many schedules agree on. In practice, answers that show up across schedules are more reliable than answers from any single schedule.
Takeaway. These results suggest that the reasoning ability of diffusion LLM remains latent and can be unlocked at inference time through block boock--aginnalzed ensembling, without any fine-tuning.
5.3 ANALYSIS OF SCALING AND COMPUTE TRADE-OFF
Figure 6 shows that HEX's accuracy improves monotonically as the number of voting samples in- creases, while the tie rate, an indicator of ambiguity, steadily declines. Intuitively, different semi-AR schedules make different mistakes but tend to agree on the correct answer; adding schedules can- cels schedule-specific errors and strengthens consensus, so ties resolve and accuracy improves. This trend holds consistently across all four benchmarks. Because sampling more trajectories linearly increases compute cost, HEX effectively exposes a tunable accuracy, compute knob: practitioners can trade inference cost for accuracy in predictable way, without retraining.
GSMBK
MATH
ARC-C
TruthfuIQA
20
88
The hain (8)
Accuracy 190
85
86
Accuracy
Rate
mh Pale
Rate
50
82
75
45
Cost
Accuracy
Figure 6: As the number of majority voting samples HEX increases, accuracy improves and the tie rate decreases. The block sizes used are [8, 16, 32, 64, 128], and sampling was performed while increasing the number of seeds (1-6).
Takeaway. HEX only establishes state-of-the-art performance but also provides principled mechanism for test-time scaling, ensuring accuracy improves with more inference budget.
5.4 ABLATION STUDIES
Next, we analyze the mechanisms behind the HEX improvements, focusing on two key factors: the role of block diversity and the role of likelihood versus frequency in candidate selection.
Effect of block diversity. Beyond using a fixed set of block sizes, test whether ensembling over more varied (and even randomly generated) block sched- ules further boosts performance. As shown in Table 2, creasing the number of dynamic trajectories from to 30 on GSM8K improves accuracy from 81.96 1.96% to 84.15% while re- ducing the tie rate to less than half. This reinforces hypoth- esis that performance gains come from aggregating diverse "semi-AR experts." We note that diversity matters, but struc- tured diversity (fixed block set with multiple seeds) is even stronger (as in Table 3), yielding the highest overall gains.
Table 2: Accuracy and tie rate (%) on GSM8K across dynamic block sizes. See Figure 12 for details on block sizing.
Size
Accuracy %)
Tie (1 %)
81.96
3.87
10
82.34
3.18
15
82.49
1.59
82,79
1.59
83.47
1.52
Frequency vs. likelihood. We then examine whether HEX's gains could simply come from likelihood-based re-ranking.
30
84.15
1.06

---

Published as a conference paper at ICLR 2026
Table shows that the selection of the lowest negative log-likelihood candidate (NLL) performs poorly, in some cases worse than Random decoding (c.g., ARC-C: 70.05% vs. 60.84%). In con- trast, HEX's 's frequency-basee majority vote achieves much higher accuracy (74.57%), confirmiig that consensus among diverse trajectories is more reliable than model confidence scores. This shows that the key driver of HEX's success is ensemble agreement.
Tie break and Latency. HEX defaults to the smallest block size in tie situations, as Table indicates that jointly considering frequency and log-likeling fog-likelihood does bring clear advantage. In addition, we present the wall-time latency of HEX and the baseline inference methods in Table 5.
Table Ablations across datasets. NLL selects the candidate with the lowest NLL. HEX's tie issue diminishes the number of samples increases. Block sizes: [8, 16, 32, 64, 128).
GSM8K
MATH
ARC-C
TruthfulQA
Method
Acc (1%) Tie (1%) Acc (1%) Tie (1%) Acc (1%) Tie (1%% Acc (1%) Tie (1%)
Baselines
Random
50.87
16.80
70.05
42.40
top-k
22.52
16.60
47.87
29.82
top-k margin
24.72
16,40
54.18
28.36
dl (GRPO)
79.80
37.20
82.68
likelihood-based
NLL
76.72
4.09
34.40
16.00
60.84
2.99
28.07
4.24
НEX
HEX
82.18
4.09
38.40
16.00
74.57
2.99
45.91
4.24
HEX x5 seeds
88.10
1.36
40.00
10.20
87.80
1.11
57.46
2.78
5.5 ADDITIONAL ABLATIONS
A detailed analysis HEX's tie-breaking rule is provided in Appendix B.1. comprehensive comparison between individual experts and HEX provided in Appendix B.3. A comparative analysis under extremely restricted block diversity is provided in Appendix B.4. Finally, an analysis of how HEX's performance varies with different output lengths is provided in Appendix B.5,
CONCLUSION AND LIMITATION
In this work, study how diffusion-based language models (dLLMs) generate text. We found that their performance is fundamentally tied to the decoding schedule, the order in which tokens are generated. This is because dLLMs implicitly Iearn "set" of semi-auttoregressive experts during training. Different schedules activate different experts, and choosing the right one is crucial for getting a high-quality answer. This single insight helps explain common dLLM issues, such as why they sometimes stop generating text too early or fail even when they seem confident.
Based on this insight, we introduced HEX (Hidden semi- autorerressive EXperts), powerful infer- ence method that requires no extra training. Instead of relying on a single schedule, HEX tries many different schedules at once and lets the experts "vote" on the best final answer. By combining the strengths of the entire hidden team, HEX turns the model's flexibility into reliable tool for boost- ing performance. On challenging reasoning benchmarks, HEX doesn't just beat standard methods; it even surpasses models fine-tuned with costly techniques like reinforcement learning (GRPO).
HEX has some limitations. It requires more computation at test time, and we have mainly evaluated it reasoning tasks. Applying this method to more creative areas like open-ended stories, image generation, or long conversations remains a promising area for future work. Further, we have not established any theoretical understanding of HEX, which is valid scope of future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partly supported by IITP grant funded by the Korean Government (MSIT) (No. 00020-I1201360 Artificial Intelligence Graduate School Program (Yonsei University)). Bedi is supported by Lockheed Martin and DARPA HR0011262E011
ETHICS STATEMENT
Our research aims to reveal the untapped potential of diffusion-based Large Language Model (dLLMs) and to enhance reasoning performance across comprehensive tasks without additional training. through test-time scaling. All datasets used the evaluation are public and widely known, and to the best of our knowledge, have thoroughly examined and cited research that is poten- tially or directly related to our work. We clarify that our use LLMs was strictly limited to polish writing, such grammatical correction and fluent expression, not for generating the main content of the research.
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697 2025.
Daman Arora and Andrea Zanette. Training language models to reason efficiently. 2025. URL https: //t::////xxvvvi arxiv. org/ ht//abb/225000 04463.
Hikaru Asano, Tadashi Kozuno, Kuniaki Saito, and Yukino Baba. Where-to-unmaskk Ground-truth- guided unmasking order learning for masked diffusion language models, 2026. URL https:
/ //arxiv. org/ abs 202..660220000 09501.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:1 711179911111 1-19999 2021.
Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Am- rit Singh Bedi, and Furong Huang. Transfer q star: Principled decoding for Ilm alignment. arXiv preprint arXiv:2405.20499 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try the ai2 reasoning challenge. arXiv preprint 1rXiv 1803.05457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data. arXiv preprint arXiv:2211. 2022. doi: 00-48550/rxxi:2211.5552
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. arXiv preprint arXiv:2505.13379, 2025.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint 1rXiv:2410.17891, 2024.
GSAI-ML.
https: /ww.gggggggggac.eaeoa/aa co/GSAI-MLI LLaDA- 1110A-8B--0 ruct. MIT License, accessed 2025-09-18.
-LaDA-8B-Instruce
11

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1166440685511 2020.
James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchhoff, and Dan Roth. Deal: Decoding-tion tendingg-ime time alignment for large lan- guage models. arXiv preprini urXiv:2402.06147, 2024.
Shijue Huang, Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R Fung. Adactrl: Towards adaptive and controllable reasoning via difficulty-awan budgeting. arXiv preprint arXiv:2505. 000225555.8822 18822, 2025.
Lingji Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hybrid-reasoning models. XrXiv preprint arXiv:2505. 14631, 2025.
Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. Args: Alignment as reward-guided search. arXiv preprint 2401:2240..01694, 2024.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025.
Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances neural information processing systems, 35: 2022.
Guosheng Liang, Longguang Zhong, Ziyi Yang, and Xiaojun Quan. Thinkswitcher: When think hard, when to think fast. arXiv preprint arXiv: 14/833, 2025.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2 2021.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng- Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310. 17022, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.1933) 2025.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv: arXiv:21/2.09333 /2.09322 2021.
Shen Nie, Fenggi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongx- uan Li. Scaling up masked diffusion models on text. The Thirteenth International Confer- ence on Learning Representations, 2025a. URL https:/ /ttp//ppeeevvveeeeeeeeeeeeeeeeee net forum? id WNvvwKOtut.
https:://ppenreiii
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint NwXiv:2502.09992 2025b.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, and Simon S. Du. Decoding time language model alignment with multiple objectives. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024.
12

---

Published as a conference paper at ICLR 2026
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. Weighted policy optimization for reasoning in diffusion language models. arXiv preprint arXiv:2507.088.2 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, and Chumhua Shen. Time is a feature: Exploiting temporal dynamics in diffusion language models. arXiv preprint 250i::2500.09133333 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Selfscoons improves chain of thought reasoning in language models. arXiv preprint wwXh:2203.11177: 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2458 24837, 2022.
Yuqing Wen, Hebei Li, Kefan Gu, Yucheng Zhao, Tiancai Wang. and Xiaoyan Sun. Llada-vla:
Vision language diffusion action models. arXiv preprint arXiv:2509.069:: 069320900666,, 2025.
Wenkai Yang, Shuming Ma. Yankai Lin, and Furu Wei. Towards thinking-optimal scaling of test- time compute for I1m reasoning. 2025. URL https: //tp///aaii// org/ abs/ /t/.//5222..........
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025.
diffusion language models, 2026. URL https: //te///aareeee /wwxivvvvvv///////////aaa//aaavv org/ abs 202. 04096. Kevin Zhai, Sabbir Mollah, Zhenyi Wang, and Mubarak Shah. Core: Context-robust remasking for
Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417. 2025a.
Xiaoyun Zhang, Jingqing Ruan, Xing Ma, Yawen Zhu, Haodong Zhao, Hao Li, Jiansong Chen, Ke Zeng, and Xunliang Cai. When to continue thinking: Adaptive thinking mode switching for efficient reasoning. arXiv preprint arXiv:2505. / 15400, 2025b.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. Scaling reasoning diffusion large language models via reinforcement learning. arXiv preprint 12X1: 2025.
13

---

Published as a conference paper at ICLR 2026
A
QUALITATIVE RESULTS
A.1 QUALITATIVE ANALYSIS OF BASELINES VS. HEX
Manolo bought five lollipops and four candies that cost $3.20.
If each lollipop costs $0.40, how much will 10 lollipops and 10 candies cost him?
Ground Truth:
Top-K margin:
Random:
HEX (Ours):
First, let's determine the cost of candy.
First, we need determine of candy. Manolo bought five lollipops and four candies that cost $3 23,20, and each lollippp costs $0.40. This can expressed as:
First, let's determine the cost one candy,
Manolo bought five lollipops and four candles for total of $3. 23,20, and each lollipop costs $0.40, so the total cost of five tollipops is M5 itimes 0.40 2.00V).
lollipops and four candles cost $3,20, and each lollipop costs $0.40.
The total cost five tollipops ls: | 5
\]
5 wimes 0.400 4C 3.20 V {4C = 2.00
Therefore, the cost of four candies V/.20 2.00 = 1.20%).
This means that the total cost of
the four candies is $1.20.
[ [ \frac(2.0004]] 0.5000
Therefore, the cost one candy is: M[ Mrrc(11 20/44) 0.00 \]
Since four candles cost $1.20, the cost one candy V/1 Vdiv 4- 0.30V.
tollipop
$0.40 / 1000 $0.40. Each candy
Now, need to find the cost 10 lollipops and 10 candies. The cost of 10 candies is: M[
costs $0.4000.
Now, we need to find the cost of 10 tollipops and ten candies. The cost lollipops \/10 ttimes 0.40 4.00%)
To find the total cost of 10 lollipops candies:
0.00
The cost of 10 tollipops is: M 10 \times 0.40 4,00 \]
V[ 10 \times 0,4000 + 10 Itimes
0,5000 }]
and the cost of 10 candies is 1/10 itimes 0.30 3.00V).
|[ 10 itimes 0,400 50000 itimes
Therefore, total lollipops 10 candies $4.00. </ressoning>
100 5
</reasoning>
the is 14.00 3.00 7.00%. </reasoning
<answer>
\boxed[4.00) JJanwee><<eot_inne /oodoooodoopd///dfxxx
\boxed(5.00)
\boxed(7.00;
</answer><coot_idd iddo
Figure 7: An instance of generated text responses of different decoding strategies.
A.2 STRUCTURED PATTERNS INHERENT IN AFTEREOT COLLAPSE
Despite being highly unintuitive and unpredictable, the ordering pattern observed in AfterEoT Col- lapse exhibits a clear structure: although there is no explicit incentive for the model to unmask padding tokens in right-to-left manner during either training or inference, this regularity consis- tently emerges throughout the entire unmasking process (see Figure 2). This systematic behavior under collapse suggests that the underlying ordering mechanisms in diffusion LLMs embedded within both the training objective and the inference procedure may play more active and influ- ential role than previously recognized.
A.3 QUALITATIVE ANALYSIS OF SEMI-AR VS. NON-SEMI-AR
As shown in the right side of Figure 8 confidence based non-semi-AR decoding, the phenomenon where [AfterEoT] tokens accumulate from the end of the output towards the front indicates that the model assigning high confidence to [AfterEoT] token throughout the unmasking steps.
The input to the dLLM consists of the number of tokens that make up the prompt and the number of tokens in the desired output sequence, and during training it is subject to limits on the input sequence length for parallel computation. For LLaDA-8B-Ine Instructt (GSAI-ML), this limit is 4,096 tokens. However, in the training of reasoning tasks, most of the output finishes within 256 tokens. In other words, the majority of ground truth tokens in the output sequence (more than 93.75%
14

---

Published as a conference paper at ICLR 2026
X
(block
semi-AR (block size 256)
Unmasking step
Unmasking step
pred: gt 7.0
pred: None gt 7.0
127
127 0
Taken position
255
Token position
255
To determine how many dorens of eggs Claire will eat in weeks, we need to follow these steps: 1. Calculate number eggs
=]Jendofaest] p-d jendoff exc p-st]---jjpe p-dtenn en do ao <jendorte xtt =led-=iooextt tie lendoftext p-e extoffextt leedof efeendaf
day. 2.
-jendolftee eoffeeeepee othert bedlendo foott oendooeeett Jen ndoft ext] p-o (ende freat [-- ndorteed p-d cendoft exel p-c jendoftaxtt-- </endofuuct]- < jeeddfhecct au deendofaxt >x jan jandaftext] -
sjjnndo
total number of aggs into dozens: 'I itestt eggs) eggs/dozen) treat dozens)
boxt x]jendofooxtt] pi Jandoofte
blue: Mask
red: AfterEoT = white: Text
green: EoT
Figure 8: Blue denotes mask tokens, red denotes [AfterEoT] tokens, white denotes text tokens, and green denotes [EoT] tokens (note that in the LLaDA-8- Insttruutt model, [EoT] and [AfterEoT] are represented as < leot id] >> and < |endo proceeds, two mask tokens are unmasked at each step (output length = 256, unmasking steps = 128). Under a semi-AR regime with block size = 32, positional constraints force reasoning to progress left-to-right while still allowing diffusion-style generation within each block. By contrast, when the positional constraint is removed with block size = 256 (non-semi-AR), the model starts from the last token with the highest confidence-[Afiieei same token backward, ultimately collapses into catastrophic output in which all tokens become [AfterEoT].
>, respectively (GSAI-ML)).
unmasking
and, due to the inertia of repeatedly generating the
seml-AR (block size 32)
not semi-AR (block size 256)
Demasking step
Demasking stop
pred: 3.0 gt
pred: 6.0 Token position
127
127
Token position
determine speed remaiing
achieve needs walk
needs to fike this, should 74 miles per hour i hours to complete the like. first, lets calc
miles hour, first need calculate total
miles
distance she has already wa
speed miles 2 hours remaining distance
distance Remaining time miles hour
AfterEoT _
white: Text
green: EoT
Figure 9; Few cases (2.96%) where the block size of output length hits and semi-AR fails. However, in those cases, the unmasking order converges from both ends toward the center. It means the model commits to answer before engaging in full reasoning process -ee. it states the answer first and only then provides a derivation. This is not only unintuitive; Wang et al. (2025) shows that the answer the dLLM settles on can flip repeatedly during the reasoning process. Consequently, when the model locks in an answer first, it is hard to claim that it truly "knows" the answer with certainty.
[AfterEoT]: Given that the training objective is to maximize the average likelihood, can infer that the dLLM is most strongly taught to generate the [AfterEoT] token.
This suggests that confidence-only decoding is fundamentally limited in its ability to prevent such phenomena during inference, and highlights why the positions of tokens to be unmasked should not be selected based solely confidence.
15

---

Published as a conference paper at ICLR 2026
Furthermore, confidence-onie decoding lead issues such as those shown in Figure 9, even when the answer is correct. This arises in reasoning tasks where the correct answer should be de- rived through step-by-step justification. Instead, the model often generates the answer first without providing supporting reasoning (with the answer tokens generated on the right first), and only after- ward begins to generate the reasoning (by unmasking tokens in the middle). Fundamentally, unless the prraady-genrrted answer tokens are revised, this generation pattern means that the reasoning cannot actually influence the answer.
Besides, as observed by Wang et al. (2025), the probability distribution corresponding to the answer tokens, whether they are correct or incorrect, continues to undergo substantial changes during the reasoning process. This indicates that the model's confidence in the produced early answer cannot be considered reliable.
B ADDITIONAL EXAMPLES AND RESULTS WHICH CAN BE USEFUL
B. EXPERIMENTAL RESULTS OF HEX'S TIE-BREAKING METHODS
Frequency-based majority voting has the characteristic that ties can occur within a finite set of can- didates. To determine a single final output, we experimented with various tie-breaking rules as shown in Table 4. We found that likelihood--asee selection did not provide any significant benefit. As shown in Figure 10, since we observed that performance tends to degrade as the block size ap- proaches the output length, adopt selecting the output with the smallest block size in tie situations as our default setting.
Gsm8k
MATH
ARC-C
TruthfuIQA
Accurary 196)
Accura cy 1%6)
70
Accuracy (96)
60
Arcurapy
40
25-
60
40
20
50
30
20
100
200
100
200
100
200
100
200
(Block Length)
(Block Length)
oBlock Length)
(Block Length)
Figure 10: Accuracy across tasks when the block size (B) is increased in fixed increments of 16. Here, block count (256 // B) when 256 % B is O otherwise (256 // B) 1. Despite the uniform increase in block size, the performance drops drastically in the 128 256 range.
Table 4: Evaluation on tie breaking methods. If the most frequent output in tie situation, TIED: NLL selects the result with the lowest negative log-lik ligehiod in tie situations, TIED: first selects the result generated from the smallest block size when tied, and TIED: any treats the case as correct if a correct option exists among the tied candidates. The results of TIED: any clearly highlight that majority voting HEX works well across datasets.
GSMSK
MATH
ARC-C
TruthfulQA
Method
Acc (1%)
Tie (1%)
Acc (1%)
Tie (1%)
Acc (1%)
Tie (1%)
Ace (+%)
Tie (1%)
HEX (sie-breaking rales)
HEX, TIED: NL.L
82.18
4.09
38.00
16.00
74.49
2.99
46.20
4.24
HEX, TIED: first
82.18
4.09
38.40
16.00
74.57
2.99
45.91
4.24
HEX, TIED: any
83.09
4.09
41.00
16.00
76.11
2.99
47.66
4.24
B.2 ANALYSIS OF DECODING LATENCY ACROSS INFERENCE METHODS
For the HEX x5 seeds setting in Table 3, we further sample 5 independent seeds per expert (5 block sizes 5 seeds 25 semi-AR samples). Since each sample has roughly the same cost as a single top-k method, the total token-level inference cost is approximately 25x that of a single top-k run. The additional cost of majority voting over these samples is negligible.
16

---

Published as a conference paper at ICLR 2026
Table 5: Inference efficiency (in seconds) of HEX (xl seed) GSM8K, MATH, ARC-C, Truth- fulQA. The numbers in parentheses indicate the number of data points. Random, top-k, and top-k
margin use single sample with a block size of 32. HEX (xI seed) uses five samples, where each
sample is generated with block sizes of 8, 16, 32, 64, and 128. Across all samples, the output length set to 256, with 2 tokens being unmasked each step.
Dataset
Method
Total test set
per batch (8)
per datapoint (I)
ratio
random
2775.73 2921.70
16.76
2.09
X x1.0000
top-k
17.64
2.20
x1.0526
GSM8K (1319)
top-k margin
3187.56
19.25
2.41
x1.1484
HEX
14613.72
88.23
11.03
x5.2648
random
1300)62
20.46
2.56
x x1.0000
top-k
1365.98
21.51
2.69
x1.0503
MATH (500)
top-k margin
1477.33
23.28
2.91
x x1.1359
HEX
6823.17
107.50
13.43
x5.2461
random
2679.99
18.15
2.27
x1.0000
top-k
2813.69
19.05
2.38
x x1.0499
ARC-C (1172)
top-k margin
3048.49
20.64
2.58
x x1.1375
HEX
14062.59
95.22
11.90
x5.2473
random
1532.40
17.71
2.21
X x1,0000
top-k
1608.77
18.58
2.32
x1.0498
TruthfulQA (684)
top-k margin
1739.78
20.12
2.52
x1.1353
HEX
8038.26
92.90
11.61
x5.2455
In practice, these 25 samples can be generated in parallel on modern accelerators, so wall-clock la- tency can be significantly lower than strict 25x factor, but the total inference FLOPs scale linearly with the number of samples. We therefore view the number experts x seeds as an explicit accu- racy-compute knob: practitioners can trade inference cost for accuracy in predictable way (Figure 6), without retraining.
B.3 COMPREHENSIVE COMPARISON WITH THE SEMI-AUTOOFEGRESSSIV BASELINE
Table 6 illustrates that the average performance of the samples used in HEX's majority voting that is, the individual semi-AR experts consistenny falls short of HEX across all output lengths (128, 256, and 512). This indicates that dLLMs have implicitly learned diverse of inference behaviors from multiple semi-AR experts. Therefore, uncovering and leveraging the correct judgment among these hidden experts essential ability that HEX demonstrates with remarkable effectiveness..
Method
GSM8K
MATH
ARC-C
TruthfuiQA
128 [4, 8. 16. 64] seed [8, 16, 64, 128] seed
68.57
28.00
79.44
48.74
75.39
33.16
67.27
39.59
[16, 32, 64, 128, 256] seed
66.52
32.88
66.35
44.82
[4, 8, 16, 64). HEX X seed
74.22
30.60
85.41
53.36
[8, 16, 32, 64, 128), HEX seed
82.18
38.40
74.57
45.91
512 116, 32, 64, 128, 256J, HEX seed
83.24
39.60
84.81
57.02
Table HEX (the bottom three rows) consistently surpasses the mean accuracy (the top three rows) of samples used in majority voting across various output length settings. The number preceding [ represents the output length, and the numbers inside correspond the block sizes used.
B.4 COMPARISON BETWEEN HEX AND SAMPLING-BACE VOTING
We compared a single block schedule with 25 different random seeds to HEX (structural diversity: five block schedules five seeds). Following common practice, used block size of 32 for the single block schedule. HEX with structural diversity outperforms the plain method (with only one block schedule and 25 seeds) by 4.2% in MATH, and 3.57% in GSM8K.
17

---

Published as a conference paper at ICLR 2026
Table 7: HEX outperforms the baseline with single block schedule and 25 samples.
Method
MATH (accuracy, tie rate)
GSM8K (accuracy, tie rate)
semi-AR (single block schedule)
32. 80%,
76.27%. 79.53%, 0.68%
single block schedule (25 socds)
35.80 35.80% 4.20%
HEX (structural diversity: 5 block schedules seeds)
40.00% 10.20%
88.10% 1.36%
Interpretation. As shown in Table 7, simply increasing the number of samples for a single block schedule by different seeds does not create sufficient diversity for majority voting; it only perturbs noise, not structure. Structural diversity, using multiple block schedules, changes the underlying context for each semi-AR expert and therefore produces different reasoning trajectories. This is why the plain 25-seeds baseline provides only limited gain, while HEX shows significantly stronger improvements.
B.5 ABLATION STUDY OF HEX ACROSS DIFFERENT OUTPUT LENGTHS
Across various output lengths, HEX consistently proves to be an effective method. As shown in Table 8, for output lengths 128, 256, and 512, HEX consistently surpasses both other inference methods and GRPO trained models.
Table 8: Ablation study of HEX across output lengths of 128, 256. The results demonstrate that
HEX consistently exhibits robust performance irrespective of output length. Block sizes used are the same as Table 6. Refer Table 4 for detailed explanation about (b). (c), (d).
Output Length 128
Method
GSM8K (acc, tied)
MATH (ace, tied)
ARC-C (acc, tied)
TruthfulQA tied)
Random
48.82%
20.60%
66.21%
41.96%.
top-k
53.30%,
20.60%,
56.74%
36 40.%.
top-k margin
55.57%.
22.60%.
61.35%,
38.30%.
dl (GRPO)
72.60%.
33.20%
82.68%.
NLL
70.81% 7.28%
29.80% 20.20%
78.41%; 1.54%
47.95%. 4.24%
HEX, TIED: NLL (b)
73.69%, 7.28%
32.00%, 20.20%
85.24% 1.54%
52.63%, 4.24%
HEX, TIED: first (c)
74.22%, 7.28%
30.60%, 20.20%
85.41% 1.54%
53.36%, 4.24%
HEX, TIED: any (d)
75.82%, 7.28%
35.60% 20.20%
85. 35.92%, 1.54%
$4.39%, 4.24%
HEX seeds, TIED: first (e)
81.05 %,, 3.64%
33.40 %, 15.60%
88,40% 0.60%
53.80% 2.78%
Oulput Length 256
Method
GSMKK (ace, tied)
MATH (ace, Tied)
ARC-C (acc, lied)
TruthfulQA (acc, fied)
Random
50.87%..
16.80%
70.05%.
42.40%
top-k
22.52%.
16.60%.
47.87%.
29.82%
top-k margin
24.72%,
16.40%
14.18%
28.36%
(GRPO)
79.80%
37.20%,
82.68%
NLL (a)
76.72%, 4.09%
34.40% 16.00%
60.84%. 2.99%
28.07%. 4.24%
HEX, TIED: NLL
82.18%. 4.09%
38.00% 16.00%
74.49%. 2.99%
46.20%. 4.24%
HEX, TTED: first (c)
82,18%, 4.09%
38.40%, 16.00%
74.57%, 2.99%
45.91% 4.24%
HEX, TIED: any (d)
83.09%, 4.09%
41.00%
66.00%
76.11%. 2.99%
47.66%. 4.24%
HEX x 5 seeds, TIED: first (e)
88.10%, 1e, 1.36%
40.00 % 10.20%
87.80% 1.11%
57.46%, 2.78%
Output Length 512
Method
GSM8K (ace, tied)
MATH (ace, tied)
ARC-C (ace, tied)
TruthfulQA (acc, tied)
Random
31.71%,
16.60%
73.29%
43.80%
top-k:
9.10%
10.00%,
50.17%,
29.68%,
top-k margin
10.08%.
9.40%.
54.35%
31.14%,
dl (GRPO)
81.9%%
39.20%
83,28%
NLL (a)
78.85% 4.25%
25.00% 20.80%
63.82%, 5.03%
34,50% 7.75%
HEX, TIED:
(b)
82.11%, 4.25%
40.20% 20 80%
83.11%, 5.03%
$4.97%, 7.75%
HEX, TIED: first
82.03%, 4.25%
40,20% 20.80%
84.13%. 5.03%
55.85%. 7.75%
HEX. TIED: any (d)
83.47%, 4.25%
44.80% 20.80%
84.815 15. 5.03%
57.75%. 7.75%
HEX x 5 seeds, TIED: first (e)
88,40% 1.14%
47.60%, 10.40%
87.12% 5c 0.85%
59.80 %, 2.63%
B.6 HOw THE SEMI-AR SCHEDULE LEVERAGES LEARNED PREFIX-LIKE CONTEXTS
Let (z1, L2, Z3. 24). To predict I4, the visible set is subset of (1, 2,3), i.e.
22,
Ue { 20 (1), (2), (3), 2), (1,3), (2, (1,2,3)
18

---

Published as a conference paper at ICLR 2026
Suppose (due to sequential bias in the data) the model learns well only the prefix-like contexts
$ {{1). [1.2), (1.2.3)}
(6)
Then left-to-right semi-autoregressive (semi-AR) schedules realize exactly these conditionals by changing the block size B:
= 1,1,1, (blocks (1), (2), (3), (4)): when 24 is predicted, the visible set is U = [1,2,3} so the model uses po (T (2,
- 2,2 (blocks {1,2), 43)4)): when Iı is predicted (with 23 in the same step parellelly), the visible context the completed first block, U (1,2). hence po 22).2).
B = 3, (blocks (1,2,3), (4)): when 24 is predicted, U 2,, again, hence (a(4 Tz))
One can also realize U = (1), (1), by B 1, (blocks {1), (2, 32,4)4)) T2. 23 the same step in parallel), the visible context the completed first block. {1}, hence (x4 21). Furthermore. if we set additional within-bloo order constraints Conter (c.g.
4)): when is predicted (with
descending order of confidence), another possible condition of U {1} can oceur by
B 4 (blocks {1,2,3,4) 43) A Conder K 24 D2 < I3 within B),
hence the model uses p0(x4 .1))
Key point: By varying the semi-AR block size and within- order, decoding selects among the learned conditionals in S.
B.7 ADDITIONAL CONTEXT OF RELATED WORK
Test-Time Scaling in Autoregressive LLMs. A growing body of work in autoregressive LLMs has begun to formalize the phenomenon of test-time scaling, the empirical observation that allocating more inference-time compute (e.g., longer reasoning traces cantly improve model performance without retraining. Mucnnighoff et al. (2025) introduced budget forcing to emulate the compute-des dependent reasoning behaviors, while Aggarwal & Welleck (2025) proposed length-controlled policy optimization to explicitly regulate the reasoning horizon. There also complementary efforts, such as thinking-optimal scaling (Yang et al., 2025) and adaptive reasoning policies (Arora & Zanette, 2025; Fang et al., 2025; Zhang et al., 2025b; Jiang et al., 2025; Liang et al., 2025; Zhang al., 2025a; Huang et al., 2025). that highlight the goal of dynamically adjusting cognitive effort based on task complexity and available compute budgets.
sampled candidates) can signifi-
In parallel, researchers have explored how controlled sampling and decoding strategies can steer model behavior toward desired objectives without fine-tuning. Best-of-N selection (Stiennon et al., 2020: Nakano al., 2021; Touvron al., 2023) provides simple form of test-time search (which requires a reward signal access) over diverse outputs. We also include an ablation where we compare an instance Best-of-N with majority vote as the reward signal in Table 9, We note that this lower than random sampling (cf. Figure 5(a)). On the other hand, more structured methods recast decoding as reward- guided search (Khanov et al., 2024; Huang et al., 2024; Mudgal et al., 2023; Chakraborty et al., 2024). Recent multi-objective decoding frameworks (Shi et al., 2024) further generalize this idea, balancing several alignment criteria simultaneously.
Table Comparison majority voting between another non-semi-AR (BoN) and semi-AR experts on GSM8K. The number preceding represents the output length, and the numbers inside [] correspond to the block sizes used.
Method
Accuracy (1 %)
Tie (1 %)
256 [256, 256, 256, 256, 256], BoN with majority voting
44.35
44.58
256 [8. 16, 32, 64, 128], HEX
82.18
4.09
'Adding this condition changes the number of unmasking steps within the block size.
19

---

Published as a conference paper at ICLR 2026
B.8 FURTHER SEMI-AR ANALYSIS
Figure 11 shows that is rare for outputs generated with different block sizes under the same prompt all fail simultaneously. also reveals that no particular block size consistently dominates in performance. This indicates that heuristically determining the block size for given prompt can be suboptimal.
Answer correctness per datapoint
100.00%
Block Size
256 128
78 47%
32
76. 27%
75.82% %
75. 28%
Match
Mimmathh
ia None
GSMBK datapolnts (1319)
Figure 1I: Visualization of the correct (blue) and incorrect (white) answers for each data point in GSM8K when reasoning with various block sizes. Apart from the case where the block size equal to the output size, i.c., non-semi-Ar generation (256), there is no consistent pattern across block sizes in this figure that would allow to conclude which block size is most suitable for semi-AR reasoning for given tasks. However, the observation that it is rare for all semi-AR outputs to fail on given data point suggests that the model has the potential to arrive the correct answer even without additional training, simply by selecting the appropriate decoding policy for the question.
B.9 DETAILS OF DYNAMIC HEX BLOCK SETTINGS
214
13
T8
32
26
1
31
28
26
36
1.16
16
2-
53
90
50
1.
14
56
30
1
46
1.
22
34
40.
24
26
40
1:0
48
20
40
12
20
24
25
Figure 12: Examples of the block sizes and counts used in the dynamic HEX block settings. Block sizes and counts were randomly chosen and adjusted to match the total output length. The output length is 256 and the number of unmasking steps is 128, meaning that each step unmasks tokens. Accordingly, all block sizes are multiples of 2, and decoding was performed in attrgrrvrree manner.
20


Reviews:
[{"review_id": "Qyuuw4Pbdj", "reviewer": "Reviewer_bU9H", "summary": "This paper shows that by ensembling over multiple semi-autoregressive decoding orders, the current dLLMs can get a significant improvement in capability. They empirically show that confidence-based decoding fails to beat the random order baseline. With autoregressive sampling, the performance jumps more than 3x on benchmarks including GSM8k. Ensembling over block-size and randomness further improve the performance.", "questions": "1. Please refer to weakness 2. \n\n2. Have authors think about how to incorporate HEX into the post-training of dllm, in order to get either faster decoding or better performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly illustrates that previous methods such as confidence-based sampling fail in the context of long reasoning. This provides a clear motivation for designing new algorithms.\n\n2. The method proposed is very intuitive and easy to apply.\n\n3. The potential mechanism behind the empirical improvement is clearly outlined in Sections 3 and 4.", "weaknesses": "1. It should be noted that in the abstract, the 24.72% performance is not using semi-autoregressive baseline, which is widely adopted.\n\n2. In section 5.4, regarding the importance of block-diversity, the author shows that ensembling 30 different block schedules can improve performance but is worse than 'structural diversity' with a fixed set of 5 block schedules and 6 random seeds. However, it remains unclear whether the proposed algorithm can beat the plain method with only 1 block schedule and 30 seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jYG4OWGuqY", "reviewer": "Reviewer_BX1J", "summary": "The authors propose a specific choice of block schedule for sampling from diffusion language models which is “semi-autoregressive”, meaning that parallel sampling is carried out in blocks that are autoregressively revealed left to right. The authors associate different choices of block schedule with different “expert” components of the dLLM and reveal large gains in accuracy by taking the majority vote over predictions by such experts (in effect marginalizing over different choices of block schedules).", "questions": "- Can you clarify the impact of your proposed methodology on computational efficiency? Please compare both to regular sampling from diffusion LLMs and to autoregressive sampling from transformer-based LLMs.\n\nSee the section on weaknesses for additional questions.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- Massive performance gains\n- Interesting interpretation of block schedules as experts", "weaknesses": "- Identification of block schedules as experts is tenuous - do these block schedules really encode specific knowledge/expertise in the way that, say, the experts in an MoE model do? Can you provide more evidence of this?\n- If semi-autoregressive decoding uses all block schedules down to very small granularity (b=1), can performance gains be explained in terms of increased granularity (and higher computational cost)? The b=1 case in particular seems to perfectly mimic standard auto-regressive sampling in transformer-based LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ez9pIYjkKz", "reviewer": "Reviewer_27TW", "summary": "The paper argues that dLLMs trained with masked-token objectives implicitly encode a mixture of semi-autoregressive experts: different left-to-right block sizes activate different conditional distributions learned during training. Committing to a single decoding schedule can collapse performance. The proposed training-free method HEX ensembles across heterogeneous semi-AR block schedules and aggregates final answers via majority vote. On LLaDA-8B-Instruct, HEX reports strong gains while avoiding collapse modes.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper shows dLLMs are sensitive to semi-autoregressive schedules, which provides confirmed motivation for HEX to ensemble diverse schedules for robust inference.\n\n2. HEX is a training-free plug-and-play method, enabling a trade-off between accuracy and computation via the number of voting trajectories, and is competitive with (or exceeds) RL-fine-tuned variants under the reported settings.", "weaknesses": "1. Although the paper treats the number of voting trajectories as a tunable inference time knob, the marginal gains from adding schedules are modest relative to the near linear growth in compute (for example, GSM8K improves only from 81.96% to 84.15%). Moreover, while HEX can surpass RL fine tuned baselines, the cost profiles differ: RL incurs a one time training expense and then serves at single pass inference cost, whereas HEX must sustain a $k$ times inference overhead on every request to remain competitive, which is an unfavorable trade at deployment scale.\n\n2. While HEX’s wall-clock latency is reported for GSM8K, the paper does not provide budget-matched comparisons across all datasets, which limits a fair cost-effectiveness assessment.\n\n3. Despite a persuasive mixture-of-experts interpretation, the method remains theoretically under-specified: there are no conditions or guarantees for the consistency/convergence of majority voting, sample-complexity requirements, or optimal schedule priors, weakening the theoretical closure of the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cecm6B5NfH", "reviewer": "Reviewer_6HRH", "summary": "The paper proposes HEX (Hidden semi-autoregressive EXperts), a training-free inference method for diffusion-based large language models (dLLMs). Through a series of intuitive experimental observations, it reveals that dLLMs implicitly learn a mixture of semi-autoregressive experts, where different masking schedules activate distinct expert behaviors. By ensembling multiple decoding paths with majority voting, HEX achieves remarkable performance on reasoning benchmarks such as GSM8K and MATH.", "questions": "What are the differences and connections between the proposed method and beam search?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Introducing the proposed method through a simple toy example makes the paper intuitive and easy to follow.\n\n2. The experimental results are impressive, showing substantial improvements across multiple reasoning benchmarks.", "weaknesses": "1. The authors repeatedly emphasize (for example, in Sec. 1 and Sec. 3) that using sampling strategies such as top-K margin can harm performance and that this issue has not been sufficiently discussed. However, previous literature has already provided explanations for this phenomenon, such as Appendix B.4 of [1].\n\n2. The presentation is poor. For example, Eq. (1) is incorrect because it lacks the necessary reweighted coefficient. The description of the main method—such as how the majority vote is performed—is neither clearly introduced in the text nor reflected in the algorithmic formulas.\n\n3. The evaluation is neither fair nor comprehensive. For instance, in the abstract the authors state that LLaDA-8B-Instruct achieves 16.4 % and 54.2 % accuracy on MATH and ARC-C, which HEX improves to 40.0 % and 87.8 %. However, the original LLaDA paper reports 42.2 % and 88.5 % for these datasets. The authors should not use the widely known underperforming random remasking as the baseline. In addition, the paper seems to lack a comparison of inference efficiency.\n\n[1] Nie et al. Large Language Diffusion Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)