Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
Authors: Iskander Azangulov
Teodora Pandeva
Niranjani Prasad
Javier Zazo
Sushrut Karmalkar
Venue: ICLR 2026 Poster (2026)
Keywords: masked diffusion models
language models
inference
Primary area: generative models
TL;DR: The paper introduces a training-free sampling algorithm for masked diffusion models that resolves conflicts among proposed tokens by deleting lower-confidence candidates

Abstract: Masked diffusion models (MDMs) offer a compelling alternative to autoregres-
sive models (ARMs) for discrete text generation because they enable parallel
token sampling, rather than sequential, left-to-right generation. This means po-
tentially much faster inference. However, effective parallel sampling faces two
competing requirements: (i) simultaneously updated tokens must be conditionally
independent, and (ii) updates should prioritise high-confidence predictions. These
goals conflict because high-confidence predictions often cluster and depend on
each other, opportunities for parallel updates.

We present PUNT, a model-agnostic sampler that reconciles this trade-off. Our
method identifies token dependencies and removes lower-confidence tokens from
conflicting groups. This produces sets of indices for unmasking that satisfy both
independence and confidence criteria. Our approach ensures improved parallel
unmasking through approximate conditional independence testing.

Our experiments show that PUNT delivers a superior trade-off between accuracy
and compute when compared to other strong training-free baselines, especially for
generation of longer sequences. On the IFEval benchmark, it achieves up to 16%
higher accuracy over baseline methods, including sequential generation (one-by-
one). These gains hold across different values of hyperparameters, mitigating the
need for brittle hyperparameter tuning. Moreover, we observe that PUNT induces
an emergent hierarchical generation strategy, where the model first establishes
high-level paragraph structure before local refinement, suggesting a planning-like
generation process that contributes to strong alignment performance.

Published as a conference paper at ICLR 2026
PARALLEL SAMPLING FROM MASKED DIFFUSION MODELS VIA CONDITIONAL INDEPENDENCE TESTING
Iskander Azanguloy
Department of Statistics, Oxford University
Teodora Pandeva, Niranjani Prasad, Javier Zazo, Sushrut Karmalkar
Microsoft Research Cambridge ftpandeva, niran praapaa javierzazo, skarmalkary@ellorrsfff.coo com is iskander www.aallvvsspc.coooooo ox.ac.uk
ABSTRACT
Masked difffusinn models (MDMs) offer compelling alternative to autoregres- sive models (ARMs) for discrete text generation because they enable parallel token sampling, rather than sequential, left-to-right generation. This means po- tentially much faster inference. However, effective parallel sampling faces two competing requirements: (i) simultaneously updated tokens must be conditionally independent, and (ii) updates should prioritise high-confidence predictions. These goals conflict because high-confidence predictions often cluster and depend on each other, opportunities for parallel updates.
We present PUNT, model-agnostic sampler that reconciles this trade-off. Our method identifies token dependencies and removes lower-confidence tokens from conflicting groups. This produces sets of indices for ummasking that satisfy both independence and confidence criteria. Our approach ensures improved parallel unmasking through approximate conditional independence testing.
Our experiments show that PUNT delivers superior trade-off between accuracy and compute when compared to other strong training-free baselines, especially for generation of longer sequences. On the IFEval benchmark, it achieves up to 16% higher accuracy over baseline methods, including sequential generation (one-by- one). These gains hold across different values of hyperparameters, mitigating the need for brittle hyperparameter tuning. Moreover, we observe that PUNT induces an emergent hierarchical generation strategy, where the model first establishes high-level paragraph structure before local refinement, suggesting planning- generation process that contributes to strong alignment performance.
INTRODUCTION
The widespread deployment of Large Language Models (LLMs) has created massive computational workloads, consuming significant datacenter resources and electricity, thereby incurring substantial operational costs. A primary driver of this inefficiency is inference speed, which is bottlenecked by the sequential, left-to-right generation process inherent in standard autoregressive models. To overcome this, alternative methods have been developed to enable multiple tokens to be generated simultaneously.
Among approaches with the potential for parallel decoding, Masked Diffusion Models (MDMs) have emerged as a particularly promising framework (Austin et al., 2023 Lou et al. 2024 Shi al., 2024 Nie et al.) 20255). Unlike autoregressive models, MDMMs iteratively refine masked sequences by predicting subsets of positions simultaneously, enabling parallel decoding. However, determining which tokens to unmask in parallel without degradation in quality remains challenging.
Various inference strategies have been proposed accelerate MDMs, including confidence-based token selection (Sahoo et al. 2024 Patel et al.) 2025), structured unmasking patterns (Luxembourg
Work done during an intersship Microsoft Research Cambridge.

---

Published as a conference paper at ICLR 2026
al. 2025 Arriola al. 2025). remasking (Wang et al. 2025), and distillation (Zhu et al. 2025b
However, these approaches share a critical limitation: they do not explicitly test for inter-token interference during parallel decoding. Structured patterns impose rigid, dlta-agnostic schedules that ignore sequence-specific dependencies, while remasking and distillation either add computational overhead require expensive retraining.
Our Contribution. We propose a different approach to parallel decoding based on contextual independence -estinn whether tokens can be decoded in parallel by checking for independence the sampled point, rather than for all possible outcomes. Unlike standard conditional independence, which requires integrating over all possible outcomes (which computationally prohibitive for large token spaces), contextual independence provides the part that matters at the current decoding step.
To find the contextually independent subsets, we propose PUNT (Parallel Unmasking with Non- influence Tests), training-free procedure that employs a divide-and-come rithm selects "anchor" subsets and tests entire "candidate" groups for dependence in batch. By carefully designing splits, PUNT certifies a large block of tokens for parallel generation using only O(log m) model calls per step (compared with m for fully sequential unmasking) where m the number of masked tokens.
strategy. Our algo-
PUNT enjoys the following advantages: first, PUNT training-free and requires no model fine- tuning distillation. Second, unlike rigid structured patterns or confidence-basse approaches, PUNT dynamically adapts to sequence-specific dependencies. For instance, we see that it ex- hibits an emergent hierarchical generation strategy. where the model first establishes high-level paragraph structure before refining the details. Third, for long-form text generation on alignment benchmarks well de novo protein generation tasks, PUNT outperforms other standard base- lines and quickly reaches its maximum quality with very few forward passes when compared to other algorithms, resulting in stable Pareto frontier the number of forward evaluations of the MDM.
Organization. Section 2 introducs masked diffusion models and formalizes the parallel decoding problem. Section 3 presents our main algorithmic contribution, Section presents empirical eval- uation, and Section discusses related work. Finally. Section discusses implications and future directions.
2 BACKGROUND ON MASKED DIFFUSION MODELS
In this section, we review the fundamentals masked diffusion models and establish the notation used throughout this paper.
Notation. We denote vectors with bold lowercase (e.g., x) and scalars with regular lowercase (e.g., y); random variables use uppercase (e.g., X, Y) with corresponding lowercase for their realizations (e.g., y). We will also use uppercase letters to denote sets and tensors, when it is clear from context. Let L [8] := {I, denote integers up to L. For I C L, -1 := L \ is its complement, and x
fx"
represents the indexed subset of sequence x = (x ".."....."""
X+) we write p(x) = P(X = x) for outcome x E V4, and p-(-) for the marginal at position j. Extending to VMASK V (MASK), token x² is masked if x² MASK. For any sequence, M = (x) == {i x² = MASK] denotes masked indices, with unmasked indices denoted -M. Conditional distribution at position given observed tokens x/ is written as p² (- x'), shorthand for P(X X x'). For sequence with masked indices M, x denotes unmasked tokens. In the iterative generation process, X; represents the sequence at step and M; == (xx) denotes masked indices at step f.
With vocabulary V. consider discrete state space V For random sequence X = (X
Masked Language Modeling, Masked language modeling trains neural networks predict miss- ing tokens from context. Given sequence x E MASK with masked positions M, the model parameterized by learns conditional distributions:
p |MM) R p'( x-11xx-M MM, Vi M.
During training. a clean sample (i.e. without any masked coordinates) Xelean ~ p(-) is drawn from the true data distribution. A random subset its tokens M C is then selected to be masked, creating the corrupted sequence where x M x deem and x consists of MASK tokens.
2

---

Published as a conference paper at ICLR 2026
The model parameters A optimized to maximize the conditional log- leg-likelhhood of the original tokens at masked positions:
(()) = 1 Ex M Σ log (F)
x mr)
LiEM
Generation proceeds iteratively using the trained model. Starting from fully masked sequence Xo (MASK,... ..,, MASK), each iteration performs two operations at timestep t: (1) sample can- didate tokens yf ~ p-(- x, M)) for all masked positions i M,, and (2) update a subset S M, of these positions with their sampled values, that x + vi for E R and xi+1 < xa for i $ R. This process repeats until all positions are unmasked, producing the final sequence Xy-
The iterative sampling process introduces two key sources of error at each step that can compromise sample quality. For clarity, we analyze the error within a single step and drop the time index t in the following discussion.
Approximation Error. The learned model Pb(- x only approximates the true conditional distribution p'(- x) This potentially leads to suboptimal token predictions. To mitigate this, we employ a confidence score di per position i, to guide mask selection at each step. updating only those positions where the model exhibits high confidence. This strategy improves generation quality by prioritizing high-confidence predictions. Common confidence scores Φ; for position include: negative entropy Enev (z (x: x x-M) log Pb (2 x M), confidence pa (v² x- M) of the sampled token, and top margin between the two most likely tokens.
Joint Dependencies. A more fundamental limitation arises from the sampling strategy itself. When sampling masked tokens y independently from their conditional distributions pà( x-M), we implicitly assume conditional independence among all masked tokens given the unmasked context. Natural sequences, however, exhibit complex dependencies that violate this assumption. True con- ditional independence for candidate tokens R C M requires the joint probability to factorize as:
P
x M) II (-1xx-
iER
In general, the joint distribution does not factorize in this way. Finding subset of tokens where this factorization holds presents significant computational challenges, as verifying this condition requires checking that equation holds for all outcomes y V space that grows exponentially with [R) The following section presents an efficient method to identify token subsets that approximately satisfy this independence condition, thereby mitigating this source of error.
3 METHOD
In this section, we introduce our method for one step of parallel unmasking. We first establish con- textual independence the criterion for safe parallel unmasking ($3.1). then present efficient subset discovery algorithm that identifies independent token sets using only O(log [M]) model evaluations ($3.2
3.1 CONTEXTUAL INDEPENDENCE
To address joint dependencies, we adopt the notion of contextual independence as our criterion for parallel unmasking. This property precisely characterizes when parallel sampling yields the same distribution as sequential sampling. Unlike full statistical independence (overly restrictive) or confidence-baat heuristics (which ignore dependencies), contextual independence identifies tokens that can be unmasked simultaneously given the current context.
Definition 3.1 (Contextually Independent Random Variables), A random variable X is contextually independent a random variable Y at a point if the conditional distribution of X given Y is identical to the marginal distribution of X, i.e., Px y( Y y) (x)
Definition 3.2 (Contextually Independent Sequences). A sequence of random variables (X' X²) is contextually independent at an outcome (x.1 (x!,...,a²), ) if for each i E L, the conditional distribution of X given the preceding outcomes Xei (x) x is identi- cal to its marginal distribution. Formally, for all illi- E L: PxsXxe
xci) (x) where
X (X1 ,...,,.. Xt-1).
3

---

Published as a conference paper at ICLR 2026
U.A.3.4)
Exput
[oAGR]
Compute confdiizee
mope
mep
yiue
egg
66 [1
St e B4
Applly soak (1.5)
mope
Ff (mogano, (he)
Texi (9,4)
nopp
renuo
11
14
Applly monk (1.9)
rocpe
100
Jesi (3.4]
The
ruppe
ranoo
Onrpet
100p
Figure 1: Illustration of one iteration of PUNT on 4 masked tokens, which consists of log, 4 tests. Left: A binary tree representing the recursive partitioning. Each level corresponds to a sin- gle parallel test in the iterative algorithm. Right: each round, confidence-ordess numbers) are partitioned into "anchor" (green) and "test" sets. Test tokens that are dependent on the anchor set are rejected (red), while independent ones (blue) are kept. Each token must pass all independence tests to be accepted. Here, "mince" passes (independent of ("requires", "the") ) while "egg" fails (dependent on ("requires", "mince")). The final set deet(""qquu ("requires", "the", "mince") satisfies contextual independence: p("requires", "the", "mince" Xunmasked) p("requires" Xunmasked) . p("thee Xummaske) p( p("mince" Xunmasked).
tokens (circled
In other words, under the contextual independence assumption, sampling the vector (x (e'....,2"" 2²) sequentially is equivalent to sampling its components in parallel.
Our goal is, given a candidate vector y to find an ordered set of masked indices R Vi.....P[Rri S .M that are contextually independent relative to the unmasked context x-M For- mally, for any i {l...., [R],, the distribution at position r; must be independent of the outcomes at preceding positions Rei == (ry < i}, given the unmasked context:
p- x M y Rei) = ppp(" M).
(2)
A naive, greedy approach to construct such a set would be to iterate through all masked indices m E M and sequentially add an index to R it satisfies Equation indices. However, this requires O([M) sequential model evaluations, which defeats the purpose of parallel sampling.
2 given the previously added
We propose efficient recursive algorithm based recursive divide-ande sonde-and-cooquer strategy, which we will later provide an an efficient iterative implementation for. The validity of this approach relies on the following stability assumption regarding the conditional independence structure of the model.
Assumption 3.3. (Independence Stability) Let i M be masked index, and let U A {i) be a subset of masked indices. If for some sequence of tokens y. we have p y x x-M) = p²
x xM), then for any W C holds that p' y W x M
= p' x_
M-).
This assumption represents a balanced compromise between complete independence and simple contextual independence. It states that if a set of positions U does not influence the prediction at position i, then any subset W C U will also not influence that prediction. This property ensures that independence tests conducted at any stage of our recursive algorithm remain valid throughout all subsequent stages. Section B provides justification for why this assumption is reasonable for transformer-basee architectures and empirical evidence that it approximately holds in practice.
3.2 EFFICIENT SUBSET DISCOVERY
we can construct the set It in Ologg [M] parallel steps. If there is at least one masked position (i.e. JM] Z 1), the recursive algorithm starts with S . and proceeds as follows (see Figure I for illustration of one iteration):
Under Assumption 3.3
Eventually will he ordering these via confidence metrics di, see Section 3.3
4

---

Published as a conference paper at ICLR 2026
At each recursive call, its input is (confidence) P ordered subset of masked candidates S ($1, $2...., $1S)) _ M. The base case for the recursion is when |S] S 1, in which case the proce- dure returns S. For larger sets, the algorithm proceeds as follows:
1. Divide: The ordered imput set S is split into two balanced halves: the "anchor" set So = (s1....,8pp) and the "test" set S = ($pt1)...,, where is split point of the designer's choice.
2. Prune (Filter): The "test" set S1 is pruned based on its dependency on the candidates y For each index i S₁, we compute its new conditional distribution and measure the change from the baseline using the KL divergence:
DKL. (p'(: x xx~ || xx,)
A filtered set S is then formed by retaining only those indices for which the change is below threshold 0: Sj {i S ex <=).
3. Recurse:
The algorithm then makes two independent (parallel) recursive calls: one on the "anchor" set So and another on the filtered "test" set S and obtains Rg and R1 respectively.
4. Combine: The final result R for the input set S is the union (ordered sum) of the outputs R == Ro Li R1 from the two recursive calls above. Note that by construction any token in Rí contextually independent of So and by Assumption 3.3 it is contextually independent of subset Ro So.
Choosing [|S]/2] for each recursive iteration, we ensure that the recursion depth is O(log IM])- In the next section, we discuss how to execute all calls at the same recursion level using a single network evaluation, thereby achieving Olog [M cost per round.
3.3 CONFIDENCE ALIGNMENT AND IMPLEMENTATION DETAILS
This section addresses two key implementation aspects: (i) incorporating confidence-based prioriti- zation into our recursive algorithm maintain generation quality, and (ii) transforming the recursive procedure into an efficient iterative implementation.
Confidence-Ordered Splits. At each recursive step, the candidate S is split into So and Si, and positions in SJ are pruned if they exhibit strong dependence on So. By sorting the initial candidate set S in descending order of confidence (see Section for options), 6s, > Φ.अ > >... 0 we ensure that Sy always contains tokens with at least median-leve confidence. Consequently, tokens pruned from Si necessarily have lower confidence than those retained in Sg. In fact, this also ensures that during each unmasking step, the highest-confidence token in M is always included in the final set R, since will never be pruned.
Binary Encoding of Recursive Calls. To enable parallel computation, we transform our recursive algorithm into an efficient iterative procedure. At each level of the recursion tree, we combine all independence tests into a single, parallel model evaluation.
To see how this might be done, suppose at some recursion level we test pairs (S).S)). (S2. (T).... ..)....(S))) (Sh. St). by construction, these sets form partition of subset of M. We propose, instead of performing these tests independently, to test all "test" tokens against the union of all "anchor" sets Le S. Then, Assumption 3.3 ensures that passing this combined test implies pass- ing the individual tests. Formally, for any l and any E Sf if pb yL x-M) = pb(. x-M,, then it also satisfies (: 56 M) p (: x x-M)
...
Binary Representation of Recursive Splits.
We now describe how this idea can be employed to convert recursive algorithm to iterative one. The recursive splits are determined by each token's 's position in the confidence-ordered list M. This allows us to pre-determine all splits by assigning binary code to each position. More precisely, we assign each position i E {1,..., JM binary representation bin(i) with [log2 [M]] bits, padded with zeros if necessary. Tracking this binary encoding allows us identify the path of each node in the recursion tree.
See Section 3.3 for details
5

---

Published as a conference paper at ICLR 2026
Denoising Speed Depending on Prompt Type
Step
Number of Unmasstdd takens 1%)
whree 1o1.0
Step
Numbee of Known Tokens (%)
Figure 2: Left: Unmasking efficiency for various prompt types number of unmasked tokens.
Right: Visualization of the denoising process at steps 9 and 18 for sample prompt ("Whhtt should I do the end of the internship."). Tokens are color-coded: green tokens are accepted by PUNT for parallel unmasking in the current step, red tokens are rejected, and uncolored tokens were unmasked in previous steps. See Apeendpp D for more examples of intermediate denoising steps.
At recursion level b, we have 2 nodes, with each node indexed by a unique binary prefix u E {0, 1) corresponding to the subset Su {i S [M] prefix,(bin(()) = u). which is then partitioned into "anchor" (Suo) and "test" (Su1) subsets (see Figure [ (Left) for an example).
We would like to combine all 2 tests at recursion level b into single test. To do so, define a global partition for level based on the b-th bit of the binary encoding. B, = fi [M] : the 6-th bit of bin(1) 0}.
Starting with R = M, each round can now partition the current set using the predefined binary split BB: the "anchor" tokens (S. R. Bb) and "test" tokens (51 R B₂). All tokens in Si are tested for dependence on S in a single forward pass, dependent tokens are removed from R. After all log [M| rounds complete, the remaining set R contains only contextually independent tokens.
The resulting procedure, summarized below (and in Algorithm 1.. requires only O(log) MM) for- ward evaluations of the model per denoising step and guarantees that the returned set R consists of contextually independent, high-confidence tokens that can be unmasked in parallel.
Iterative Algorithm. Given confidence-ordered masked tokens M {m1, m,...., m where Øm, 2 0mz² 2 2 Ompar we initialize R t- M and execute [log2 MM] iterations.
m
For each iteration b E (1,..., [ogz [M]]}
1. Test: Partition R. into anchor tokens So = R. n Bi and test tokens S = R. Bb.
2. Prune: For each E S1. compute the KL. divergence dj (xL x x-M) || 1l((
x M, ())) in single forward pass.
3. Update: Remove dependent tokens from R: R + R {j S dj >}}
The final R contains tokens that be unmasked in parallel without interfering with each other. Remark 3.4 (Conservative but Parallel Testing). The iterative implementation uses batched indepen- dence tests: tokens at each tree level are tested against the union of all anchor tokens at that level, not just their specific recursive subset. This stricter condition enables full parallelization. Empirically, this conservative approach maintains strong performance on congecontext tasks while significantly reducing runtime.
3.4 ALGORITHMIC PROPERTIES AND INDEPENDENCE STABILITY
This section discusses PUNT's properties for text generation and justifies Assumption
Adaptive Unmasking. Our sampler exhibits emergent hierarchical generation, first establishing high-level structure (c.g.. paragraphs, headings) before filling in details, as shown in In 2 (right). As can be seen, step 9, the model has already generated the main headings and subheadings of

---

Published as a conference paper at ICLR 2026
Parallel Sampling
depending Previously Revealed
Mean KL Divergence
NFE
Number of Previously Revealed Tokens
Figure 3: Parallel sampling emror (equation B Left: Average enror for different samplers com-
pared to number of forward evaluations (NFEs). Right: Median together with confidence intervals (Qs, Qas) of the error for PUNT samplers with different as function of the number of previously revealed tokens. Note that Q3 remains below 10 across all positions.
the article, while the rest the text remains masked. By step 18, the model has begun filling in the details under each heading.
We hypothesize this behavior stems from the conditional independence between high-level struc- tural tokens and fine-grained details. Because the latter minimal influence the former, the structural tokens pass the independence tests PUNT's filtration stage and are unmasked early. A formal investigation of this phenomenon is left for future work.
This hierarchical generation has a cascading effect. Once revealed, high-level tokens act as contex- tual anchors, partitioning the text into conditionally independent sections. This allows the sampler to unmask tokens in different sections in parallel, adapting its denoising speed to the task's 's inherent structure. As shown in Fig. (left), this results different performance profiles for various prompts.
Independence Stability of Transformers. Assumption 3.3 (Independence Stability) is a direct consequence of the Transformer architecture's attention mechanism. In Transformers, the influence of one token on another is governed by attention weights; if the attention from position to position j is zero, then position j has no direct influence on the representation position
This relationship between attention and influence allows us to connect conditional independence to attention scores. Specifically, we argue that token yi is conditionally independent of a set of tokens y given the remaining tokens x if and only if the total attention from position i to all positions in R is negligible across all layers and heads, i.c..
The conditional distribution plg(- 1 x-Mx
1-M if and only of the cumulative atlention weight from position to all positions in R is negligible.
,xxxxx
M) equals 1
This property directly implies Independence Stability. Since attention weights are non negative, if the attention from position i to set negligible, the attention to any subset C must also be negligible. A detailed justification is provided in Appendix B
EXPERIMENTS
We evaluate our proposed sampler, PUNT, on a number of natural language tasks. Our empirical results validate the effectiveness of our approach and support the attention hypothesis introduced in Appendix B We evaluate: (i) PUNT's performance on long-form text generation tasks such as MTBench; (ii) PUNT's effectiveness short-answer benchmarks for mathematics and code gener- ation: (iii) The error introduced by parallel token sampling and its relationship to the exploration rate e: (iv) Deviation in empirical attention patterns (supporting theoretical independence assumptions).
Experimental Setup. We evaluate PUNT on two powerful, open-source large language models: Dream 7B Ye et al. 2025a) and LLaDA 1.5 (Zhu al. 2025a).
Baselines. We compare PUNT against three strong, training-free baseline samplers. These base- lines include: (i) standard top-k sampling; (ii) the EB-sampler (Patel al. 2025); and (iii) the
Except for padding sedeec--eequeecc(HOOO) tokens, see details in Appendix

---

creative Published as a conference paper at ICLR 2026
0.725
Average
Sampler
DILATED
0.700
8,62
leve
0.008
0.075
200
400
630
800
1000
200
400
600
800
1000
200
400
600
830
1000
Hoan NFFE
Moan NFE
Figure 5: IFEval and MTBench performance of PUNT compared to baselines on Dream 7B, Bench- mark specific scores (higher is better) vs mean number of forward passes.
Dilated- sampler (Luxembourg al., 2025). Note that all the samplers are implemented for a given context length M in non-semi-autee autreeesutoeeressivve manner and the exact parameter configurations are provided Appendix C.3
We also evaluate the performance of PUNT generation in a structured biological domain. Specif- ically, we look at unconditional generation of de novo membrane proteins using McmDLM (Goel et al. 2024), state-of-the-att protein language model. Full details are provided in Appendix C.S
We provide additional analysis in the appendices, including performance-vs-e plots AAppeddxx E,. experiments with varying schedules generation trajectories (Appendix F). and comparisons and discussion with APD (Appendix C.4), proposed by Israel et al. (2025).
4.1 ALIGNMENT BENCHMARKS
We evaluate PUNT on the instruction-followinng benchmarks MTBench (Bai al. 2024) and IFE- val (Zhou et al. 2023). MTBench comprises 80 tasks across diverse domains-inehle writing, logical reasoning, and code cnveration-provide performance. Each task consists of two turns, with the second turn depending the output of the first. IFEval complements this by specifically measuring instruction-following accuracy through a collection of carefully designed test cases that evaluate precise adherence complex instructions. Full benchmark details are provided in Appendix C.I] As shown in Figure PÚNT consistently outperforms all baseline samplers both benchmarks, achieving higher scores across both metrics and mean score; higher is better). Importantly, it delivers these gains while requiring substantially fewer forward evaluations (NFEs).
a robust evaluation framework for model
Note on NFE regimes: We see that PUNT's primary ad- vantage lies in the low-to-mid NFE regime on long- generation tasks. Since each PUNT iteration requires [log2 [M]] forward passes for independence testing (e.g., 10 passes for 1024 tokens), the method excels when the NFE budget allows for meaningful independence check- ing while still providing efficiency gains. At very high NFE budgets (e.g., NFE 400 on MT-Bench), curves may converge cross fixed-geometry schedulers like Dilated can afford many denoising steps, while PUNT's overhead from independence testing becomes proportion- ally larger.
pass, at (mean)
0.15
100
200
300
NFE (mean)
Figure MBPP performance of PUNT compared to baselines on Llada. MBPP pass (higher is better) vs mean num- ber of forward passes.
4.2 SHORT-ANSWER BENCHMARKS
We evaluate PUNT across diverse benchmarks spanning mathematical reasoning (GSM8K (Cobbe et al. 202Th)
and code generation (HumanEval (Chen et al. 2021). MBPP (ustin et al. 2021
As expected, PUNT underperforms short-answer tasks with limited context since it requires multiple forward passes for complete generation. For instance, in the MBPP benchmark on LLaDA with temperature 0.7 (see Fig. 4), PUNT's performance aligns closely with EB when evaluated
8

---

Published as a conference paper at ICLR 2026
by the number of forward evaluations (NFE). However, when measured by the number of denoising steps-that that is, the number of algorithmic steps, where each step may involve multiple forward passes in parallel-PUNN
outperforms other samplers. A simple fix could be to PUNT only for the latter part of the generation, but we leave this for future work. Similar trends are observed across other benchmarks in this group. Results comparing all samplers, models, and hyperparameters appear in Appendix C.3
4.3 PUNT SAMPLER ERROR ANALYSIS
We empirically quantify the parallel sampling error on the LLaDA model. For this, we generate 1024-token responses for MTBench (Bai et al. 2024) with exploration rates E {0.01,.... ...3). Within each parallel generation step, tokens are ordered by confidence before sampling. For the i-th token position r; unmasked in parallel, we compute the error between the true conditional distribution and our independence approximation:
KL DRL. (p |-1x-*
x (1 (
This KI, divergence quantifies the information lost by assuming token r, is conditionally indepen- dent of other tokens unmasked in the same step, y yRs. As shown in Fig. 3 PUNT achieves a low parallel sampling error while maintaining a small NFE compared to other samplers. Furthermore, the parallel decoding error, ŌKL, remains robustly below the threshold, irrespective of the number of tokens previously revealed in the step.
RELATED WORK
Our work builds on recent advances in discrete diffusion models and inference-time time planners.
Masked Diffusion Models (MDMs). Discrete diffusion models (Austin et al. 2023) offer a non- autoregressive alternative for text generation. Training objectives based on score matching (Lou 2024) and masked language modeling (Sahoo et al., 2024) have enabled large-scale models like LaDA (Nie et al. 20256) and others (Nie al. (2025a). Commercial implementations include Gemini Diffusion (Google DeepMind] 2025) and Mercury (Inception Labs (025). MDMs face two key limitations: compounding errors from parallel unmasking and inefficient KV caching. We ad- dress the former by identifying token sets for safe parallel unmasking, which minimizes interference
et al.
and improves both efficiency and quality.
Inference-Timee Time Planners for Acceleration. Efficient inference scheduling remains MDMs' tral challenge. Various training-free planners aim to minimize function evaluations (NFEs) while maintaining generation quality.
Confidence and Entropy Gating. Confidence-based scheduling iteratively unmasks tokens with high- model confidence (or lowest entropy) (Sahoo al., 2024). The EB-Sampler extends this by dynamically unmasking variable-sized token sets whose aggregate entropy stays below threshold (Patel et al. 2025). While adaptive, these methods remain conservative, ignore token independence. and typically unmask only small subsets.
Remasking and Refinement. Several methods correct parallel decoding errors through remasking. ReMDM (Wang et al., 2025) iteratively remasks and updates generated tokens. Path-Planning (P2) (Peng et al., (225) and DDPD (Liu al.) (025) separate inference into planning (selecting tokens to update/remask) and denoising stages. While improving quality, these approaches increase NFE
through corrective passes.
These fixed-geometry (non-adaptive) methods enforce spatial separation be- tween parallel unmaskings. Dilated scheduling unmasks non-adjacent token groups for improved stability (Luxembourg et al.) 22025. baased schedulers use low-discrepancy sequences for uniform spacing (Besnier et al. (2225). Block Diffusion balances AR and parallel generation by processing contiguous spans (Arriol al. 2025).
Spacing Schedulers.
Analysis of Ordering and Scheduling. Recent theoretical and empirical work has deepened the com- munity's 's understanding of these schedulers. Kim et al. (2225) study the impact of token ordering, showing that adaptive inference can sidestep computationall hard subproblems. Park et al. (2024) focus optimizing the temporal schedule (the number and placement of diffusion steps) reduce

---

Published as a conference paper at ICLR 2026
Others have explored MDLMs for complex reasoning, where planning is critical YYe al. 2025b), and for specialized domains like code generation (Gong et al. 2025
NFEs.
Comparison to Autoregressive Accelerators. While autoregressive models like LLaMA-3
(Grattafioni et al. 2024) are accelerated by speculative decoding (Leviathan et al. 2023; Xia et al,
2022). this approach remains fundamentally sequential. In contrast, method reduces NFEs by
leveraging the non-sequential, any-order generation capabilities of MDMs. Orthogonal optimiza- tions like KV caching are applicable both paradigms (Ma al.
2025
Hu al. 2025).
CONCLUSION AND FUTURE WORK
We introduced PUNT, a training-free sampler that looks to resolve the conflict between speed and quality in MDMs by efficiently identifying sets of approximately conditionally independent tokens for parallel unmasking. This enables significant reduction the number of model evaluations needed for generation while preserving output quality. We provided conceptual justification for its applicability to transformer architectures and validated its effectiveness on mathematics, code, and long-form text benchmarks. We also observe that PUNT induces an emergent hierarchical genera- tion strategy: coarse paragraph structure is established early, followed by localized refinement.
Future work can extend this approach in several directions: (i) developing adaptive or curriculum- style schedules for the independence threshold to balance early exploration with late precision; (ii) distilling PUNT into a student model that predicts contextually independent reveal sets in a single forward pass; and (iii) combining PUNT with orthogonal efficiency techniques such as K KV-caching, to further shift the securacy-compute Pareto frontier.
Reproducibility. We provide detailed descriptions of our algorithms and experimental methodol- ogy throughout the paper. The core PUNT algorithm is presented in Section [3 and in |Algorithm 1 with the recursive structure detailed Section 3.2 Our implementation of binary encoding for
efficient parallel pruning is described in Section
3.3
Our experimental setup is described Section with comprehensive benchmark details provided in Appendix C
REFERENCES
Miguel Arriola, Aaron Gokaslan, James T. Chiu, Zhilin Yang. Zichao Qi, Jialiang Han, Subhrajit S. Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffuuson language models. arXiv preprint arXiv:2503.09573, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Jonathan Terry, Quae V Le, et al. Program synthesis with large language models. arXiv preprint (00.v:::08.077:: 2021.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, arXiv preprint 117.:2207.03000 2023.
Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, al. Mt-bench-101: fine-grained benchmark for evaluating large language models in multi-tum dialogues. arXiv preprint arXiv:2402.1 14762, 2024.
Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer, 2025. URL
77766
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Wojciech Zaremba, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 10.1101.44 14168, 2021.
10

---

Published as a conference paper at ICLR 2026
Shrey Goel, Vishrut Thoutam, Edgar Mariano Marroquin, Aaron Gokaslan, Arash Firouzbakht, Sophia Vincoff, Volodymyr Kuleshov, Huong Kratochvil, and Pranam Chatterjee. Memdlm: De novo membrane protein design with masked discrete diffusion protein language models. arXiv preprint arXiv:2410. 16735, 2024.
Shuyang Gong, Ruiqi Zhang, Linjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation.
arXiv preprint arXivv2506.20669, 2025.
Google DeepMind.
Gemini diffusion.
https: http:://Beemm.ad.ggglle/oooooo
gemini ni di fusion/ 2025.
Andrew Grattafiori, Abhimanyu Dubey, Abhishek Jauhri, Abhishek Pandey, Abhishek Kadian, An- thony Al-Dahle, Ariel Letman, Ayush Mathur, Armin Schelten, Austin Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Zihan Hu, Jiarui Meng, Yogesh Akhauri, Mohamed S. Abdelfattah, Jae-sun Seo, Zongyang Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. arXiv preprint arXiv:2505.21467. 2025.
Inception Labs. 2025.
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding. arXiv preprini wrXiv:2506.00413 2025.
Jachoon Kim, Kunal Shah, Vasileios Kontonis, Sham Kakade, and Sham Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint 067708,002.06688 2025.
Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast inference from transformers via speculative decoding. In International Conference Machine Learning, pp. 19274-19286. PMLR, 2023.
Zeming Lin, Halil Akin, Roshan Rao. Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with language model. Science, 37966637):1123-1133 2023.
Sulin Liu, Juno Nam, Andrew Campbell, Hannes St"ark, Yilun Xu, Tommi Jaakkola, and Rafael Joanez-Bombarelit Think while you generate: Discrete diffusion with planned denoising. arXiv preprint arXiv:2410.0622
2025.
Alex Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310, 10. 16834, 2024.
Omer Luxembourg, Haim Permuter, and Eliya Nachmani. Plan for speed: Dilated scheduling for masked diffusion language models, 2025. URL BtLLtttss///xxxy>> http://wwxiv......aaaa wwtp://.r/aaaaaaaa/e 2506 120556666177777
Xiaoyu Ma, Rui Yu, Guanchu Fang, and Xuan Wang. dky-cache: The cache for diffusion language models. arXiv preprint 1xXx::2505.177.1 2025.
Shengqi Nie, Fenglin Zhu, Chengpeng Du Tianyu Pang, Qi Liu, Gang Zeng, Min Lin, and Chen- guang Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514 2025a.
Shengqi Nie, Fenglin Zhu, Zhen You, Xin Zhang, Jing Ou, Jing Hu, Jun Zhou, Yichang Lin, Ji-Rong Wen, and Chenguang Li. Large language diffusion models. arXiv preprint 2xXiv:2502.09992 2025b.
Yong-Hyeok Park, Chin-Hui Lai, Shota Hayakawa, Yuki Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. arXiv preprint arXiv:2410.07761, 2024.
11

---

Published as a conference paper at ICLR 2026
Soham Patel, Zander Bezemek, Fei Z. Peng. J. Rector-Brooks, S. Yao, A. J. Bose, A. Tong, and P. Chatterjee. Accelerated sampling from masked diffusion models via entropy bounded un- masking. arXiv preprint arXhv:2505.24857, 2025. tRtps://///axx www.vvv.rggabbaaa.
24857
Fan Z. Peng, Zoran Bezemek, Sachin Patel, Jason Rector-Brooks, Shunyu Yao, Allen Tong, and Pradeep Chatterjee. Path planning for masked diffusion model sampling. arXiv preprint arXiv:2502.03540, 2025.
Qwen, An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report, January 2025. URI
org. abs 241.. 15115 MrXiv:2412.15115 [es].
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024. URW.tttps:
/ttttps://axxxv...m org. abs/ 2406.075244
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and gen- eralized masked diffusion for discrete data. In The Thiny-eighth Annual Conference Neu- ral Information Processing Systems, 2024. TRL.http::///t /tattss///ppenreieww...o hwtp://wwwwe.eo.tteeeeeee XcqSOfät
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL htttt://///aaxiv
org/abs. 1706. 03762
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking dis- crete diffusion models with inference-time scaling, 2025. URL httth:::///arttt org/ /bs/
LI Wenran, Xavier Cadet, Cédric Damour, LI Yu, Alexandre de BREVERN, Alain Miranville, and Frédéric Cadet. Benchmark of diffusion and flow matching models for unconditional protein structure design. In JCML 2025 Generative Al and Biology (GenBio) Workshop. 2025.
Han Xia, Tao Ge, Peng Wang, Shuming- Qin Chen, Furu Wei, and Zhifang Sui. Speculative de- coding: Exploiting speculative execution for accelerating seq2seq generation. arXiv preprint arXiv:2203. 164.2 2022.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025a.
Jiayi Ye, Jianfei Gao. Shiyang Gong, Liyiming Zheng, Xiang Jiang, Zhen Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning, arXiv preprint arXiv:2410.14157, 2025b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems. 01446595-466623 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooiingg evaluation for large language models, 2023. URL https:
/arxiv. /arxiv.org/abs/2311 07911
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505.19223 2025a.
Yifei Zhu, Xue Wang, Stéphane Lathuilière, and Vassileia Kalogeiton. Di[m]o: Distilling masked diffusion models into one-step generator. arXiv preprint arXiv.2503114557, 2025b. URL
org/ hwtpg//b//55500 15457
12

---

Published as a conference paper at ICLR 2026
Appendix
USE OF LARGE LANGUAGE MODELS
During the preparation of this work, the authors used large language models (LLMs), including Google's 's Gemini and OpenAl's GPT-5, as writing, search, and coding assistants. The use these tools was restricted to the following tasks: improving the grammar, clarity, and readability of the text; assisting with the discovery of related literature to support our literature review process; and generating code for implementation and visualizations of experiments. The authors have reviewed and edited all model-generated content and take full responsibility for the final content of this paper, including any errors omissions.
A ORGANISATION OF APPENDIX
The rest of the appendix is organized as follows. In Appendix B] we justify Assumption 3.3 by demonstrating that it holds for Transformer-bast based masked language models, which is direct con- sequence of the Transformer's attention mechanism. In [Appppddix G. we provide additional experi- mental details and results. In Appendix C.5 we provide masked diffusion model. In Appendix D
protein we provide two examples of text that is generated by
preliminary experiments
PUNT.
Remark on Notation: In addition to standard notation as defined in the paper. in the appendix, we will also use upper-case bold letters (such A) to denote tensors. We will lowercase and unbolded letters to denote scalars (such as A;y). In addition, we may have uppercase letters (such as Q, K, V) annotations to help annotate different matrices. This is accommodate standard notation used in the literature.
B INDEPENDENCE STABILITY
In this section, we demonstrate that Assumption 3.3 holds for Transformer- masked language models, which is a direct consequence of the Transformer's attention mechanism. Let us start with recalling the assumption.
Assumption B.1. (Independence Stability) Let E M be a masked index, and let U CM \ {i) be a subset of masked indices. If for some sequence of tokens yd we have p'(- yy x x-M = p²
xMM), then for any W C holds that p'( W x M) = p x M).
Next, we recall the design of attention mechanism and discuss prior works
Aitention-Basse Independence. In transformers Vaswani al. (2023), trol information flow between positions. For an input sequence X (X attention head computes query, key, and value vectors for every position:
the attention weights con-
x²) E R L.xdi each
Q' X x
K w
V w
where Wº WK E Rde and V Rdd xdd. are learned weight matrices.
The attention mechanism then computes pairwise attention scores between all positions through scaled dot products:
QK T
= softmax
E RLLX
Vd
where Q, E Lxdx stack the query and key vectors across all positions. The attention weights A quantify how much position influences position i, computed via normalized dot-product similarity. The output of one head combines value vectors weighted by these attention scores: head, = AV e LLdd Finally, outputs of different heads are stacked to get, Z₁ = concat (head", ..., .. head',). The output the layer Y = (Y' ...... Y" ) IR Ixd calculated as Y = F(Z)) by application each of the coordinates of ML.P together with normalization layers and skip connections.
13

---

Published as a conference paper at ICLR 2026
Attention Weights KL. Error by Pruning Level
Figure 6: Joint distribution of ōKL the sampling error and 8 the total attention the previous tokens revealed parallel.
Query
Layer
23
Density
0.3
po
Normliie Diflererees
Nomelied Gillerencas
Rnrmwlleed Diflerences
Narmeloodd Diflerencee
PUNT 0.61
PUNT 0.04
PUMT 0.16
PUNT 0.64
Figure 7: Difference between attention statistics for pr (.) x-M y Res) relative to the same statistics computed while evaluating p. x x-M) ).
Crucially, the attention weights AG directly control information flow: when As = 0, position j's value vector V contributes nothing to position i's output. The model's final predictions are obtained by applying softmax to the last layer's output: pb(- x-M) softmax(Y" Therefore, (- x-M) ) = (- y x x-M holds if and only if Y remains unchanged when tokens at positions R are revealed.
Stability of Unmasked Tokens. Recent works Hu (2025 Ma al. (2025) have demon- strated that during iterative inference, the query, key, and value vectors (Q- K M V-M) for already unmasked tokens, remain stable and can be cached for computational efficiency.
First, we discuss padding end-of-sequence (EOS) tokens, which are used to fill the unused suffix reserved for an answer. By design, if there is an EOS token in x M to the left of position i then p' (EOS = M) 1 and the assumption automatically holds.
Why Independence Stability Holds. Let us return to Assumption 3.3
For regular tokens, note that the stability property implies that in both cases, when condition on (y x xxMM, M) or (x- (x-M), the representations (Q MM K M V-M) stay the same, while the main change happens for tokens in U.
The stability property allows us to concentrate on the information flow between position i and tentatively unmasked subset y" which recall governed by attention weights vector A AAu Specifically, we argue that token is conditionally independent of set of tokens " given the remaining tokens x M if and only if the total attention from position i to all po- sitions in U is negligible across all layers and heads, or more formally, A = Enev Aiu < for some small 8 > 0.
14

---

Published as a conference paper at ICLR 2026
Now consider any subset W C U. The non-negativity of attention weights (a direct consequence of the softmax operation) yields the inequality:
Awwll = Σ
An
A
Σ Ar = 11wll <0 wEt
WEW
This demonstrates that if position pays negligible attention to the entire set U. it necessar- ily pays negligible attention to any subset W C U. Consequently, the conditional distribution at position remains approximately invariant when conditioning tokens at positions in W: P-(- y x- x x-M). This relationship directly corresponds to Assumption 3.3 (Inde- pendence Stability).
Empirical Validation We use the same setup as was used in Section 4.3) we use the first round requests from MT'Bench, and sample the responses using the PUNT algorithm with different thresholds
as source of prompts,
(0.01, 0.04, 0.16, 0.64).
For step of the PUNT sampler with threshold let R denote the set of tokens unmasked at this step, sorted according to the confidence, y denotes the set sampled candidates, and x denotes the set of already revealed tokens.
As we demonstrated
Fig. sampled tokens satisfy
KK
DKL (p"+ x *==1 p" x-M
AM .yBr)) <.
For each token we compute the total attention from token to previously revealed tokens Re heads the last layer, i.e.
8 = ||Ar, = Σ Arrr
and plot Fig. 6) the distribution of pairs (ŌKI., dA) for different thresholds.
We also compute the change of the layer output Y and how it changes when we condition on yRs We use the normalized difference metric to compute the change, which is defined as Lewnaliizedddifffeeeeee..
b) := |a - /|al// and plot (Fig. 7) the distribution of the change. Finally, similar to unmasked tokens, observed that representations , V of masked to- ken r; in the attention layer also stays stable when we additionally condition on previously revealed
tokens Rei,
Algorithm PUNT (Parallel Unmasking with Non-influence Tests)
1: Input: masked sequence x, vector of candidates y, threshold
2: Output: certified set R C M to unmask in parallel
Sort masked indices w.r.t. confidence heuristic decreasing order
4: Construct M, the set of all masked indices.
5: x- M
6: Let By : (i E [M]] the b-th bit of bin(t) 0}.
7: for in [log MM] do
8:
So Rr Bs;
(positions to tentatively unmask) (positions to check dependence)
9:
SI + R Bsi
10:
for each j S1 do
11:
dy + DKL (p'
x M)
(. x M M
if d, then
13:
R + R (i)
end
15: end for
end for
17:
return R
C IMPLEMENTATION AND EXPERIMENTS
This section evaluates the proposed planner PUNT (Algorithm) across diverse sequence generation tasks. All experiments are conducted A100 GPUs with 40GB memory.
15

---

Published as a conference paper at ICLR 2026
Experiment
NumFewshot
length
GSM8K
4
512
HumanEval
512
MBPP
512
1FEval
0
1024
MT-BENCH
1024
Table Experimental configuration for each benchmark task.
PUNT offers clear win in step efficiency without compromising quality. However, this is not indicative of the underlying compute used, which is better captured by the number of forward passes (NFE). In terms of NFE, it performs competitively, and particularly long- -eng--eeuence tasks, it often surpasses the baselines. We leave further per-step optimisation for future work.
C.1 EXPERIMENTAL SETUP
We evaluate state-of-the-arr discrete diffusion models for natural language: LLaDA-1.5 (Zhu et al. 2025a) and Dream-v0-Instrutt-71 (Ye et al. 2025a) (referred to as Llada and Dream, re- spectively). In this section, we detail the experimental setup, including tasks, datasets, evaluation metrics, and baseline methods.
TASKS AND DATASETS
We assess PUNT's performance on variety of sequence generation tasks. The evaluation relies on the following standard public datasets and their corresponding protocols:
Math word problems and formal math: GSM8K (Cobbe al. 2021). MATH (Hendrycks
et al. 2021
Code generation: HumanEval (Chen et al. 2021 and MBPP
Austin et al. 2021
instruction-folloinng evaluation: IFEval Zhou al. 2023)
Open-ended question benchmarks: MT-Bench (Zheng al. 2023
EVALUATION METRICS AND CONFIGURATION
We use task-specific evaluation metrics and measure efficiency in terms of the number of forward evaluations and the number of iterations PUNT takes.
Quality Metrics:
Math problems: Match accuracy (GSM8K)
Code generation: Pass@1 success rate (HumanEval, MBPP)
Instruction following: Strict/Loose prompv/instuuttion adherence (IFEval)
Open-ended generation: GPT-4o scoring 1-10 (MT-Bench)
Efficiency Metrics:
Number of network function evaluations (NFE) per sequence
Number of generation steps (PUNT-specificc)
BASELINE METHODS
We compare against representative training- free schedulers with the following parameters:
Top-k Sampler with A = 1,2,3,4,5,
EBB-Sampler (entropy-bounded unmasking) with = 0.01, 0.05, 0.1, 0.5, 1.0, 2.0, 4.0 tel et al. 2025);
Pa-
16

---

Published as a conference paper at ICLR 2026
Preometry-wwar spacing: dilated with log window size in (3,4, 6, (Luxembourg ct al. 2025 )
Each of these baselines utilizes confidence score to rank positions by certainty. Different options for the confidence score are described below.
CONFIDENCE SCORING STRATEGIES
All confidence scoring strategies operate the model's output probability distribution. For each position in a sequence, the model produces logits ls,t.v for every token e in the vocabulary. These converted into probability distribution using the softmax function:
cla
P6,5,0
Σ elaw
From this distribution, compute scalar confidence score Sh,t that quantifies the model's certainty at that position. A higher score indicates greater confidence, prioritizing that position for earlier unmasking. To define the scoring strategies,
the following notation:
Pb,t,(8)i The k-th largest probability at position t, such that Pb,t,(1) Pb,1,(2) 2 Po)+,VV)-
Nb, The token actually sampled at position f.
Negative Entropy
V
Sb,6 Pb,t, log Pb,t, = (po,)),
H(b,,) = Σ
Pb,c,v log Pb,t.e-
This is the negative Shannon entropy. Values lie in [- log V, 0]. Scores closer to 0 correspond to more peaked (certain) distributions.
Top Probability
Sb,t max Pb,t, Pb)t,1)*
simple peak-confidence hcuristic. Ignores how close competitors are.
Top Probability Margin
St,t Pb, Po.1.2)-
Measures local ambiguity between the two most likely tokens. Larger margin = clearer preference.
Positional Schedule
$b,8 t.
A deterministic curriculum ignoring model uncertainty (e.g. left-to-right). Negate or reverse indices the opposite order desired.
C.2 IMPLEMENTATION DETAILS
Our implementation of PUNT follows the procedure outlined in Algorithm T comparison, both PUNT and the baseline methods use the same confidence scoring strategy for each model. Specifically, we use the top probability margin for LLaDA and negative entropy for Dream.
ensure a fair
SAMPLING AND TEMPERATURE SETTINGS
All methods employ nucleus sampling with nucleus set to 0.9. We present results for two temperature settings: 0.1 (low temperature, focused sampling) and 0.7 (higher temperature, more diverse sampling) evaluate robustness across different generation regimes.
17

---

Published as a conference paper at ICLR 2026
6.3
DMM 0MM, (moar))
DILTTED
02
0.2
0.2
PLNT
-82
900
150
200
400
100
MFF (mwwan)
NFE (mween)
Densienng staps (riean))
Denoising steps (meent)
(a) GSM8K results at temperature 0.1
gumbi
strie marth goatt Lntte (nee))
0.-
0.
0.5
AZD
DEAARD
64
04
041
0.1
300 400 NIF (meen)
600
400
320
400
330
100
300
NFE (meen)
Denolsing stope (meen))
Denoising stops (meen)
GSM8K results temperature 0.7
Figure 8: GSM8K performance comparison across different temperature settings showing NFE/steps vs match accuracy (feexible-extract filter)
END-OF-SEQUENCE HANDLING
To prevent premature termination, down-weight positions corresponding end-of-sequence to- kens when early termination is undesirable: yb, equals special end-of-sequence token EOS and early termination is undesirable, enforce
$b,1 + Cueg
Caeg < O,
to deprioritize revealing that position.
C.3 RESULTS AND ANALYSIS
We present our results grouped by sequence length, as this factor significantly impacts the relative performance of the scheduling methods.
SHORT-SEQUENCE BENCHMARKS
We evaluate PUNT GSM8K, HumanEval, and MBPP -all tasks with sequences shorter than 1024 tokens (see Table D. These benchmarks test mathematical reasoning and code generation capabilities under constrained generation lengths.
When measured by the number of generation steps, PUNT consistently outperforms all baseline methods across both temperature settings (0.1 and 0.7). However, when evaluated by NFEs per sequence, PUNT shows competitive but not dominant performance. PUNT's strength lies in reducing the number of sequential generation steps through aggressive parallelization, but each step may require more network evaluations due to its comprehensive independence testing.
Results:
BENG-SE BENCHMARKS
For longer sequences (2 1024 tokens), we evaluate on MT-Bench and IFEval. These tasks require sustained coherence and complex instruction following over extended generation windows.
MT-Bench Results: MT-Bench consists of open-ended questions spanning creative writing, rea- soning, and coding. Each question includes two rounds, where the second builds upon the first response. Answers are evaluated by GPT-4o using a 1-10 scale. All experiments are carried out with temperature 0.7.
18

---

Published as a conference paper at ICLR 2026
Utds
Dieam
Usde
Dream
Cre01 2952 pa0000 (rrem))
14
014
Semple
0/2
0.2
TONK
0.1
MFE (mwwan))
NFE
Denstuing (mwwan)
Demttiiing (roman))
(a) HumanEval results at temperature 0.1
2.40
0.51
APO
DILTTD
0.0
0.20
2.2
3.20
ESPUNTHANLLE
010
0.10
0.0
332
PURT
0.0
6.00
0.00
3.0
200
400
1200
500
3993
100
(11907)
NFL. (meen)
Dencning abacc (mrenn)
Jendarng stapa (mmeat))
(b) HumanEval results temperature 0.7
Figure 9: HumanEval performance comparison across different temperature settings showing NFE/steps vs Pass @1 success rate for both LLaDA and Dream models
mbpp
Unds
Dieam
Ubda
Dream
60-
0.6
06
0.1
DM0 JC (meer))
Sample
ATO
0.5
000
0.5
0.1
0.1
0.3
330
200
100
(a) MBPP results at temperature 0.1
Utda
Dieam
Uido
Dream
0.5
0.5
0.5
0.5
D31 2MT_,K1 (mremn)
Somple
0.3
0.3
0.3
033
TOORS
0.1
0.3
0.1
03
200
Denclsing
Denoiling (mean)
(b) MBPP results at temperature 0.7
Figure 10: MBPP performance comparison across different temperature settings showing NFE/steps vs Pass @ success rate for both LLaDA and Dream models
19

---

Published as a conference paper at ICLR 2026
Turn
Turn
Average over Turns (NFE & Mean)
Sampler
DILATED
EB PUNT
4-
3-
2-
200
400
200
200
1000
Mean NFE
Dream model results
Turn
Tum 2
Average Turns (NFE & Mean)
Sampler
DILATED
PUNT
4-
TOPK
Mean
3-
2-
1
200
400
600
800
1000
200
400
600
800
1000
0
200
400
600
800
1000
Mean NFE
(b) LLaDA model results
Figure 11: Performance comparison for MT-Bench across different models showing NFE vs mean performance
Figure show that PUNT excels particularly when NFE budgets are severely constrained. In low- NFE regimes, PUNT significantly outperforms all bascline methods. As the NFE budget increases, dilated sampling begins to show competitive performance, but PUNT maintains its characteristic stability advantage.
IFEval Results:
The instruction-folllwwe following evaluation tests adherence to specific formatting and content constraints. PUNT demonstrates consistent accuracy across both NFE and step-bau metrics, again showing its reliability advantage.
PUNT's results on IFEval demonstrate its stability across different computational budgets. As shown in Figures an] 13 it consistently leads in generation steps both temperatures, without compro- mising accuracy. Additionally, PUNT is more NFE-efficient at lower budgets and remains competi- tive the budget increases, pulling ahead temperature of 0.7.
C.4 APD EXPERIMENTS
We evaluate the performance of PUNT against Adaptive Parallel Decoding (APD) proposed in (Is- rael (2025)). APD runs have been added to all benchmarks running on Dream Yee et al. 2025a). There are no comparisons on benchmarks running on I.lada (Zhu et al. 2025a). APD re- quires a trained autoregressive model using the same tokenizer as the diffusion model, no such compatible autoregressive model exists, noted in (Isracl al. 2025).
APD requires two NFEs per sampling step: one to draw logits from the diffusion model: a second full decoding pass from the autoregressive model, build the target distribution. Both distributions are sampled using Fumbel-Softman trick, and tokens are accepted when the samples coincide in both processes, using left-to-right decoding scheme. When plotting APD's accuracy NFEs, multiplied by the denoising steps used by APD to obtain NFEs: one for the diffusion model and another construct the target distribution from the autoregressive model.
20

---

Published as a conference paper at ICLR 2026
Meval
1091 per (PANNNT)
200
002
1000
1200
1400
400
1000
NFE (maan)
Denoiing stops (mean)
Sampler
8.5
0.5
200
000
1200
1200
1400
400
1000
MFF (mean)
Denniing steps (mman)
peompt 14F
0.5
0.-
200
600
062
1000
1200
3000
400
1000
Model Dream
NFE (mean)
Denoising steps (meen)
0.5
0.5
0.4
1000
1050
NFE (mean)
Denolsing steps (mean)
Figure 12: IFEval results showing NFE/steps accuracy, temperature 0.1.
21

---

Published as a conference paper at ICLR 2026
0.55
245
(elel Jons
0.55
035
200
1000
1400
NFE (rean))
Denelsing stops (mean)
1nd 34C (10N601)
Sampler
1080
200
AP
0.50
550
250
400
800
1000
1220
1400
1000
200
420
650
1000
NIE Irrean)
Denoising steps (mean)
1180
poompe (evvl) boow
1050
050
0.40
0.4
200
150
800
200
1000
1200
1600
1000
200
400
600
000
1000
Nodel Dream Liada
NFF (rrean)
Denolsing steps (mean)
cromet StPic arc (mwea)))
1235
0.50
0.0
0.35
233
230
4001
160
100
1000
1.20
1400
1160
210
420
100
1000
Denolsing steps (mean)
Figure 13: IFEval results showing NFE/steps vs accuracy, temperature 0.7.
22

---

Published as a conference paper at ICLR 2026
All experiments discussed below refer to the Dream architecture. As illustrated in Figs. 12 and 13 PUNT demonstrates superior performance on IFEval, achieving the highest scores among all meth- ods. This indicates strong capability in adhering to complex constraints and instructions. APD shows the strongest performance HumanEval Fig. 9a and Fig. 9h) EB and PUNT surpass APD MBPP Fig. T0a at temperature 0.1, and APD surpasses them temperature 0.7. EB surpasses APD and PUNT on GSM8K at temperature 0.1, see Fig. 8a and APD surpasses all other samplers temperature 0.7, Fig. 8b
Dream is diffusion model whose weights have been initialized from a trained autoregressive model (wen al., 2025). We hypothesize that Dream retains good left-to-right sampling performance because of its weight initialization, and APD's performance benefits because of its strict left-to-right sampling order when used with Dream. In contrast, all other samplers operate independently of the diffusion model's training procedure, allowing them to be applied more broadly.
C.5 MASKED DIFFUSION MODELS FOR PROTEINS
Masked diffusion models (MDMs) have demonstrated effectiveness beyond natural language pro- cessing, particularly generating biological sequences such as proteins and DNA. To evaluate PUNT's performance in a structured biological domain, we conduct experiments on de novo mem- brane protein design using MemDLM (Goel et al. (024)), a masked diffusion model that finetunes the state-of-the-ard ESM-2 150M protein language model (Lin et al. 2023) with MDM objective to generate realistic membrane proteins.
C.5.1 EXPERIMENTAL SETUP
We evaluate PUNT on unconditional protein generation with sequences of up to 1024 amino acids, comparing against three established training-free schedulers: Top-k sampling. Cintroy--ounnh (EB) unmasking, and geometry-aware (Dilated) spacing. All methods employ a temperature 0.8, to encourage sequence novelty, and suppress end-of-sequence tokens promote longer, more realistic protein sequences. For each sampling strategy, we generate 50 amino acid sequences using the following hyperparameters:
PUNT: {0.001, 0.004, 0.01, 0.02, 0.04, 0.08, 0.16)
Top-k: k (1,2,3,4,6,8 8, 12]
EB Sampler: {0.11 5, 10}
Geometry-aware spacing: log (3, 6,,4,5,6,74. 7,
C.5.2 EVALUATION METRICS
We assess PUNT's performance across two key dimensions critical for practical protein design ap- plications (Wnrran et al. 2025):
Computational Efficiency: As with the natural language benchmarks, we measure the number of forward evaluations (NFE) and denoising steps required for generation. NFE represents the to- tal number of model forward passes needed to complete sequence generation, providing a direct measure of computational cost. Denoising steps (PUNT-specific) track the number of iterative finement steps in the masked diffusion process.
Structural Validity: Generated protein sequences are evaluated for their likelihood to fold into stable, well-defined three-dimensional structures. We feed each generated amino acid sequence to ESMFold (Lin et al. 2023) to predict the corresponding 3D protein structure. We then calculate the mean pLDDT (a per-residue measure of local confidence in the structural predictions) across all residues in each predicted structure.
C.5.3 RESULTS AND ANALYSIS
Figure ploto the mean PLDDT against NFE and number of denoising steps. We find that while the LLDD of generated structures is low across denoising methods- which may be attributed in part to
23

---

Published as a conference paper at ICLR 2026
Mean PLDDT Overall Evaluations
Mean PLDDT vs Steps
0.26
0.25
0.25
Mean PLDDT
PLDDT
0.23
0.23
DILATED
DILATED
PUNT
PUNT
0.21
TOPK
0.21
TOPK
200
400
600
800
1000
200
400
600
800
1000
NFE (mean)
Denoising steps (mean)
Figure 14: Protein generation with McMDLM: mean pLDDT vs (a) NFE, and (b) denoising steps.
our use of non-semi wew---mmmm-ttegesssiiiie generation strategy, or because of the very long sequence length and absence of multiple sequence alignment in ESMfold making this challenging domain for 3D structure prediction-PUN
consistently generates proteins with comparable or marginally higher PLDDT than the baseline samplers given the same computational budget, with stable performance across a broad range of NFE. These results suggest that PUNT is able to improve efficiency without sacrificing structural plausibility, making well-suited for rapid proposal of candidate proteins for downstream analysis.
D DENOISING PROCESS
this section, we show examples of our denoising process starting from a completely masked response for three prompts from different domains.
TEXT PROMPT
The first prompt is story generation prompt: "Compose engaging story about recent trip to Hawaii, highlighting cultural experiences and must-see attractions."
MATH PROMPT
The second prompt math word problem: "Natalia sold clips to 48 of her friends in April, and then she sold half many clips in May. How many clips did Natalia sell in altogether in April and May?"
PROGRAMMING PROMPT
The third prompt is programming prompt: "Write a function to find the minimum cost path to reach (m, n) from (0, 0) for the given cost matrix cost]] and position (m, n.) in cos[[[]."
E PLOTTING PERFORMANCE VS E
In this Figures 15 to 18 we plot the performance of different models as a function of the closeness parameter for two different temperature settings: 0.1 and 0.7.
24

---

Published as a conference paper at ICLR 2026
Dunoking Piogrss Rasqgae deppi
incerurtton: consosee mignctigg traves bang waout recest trip tmaads, nignstgeetng culTara: Higer Secces atrencttane.
Respensm piaae Daa (N00 (tw (tw ( 2 Te Lnaae (
RMSA [MM
AMSA
(nsse (AANS (MAS
(MASEJOWSE:
(ASSE
Closue Jowse Tanne p piocse 20 oo J Doecs meor Eracie Dooss fooou Foncee Eeoos Dwec 20w.0 (recs 3 Soass Toasee
Disse ponse passe 20wass poa pusse paass (EASS MASS EA00 240.0 DONSSE (AASCC aes S0 panSn penser paanii Usese jaseeseee (eese paore (uaino Hoose UNESE 21 (1655
Stapo Vieiing soeddiiis
Denaierg Progress
Medegte Dunc
Conncae sreasine travel bloe sbout recest Nouii. hinliinttiee coltarat coerieeoes attractions. Rospree
Eutrection:
*Mg Istord naneelses Riscueer ind Lie Heart cskibre or meviices
tectes Incents of ot enters, where the scren of slers and mnder of of of cife theise betsern taland, and the and and and iE aband out in Hhe braschaadss atadd of its rice citters. sataral. local, I recertie sur cur foarner to tads Inspiicd pa sanadaticler cae mact lae wis Ialund the 4rtes aolareeeer Atler waghing iormruaccoee catterd vice croetctonot to the UC tand atlens, a reee of Weett's in and caltural n in the Theate deces, se nve, aecrest sorll end, o tnd regntftcaees of the stetory The the, praceices ddane Thiie Eraiitional sos feeding where the the with staniis ratura) Sceuts, morieespiiies (SS experience thex coneectiin the the beastyy
Fioe they, to thes the sate
"Ma barttr Canal after**
froe enctent of
the if, Ansot pereiiis
Levencepes Grtastet, Mell the
Journyy wtw.eeyeeeeeettttteeee My comsorants,
sstrrritabic soelre plannine visit
Nasotil] swanec get pou explire they rtddy and the and Driancas wwwpaiseeelet_ollll_laoe
vixtilee oriin
(WASY)
Muscc
revealed Aur Soreed
prectoux Poriaay Tecald
F PERFORMANCE FOR VARYING EPSILON SCHEDULES
In this section, we present the performance of PUNT with varying epsilon schedules compared to the pruning baseline on the MBPP and IFEval benchmarks. The results are shown in Figures [9 20
and 21
25

---

Published as a conference paper at ICLR 2026
Denotong Progres
Nowggne Papie
limn
telp lig
an introssed of ascicement, EE, EX, Hel cocal lantrcapes, sarcgating the turoucdae hactfic ay into the Stanning keasty Heak Mannng our reclonting lelante, day,, faxt, and Liipkkt stare out nut copttioting, of culture, histany sibrulurr, New Jossory of tWis Inhene you edlise.
past
ruputal the Heart Asselate aig Iscasaeee
oudting Lnaha, took puitec the lcoricol SesceTedeee Todess, stassiyy recisser the inlard's cuttunal teritage. intter the astcced trasteiora) per Torsaees, denctog
doter nerser ansements. Anee
lome
Froe daks, our ady
whion parsaliee desse old kaleissscspe
sanse..
aus. aet, vintted the Matorie tomm Heness where acalenet frcele of mad Las.aa (hne, the speit port, and the zee's pourtariiin SaNISTAte.
aithnate: and mven
"Howeeel: The Feland Anyel Lenury and Traetetsen
In controct ist Aasanbadet Septureres To the royel regel of offll Nesstala, akere Pagllongg the reutlen sordece of the Anaal ansarrky. We chreupt intlectic Hollss Palace, Leorning Secendle cuttural Berttage. In axantage, froat
cocetel free Beath,
Perodisgee
of, deeser marreciatian rich rnturat caliurat hertteer, wats reture enchartine destinetion. sver starnan vlatt Hewals, Miaeaay recamnsse receneesd caploctng tng thead and yourasdy to that pervdias has offer. cleat Solot
visible
(HASK]
May View soviiiillee
hightighting
Seland Paradies: Sissovering the Magio levator
past useat sase lean iostttrrtiig shletaled sxperlences, I've cosking and sore Legttring the turcacics caral raees, set atcmuuriat rae crusstis rasscrices sealt sevng elght telaee, nani, rost, and (Nwl stwin out Nt aemoratee (nrertin eice cettare. ristarr, and eaturaz beettoluniii snllne avriiing I'd caee sondu on this lature conatlie..
eeganal The reatt of Havallas (stansgee
Startins in the caziiaz city. tenctata, uxided tour lcanic tal faaleso. nai nelootii umbbul ialand's cuttarat herltues, relerct beach, watened tradittonst perforance,, dancing myrth the made mnvementa autiled performcrs, Cslant use Pilled asperience, ecotored island's natural sanders, tncluding sturning biumord Hene stste Park the treathtaking
Moud
acenta. apent day bosttl enboinng the and mctching cocertho cuntica verta. the san ast, rdadted the heateris tave of Cenocke Say, store disezveres rEch ahop Rivation otthologs. zempled the ialand's femsur buneees, ine udth delicious lleastte
Islans Lxuary and Sattareee
cnctratt, an PasSOsed DE CORREZ (arry to the lunrrtou Leland of Lonst where ss eestones the opolann wortd ofl toe rnctare repelty. Ti vitttes tae Smarle Slcaus Palace Lities about the Estand's clcs nutional seritage, in evminng, relaved as the nnoutieut revart spwar Lanill,, stere essiued starect aiseer and savon Tiste or istana's rinest outsine.
"Conttiodon: Mauzlen Tarediaces
Assitill dil
viethle e Oriinll
TNASK]
Meder pavealed Doesee
previnus Deviaateeeeeee
26

---

Published as a conference paper at ICLR 2026
Dansking Piegess
Swipp Aep--lp--- 28
Iettrutton:: Conaste travel
veilighttng aitural expertezzee attractore.
At 25lard Naraibet Disowarting the Negis of Revell
soal soto Suee beon ottileraiing atirtainl of oueriokss, as 2'rs sorking inspied sas, centrring Uh: lanondac und resta, exchertingg offering rses
dacceerrsn sturning lenercepca lests, Anong the cige tatency, behs, fext, and Lone!!s cilture, History and natural beasty sellkt anvttine I'd come screaz this abland saradiae.
Hanilan tctarde
Scortiin de the casteak day, Henelsaay us taas guldes toun of the Scords Ees Palasa, widsatte syaboa of the sacaners casturak hurdtage- As as relsoed zs the nwam, witreed traetrtors" sals serinasens, deevling to the nkyhes of the mitt and the snseevare of the setlled periteaeet. The risted with advenure, exprooeed the seraees neturat unsoers, tecruating the shunling miteacee mes ytate sark ai the tveertnactig snlles
Nudi: Tropicat Lskend Peratise
I: Cakand
Nasl, on seceroed scmnt derry the lusarisus Exlend Lacaiii, ubere restiveed the restien honec the mest feallyy vietrre the Scorde leceas foloce, untch inoeceaca the Sstard's rish catturak hordtage. I1 the crontngs, se retorce et the aeautirul resort of Feysl Lened's unere actouet sourest sioser and naver tecte of the tslandee lisect eidedes.
toall cimes excrantie
sewill's
visibie Orginal
[SAB6] Vited
|revealed
pravious, hewway forual
27

---

Published as a conference paper at ICLR 2026
Denwrang Progress
Herigatl Segec
Inotruction: tatella aold chipo 42 of her friends in sprdz, and then ahe wole haat many chips in Aortl end Nay?
mary clise die Reballe vell atcogecher
Responce: (AAC wWA.////COOOOOOO ( (MSS) MASE ATA ( (MSCC MM (MU (A (AS (MAS))
peeso po /eeer 2/ (eeoe) tnee wfw.r (eare (nene)) /e Ma twe /oune Esece faocer
(16600)
(M6,0)) (A50)))
Framu Domose 2 Dnese Ta Pe 2 ot 2 T 31 su Deesse Te Encose
Lnaene (PHAA (2 (ADK) (1A5C)) (ASA) (AA0)) (AN0) (4S4C1/444//100000 (4450)445)))
NASK) (MA- (Weass)
11 (IGA)) (100K) (MA (NSSEE (M455)))) (M455)) (M (MSK)) (KSUJ (SS60 (ASR)) (NNSE wond (AARE)
Liveea MASK (RASS)) (MMEX))) (SASC (UASE) (1A00) (MMU)UMMUUM (MMSE) (NSSC)
vinille Drlp
[NASK]
Inroot
ravvelled
prwvious
Pavio aty ReAnne
Devilding Progres
Instruction: tatelie este cltoe 46 Ordende in lonts, and then cstd NALE mary <3ipe de Pay. Hew mary clter etc letalce esll altogetha" in Aortl ane hay)
Reaponse:
Entemiid the total
clles tetedii sold in Aeril Nos. to low troo stonsctteartatt Celoudete thc numbor co Anteciin wastor
sold April the thy Uhd Find number nanderctin
21 serowshrat Calinalate secif atencionn inceeeei
offeade wwt.://nnnmn/n///cooooooo 2twcc////22///e12220000 Eelcuieter number
chi ciler tre sold Ray, Met.e acld half cétic Aprit. faril, culas,
May2 (4)
"aa t the -ffteee 129994
sold Aprit
find fied ctips A2r sole
veet Hebo
cee rECrO deot>
L00T wtt.:ottt..oooott..oo. wtt.ohoootto.ttoo.... wott www.tootoooooooooooooo.oo 20mh ko0t
vizible Orignn
[MASK]]
Madell
revielled lad levated
Aroyvauu Wnevowyy
28

---

Published as a conference paper at ICLR 2026
Dansiing Prognos
inssed tnforcanson
and nooe
teeetory nemter cltmn Netetiee fert.. diis
Slar tatsabate
clias Melabaa
Netettee
nore eldne Hey ahe ferth. Phonefere, soce1200y cctar clien stejecues Nevticatnnz1n theienly
stasadee soia -2000 jens Aprll Aayeian Secetz) And the number cidpe ssdd Sa April the eusson cadps sadd in shar22911 42200) engo (00003
(nan the ceeus neber ex sitas saraate con--asl-- -oar--aai--aatt 300 -000--0--------0-----0------- -as---a------sssnn -00---00---0000000000000-------- -001--000000000111----0------- 4500 -ot.- dedt- corree ino ceet cor
nay V/toowe www..umeeeeessseoeoooo. saun coutocciaa souu antee (
scur
eot veaty reshs coo wet..
woth aBSts coot (EST (0th
con wetoe c200 (200-122)> (220)-02000001110000 (200:0000000000000000000 (20)) (2100050--0000000000000000000000 htt.://0oo0o0000000000000 (2D1)-00))) (2000-000--------00 (000-200X-0 (800 (200 (E30 iggro ces ce9t)
coot aetto (eSt)-cotyo---->
200 200---00--111 (EStty -0T (ott--ett---77777 (ttt0/000000-001111 (2DT)>e57T>----- 200h 20T (BST) (207 ( (00) obtt centtreeooooe bot (597
wtt.e//eett.... -ESti 20T. wtt.-/ooo.ooootooo wtt. cot 20t. HEST 007 20T HDST (0Tty 1o0T wttth - -5ST 1Dth EDft cot woth voo -ont.
-0---00--0000----- -ot---------------- 1o0t 1o0t--000----- -00---0000000000000000000000 -01--00000000000 cERT wo0.--0001--0000 (000)-220) cottt Hpp coot -oot coo -a0t-
vis Si
Sep O- Wooiing irate
Denelinn Progress
Navigato skpp 18
dracruction: hetalle soac cadga to 45 non redenda in Aprik, and than ats acid haar as muyy cldea may. noo mery cidpa latalio 2013 abcogether aprds and tay?
Raepenes:
seterease total remter cLibs Notalla seze in Raril and Hay, need to +ilLow these stessi
tey bey the wetven ciips tatat woti in deeit
Calsolate membrr
Ratalia Nhy bood the intornation.
And the vantera froe April end Hay the tetad.
Step cosatiiy
intalia cole aprit..
secaste
Ceocelate cilas iotalla aprst.
de0th EDTO 120 a 2000---00-0000 ceot>
vbott eettvv
89TV 1ot voSi
201 wott
eE0t wor -bot aot.
EDT bot. -0000-00--00000 E2T 220 EDT 200
ceorscade eve wve uedit oedro--ddr- ownts cecto-ooooeee ceeee coen
oed vorr dedi coett ventrno canee
retrr cedi beegg cete veteeeeeeeeeeeeeeeeee eedti rests eoot soces cee bectrrettto
vesto rep ooee ser seeee cote redit reat rediiveed vedits resiit
visible
(HASK
newealee Jsi Sonaizzd
Prefiauty Forezzz
29

---

Published as a conference paper at ICLR 2026
Denaidig
loentdlyy the samser clips Henalda anka April.
namsn saraute
Iniornattor.
Stes lovertlty the sumse of cltpe hanalts sald tn Aoet).
nanaita cole citpe to worti.
Calculate numar chipe latadia Therefore, cxtralates ratt any citoe Hay
Stea tee the eunsers ene April and for to +56d the total.
Ass the runter citis sato tn seeld to the nunser citor sonc Nays
10Pa1 extpo..
visible Ofiime
[MASK] Mobel revealed Tockle
previous Nrolowh Reeteet
30

---

Published as a conference paper at ICLR 2026
Donoking
Instruction: kette furctton find the oath for metrte comnt3l1
pontition
costr313.
DOAN DAASS (WW.://ASSSSSSSSAAAAASA TOAAS www.eeawwwooooooooo...o. CANSSI (NAS JOMAN) (eeve (ere pmove
(AC/////EE/////000/0000 (152 (MNSE)
(MA ( (RAS4) (MAK (EASN) (1M6)) (MAGG// (IAC 20002200000 LIML (OAC
(MASEJENEAA (IAIO))) LMM 1ASC
Disse NANSe 1 mest RESU M/At
most
pnene nanee renner /nenne AAree (NENNE moner (em Lemonc Heen (enn feeter ( menor
precce (anee (ineer) (euns MAmeu (oens) (1 (emsss Henns (1S00)) (nenar Heare Le se teaeee be (eeeor)
vistblle Orignel
Madred
revealee
Lut Deveted
Predocsty Revaled
Denciing Progres:
find path Cry 10, for the atvon estris ove11111 position be,
Reapotoe:
11ne parn reazs it, (o, for the giver extrix, zue ase appante pragrametion Toee StAry
afe Arep copeotents rxpresents the ainioss wintmaa ceech starting pusiitin Merelts Pytioa Juscilin tw ao "ppttee]
restliieetl
lenorser)
c033
(330) the 110, witnd Flort del cestenLotetas Ziodaa Zercvos ferllents Intrount cost strectiore peerthia))
directionn Ite:all 220183, (22010) ( (112210
(185))0022453 FELL the prem)) 1927 7or (N0 Iaage 3eCCrma9)
Vor (otge(cll)):oo (EEE> Spt/00) 7
Jhe st, dtracttoon)scin
30r de, de aetinn
aNd CwS. www.eeeeettoottteeeeeoooooo Ho wnndete TeReeL 68233
(P122) (111--t 201 recurn ostRite: SUPCQUOR 0500F1 Hid 9833 on 1991 seage, cus 14190d J2001 (208)) MST Stte,, (208)) [2H42) will (P0 [25,/20)) 21 2109 of Sf (an) wthetee inleled
deghien Peat win s[ose] least teriite Leae? above [209 sterediall (amit) eepnosunts wieieme and to wwus Exai (i dow
Liter, lec usd.. lnce Stlan (t.d 1d 3t [
ital Fil
1e mirdmum unt
de. 10)
provious
31

---

Published as a conference paper at ICLR 2026
Dwenkinn Progwer
Raiipla Sept
Fasction
cost pats reach (a,, the slvec MECrLR OSSTEDTE DOSSEUM
glven exteite, can droantc progranndng. arnay oell cortie pastetint...) representiol (200) neach neece postitox 15- (line Pystin furetdoe
mindrun 2oet path reach (r,
ineatty 0220022
gython sin_costtoaet, 930
row Lo (onk)
cats (015
(3852 win /intee, catst 20200001 122822 incels ressetier)
(365) aide Diserstanns 1 dirccsions
00, 00, (e. 235
G PELE the anrin wito wick abtima [XR0) For In rarge(rous))
for 5 reageCralets
(4925183) (1oae(1tnl))
4. Le eitein
(.6x 1/122 m2002201 disend te aoreet
atrlice mintcree
(253)
autelte
18
visible Ofeen
[MASK)]
Meseet
revealed
but Covected
provious Fremecyy Tovebot
Denwidng Pregraos
Evseuetioe:: artto functice to flod the sirdeus cast poth to reech Ohy from 66, 6) for the glion cost setric coatl1]] ard roscttion Ds,
Responee:
eine the ststmus cost para reach (er, n3 erse 43 mon the glves ever extrtty, cas oysonic pengrnantigg The toea sutto aSTrER
eith wid/ $1111 reprrsents windowe mecch sovl fiew tike canreat pesiitio Cle 1.. Heve Pytkie Forithon wekleee Inlss
Ditten
der ein coscccstt n21
lon(ccat)
colz Sen(coeejo))
[19 [1] (n0) (oppe(colo))
aecelnier (ran
directione
ainpriis1
ailiiiie (astae223)
Dorele costumstrrs
Hk 4), 16, 33,
est rey, rae 7.
onincinin coot mutrdo. end rou. and will) Outout:
9
Trle furction Excetatiinn tawe autiii autisto inderto LLEK coste and Die Bi pacciate direcstono, 25 than Lteratec through the CNE marrds and BEA
satetto, calnctttee tnd the ateemus nset to reach wech cxll (1, 1) by daseraselig teeeeeeeeeeeeee the soe Thd atttreeet extiie Dihy doan, right undented sintee cose ti myy instie. instr. con--oeroes cust costrrosotooooe co- www.osowwooooooooooooooooo wow.--o---------------- sos0i wow.--aaiiieeo poarr--aat
itamiiyy 15. raturne the stxtau roct raams (8, ires 06, 42 MAcce_te S-cuE Lo oot souurs Co0
1ocal
Rapai. Viewing data

---

Published as a conference paper at ICLR 2026
Denceiig Progress
(e,
swtt11l sesiitin De,
Reapatset
the stedms cent sets to reech (n, (b, 61 for slecn netris, onn use onnonde pragrameing, bitle wetrix uhare eacs wirg-1153 regresante sdeinae cast react that trom currest pocitiin (2, 5)- mare Pytter sanction to schleve til
f),
for de renententate #in157231 (lser(larr)
for dry directionss
ts resaft) weat)]
e rows sate) mtnertpe) stats1t5t:
vdrelae50 winertte (9st21213)
bxangue isveet CBSt, natria
[6, A1
(4
prantiezn costten Lnttric, onl,re,, (x2)) Ausoutt
fanction ardodeldaca the ein acords suttaday valcca and the thres peasdeis directiana, then storstes through the cost ratrix and cach strection, extectatting the eistaus cost reach ext1 15, 3) constoening patte the acfacere extle tun, den, ane right) and adetig estt. In the exist. kleallr, It returns the nixiem oot to reach Ci a3 frw (e, shudeeot.is>>
300 Arnin seecsssee saft
Accuracies vs Epsilon (t) Benchmarks
Epullon
Spallon
tysiton (1)
Epaiten
Figure 15: Performance vs for LLADA at temperature 0.1
33

---

Published as a conference paper at ICLR 2026
Liada PUNT Accuracies vs Epslion (e) across Benchmarks
IFEVAL
MBPP
54
Epaiion
Epation 6SMSK
HUMANEVAL
mi1
Accurscy
1pollon
Epolion
Figure 16: Performance vs e for LLADD at temperature 0.7
Liada PUNT Accuracies va Epsilon (e) across Benchmarkc
IrEVAL
MBPP
rel WOLL PONLLCE
Epniten HUMANEVAL
Epatten GSMBK
oo00l
spillon
Figure 17: Performance for DREAM at temperature 0.1
34

---

Published as a conference paper at ICLR 2026
Liade PUNT Accuracies vs Epsiion (e) across Benchmarks
IFEVAL
HSPP
Epstion (3) HUNANEVAL
Epalion CSMBK
ACcuuest
Epation
Egation
Figure 18: Performance vs for DREAM at temperature 0.7
lada | mopp tema 0.1: pass NFE reliow: 0.25-0.5-444 5-0.75-1.0
laaaa mbpp temp 0.7: pass VF rellos 25-0.5-0.75--..0
0.400
adspt vesum! pruring
0.30
adsptivesutt sunning
0.125
0.375
3,250
0,300
0.275
700
(a) LLADA, T = 0.1
(b) LLADA, T = 0.7
ratlos: 02
ratios: 0.25- 5-0. 5-0.0-0.75-10
adapbi-ppurt
adepeveeunt
0.45
0.400
0.375
0.50
0.45
0.300
0.255
0.250
75
102
125
150
175
200
225
100
200
300
400
500
MFE_300
IFE_3v0
(c) DREAM, T 0.1
DREAM, T 0.7
Figure 19: MBPP @ass@ I vs NFE for varying epsilon schedules. PUNT with varying epsilon schedule (blue) pruning baseline (orange).
35

---

Published as a conference paper at ICLR 2026
Rods |reval temp. 0.1 lisst level joose ace vs NFE
Madda ftevall temp 0.1 prompt level loose acc vs MFE
Tatlow: 0.25-0.5-0.
relios: 0.25-0.5-6.75-1
adsptivegutt stuning
0.50
adaptivesurt sruning
0.88
0.58
2.64
0.54
0.6
0.52
09/
0.58
(a) LLADA Instance-level
(b) LLADA Prompt-level
dream Iteval temp 0.1 inst Jevel loose acc NFE
dream Ifeval temp 0.1 prompt level loose acc vs NFE
rellos: 0.25-055-0 75-I.0
retlos: 0.25-0 235-0.--077--10
adapUixtpurt peuning
0.050
adepr.ve
0.74
pruning
0.70
0.500
Teu Tw-20 2000336601144
0.58
00:55
Km
0.350
0.6
0.323
0.6
300
400
500
360
200
307
400
500
000
100
NFE_3x0
NFE
(c) DREAM Instance-level
(d) DREAM Prompt-level
Figure 20: IFEval Loose Accuracy vs NFE at temperature 0. for varying epsilon schedules.
36

---

Published as a conference paper at ICLR 2026
dream | feevll temp 0.7: Inst level loase, acc vs NFE
cream Iteal |
temp 0.7 prompt level loose, acc vs NFE raliow: 0.25-0.5-0.7511..
reliow: 1-2.5.5-0.75-3.0
0.6
adaptivepurt
0.72
pruning
0.70
0.58
0.68
0.6
0.54
0.22
0.64
pruning
adaptvvppnt
(a) Instance-level Loose
(b) Prompt-level Loose
dream | Ifeaal temp 0.7 Inst level strict, aicc NFE
dream feeaal temp 0.7 anompt level strict NFE
retios: 0.25-0.550 3-0. 75-1.0
rellos: 0.25-0.5-0.75 3-0.75-1.5
ataptiveriir. pruning
0.58
adapiivepurt paining
0.50
0.54
0.66
032
2.64
0:50
0.48
100
200
302
400
500
000
000
900
300
400
500
900
700
NFE_009
(c) Instance-level Striet
(d) Prompt-level Striet
Figure 21: DREAM IFEval Accuracy vs NFE at temperature 0.7 for varying epsilon schedules.
37


Reviews:
[{"review_id": "0YpY7J6DKX", "reviewer": "Reviewer_Z7Ub", "summary": "This paper tackles the conflict between speed and quality in Masked Diffusion Models (MDMs). The authors propose **PUNT**, a **training-free sampler** that enables efficient parallel decoding by **explicitly testing for token dependencies**. PUNT identifies and prunes tokens that are not conditionally independent, using a recursive $O(\\log|M|)$ **algorithm**. Experiments show PUNT achieves a **state-of-the-art accuracy-compute trade-off**, especially on long-form generation tasks.", "questions": "I wonder if similar results could be explored in image generation pipelines like MaskGIT", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.  **Novel and Effective Algorithm:** PUNT is an **elegant and efficient** $O(\\log|M|)$ **solution** to a well-defined problem (confidence vs. independence in parallel sampling). The use of explicit independence testing is a strong contribution.\n\n2.  **Strong Empirical Results:** The method **clearly outperforms strong baselines** on the accuracy-compute Pareto frontier for relevant long-sequence benchmarks (IFEval, MT-Bench).\n\n3.  **Strong Theoretical Justification:** The method is well-grounded with **strong theoretical justification**, particularly by connecting its \"Independence Stability\" assumption to the properties of Transformer attention. The discovery of an **emergent coarse-to-fine generation strategy** is also a valuable insight.", "weaknesses": "1.  **Limited Scope:** The method's advantages are **diminished on short-sequence tasks** (GSM8K, MBPP), where its $O(\\log|M|)$ NFE-per-step overhead is less efficient than simpler samplers.\n\n2.  **Critical Hyperparameter Sensitivity:** The algorithm's effectiveness hinges entirely on the **KL divergence threshold** $\\epsilon$, which is not a simple-to-tune parameter but a **fundamental trade-off between speed and quality**. For a task with highly dependent tokens (e.g., code generation), most tokens will have a high $D_{KL}$. The user is forced into an impossible choice:\n    - Set $\\epsilon$ **low** for *quality*: This respects the dependencies, but will prune nearly all tokens, **collapsing the sampler's speed** to be sequential.\n    - Set $\\epsilon$ **high** for *speed*: This ignores the dependencies to unmask more tokens, but will **destroy generation quality** by violating the independence assumption.\n    This makes $\\epsilon$ a critical, task-specific parameter that requires a costly sweep for any new model or domain.\n\nI also think that this parameter is not well ablated in the paper\n\n3.  **Unclear Baseline:** The abstract's claim of outperforming \"sequential generation\" is confusing, as the `TOPK` baseline in the plots performs poorly, suggesting it's **not a true one-token-at-a-time sequential baseline**.\n\n4.  **Misleading Efficiency Metric:** The paper presents plots against **\"Denoising steps\"** alongside **NFE**. This \"step\" metric is misleading, as a single PUNT step is $O(\\log|M|)$ more expensive than a baseline step. **NFE is the only meaningful measure of compute**, and the focus on \"steps\" can obscure the true cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtdyNinSc3", "reviewer": "Reviewer_RJ3J", "summary": "This paper proposes PUNT, a new model-agnostic sampler applied in Masked diffusion models. By making the use of a proposed Contextual Independence Assumption and the corresponding recursion algorithm,  PUNT enables to select the postions from masked positions set M that relatively independent of all |-M| umasked positions for decoding, in terms of a time complexity O(log |M|). By taking contextual independence into consideration, PUNT can efficiently decode multiple tokens in parallel, while keeps its decoding accuracy.", "questions": "As shown in Figure 1, why to take the tokens  that are always dependent on anchors (such as ‘egg’) among all tests, rather than tokens that rely on anchors at least once among all tests (such as ‘mince’), as the final ‘rejected’?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper designing PUNT by skillfully utilizing a contextual independence assumption and a recursion algorithm, merging the tests of subcases of one recursion level into one evaluation, which is training-free and efficient. \nThere is a point mentioned in section 3.4 that  “assumption3.3 is a direct consequence of the Transformer architecture’s attention mechansim. If the attention from position i to position j is zero, then position j has no direct influence on the representation at position i.” This point of view shows the relationship between sparse attention and the testing of contextual independence. Perhaps this can serve as an inspiration to utilize independent testing to identify sparse locations.", "weaknesses": "I noticed that in the divide-and-conquer process, some tokens are considered as ‘dependent on the anchor’  in a previous evaluation, but soon be considered as ‘independent’ in the next evaluation (such as the token ‘mince’ in Figure 1,left). It seems that only tokens tested to be dependent on the anchors of all evaluations are considered as ‘dependent’ and absent from the current generation, which means tokens in parallel may also be dependent on each other. It seems that PUNT is less capable of separating the dependency on tokens than DILATED when NFE < 100 (Figure 3 left). \n\nAs shown Figure 4, it seems that PUNT fails to exceed DILATED at NFE 400 on MT-Bench, but  there is a lack of sufficient explanation for this phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ry9H73jbon", "reviewer": "Reviewer_vZYQ", "summary": "This paper presents PUNT, a model-agnostic sampler that reconciles the trade-off between speed and quality in MDMs by using approximate conditional independence testing to identify and resolve token dependencies. PUNT delivers a superior trade-off between accuracy and compute, especially for longer sequences, without requiring brittle hyperparameter tuning.", "questions": "How was the hyperparameter $\\epsilon$ selected? How is the method sensitive to the hyperparameter?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- PUNT is efficient, training-free and dynamically adapts to sequence-specific dependencies.\n- PUNT induces an emergent hierarchical generation strategy, suggesting a planning-like process that contributes to its strong performance.", "weaknesses": "- The \"independence stability\" assumption, which PUNT relies on, is a strong approximation but it is not proven.\n- The claim of mitigating \"brittle hyperparameter tuning\" is insufficiently supported as no sensitivity analysis is provided for $\\epsilon$.\n- The method underperforms on short-answer tasks where the computational overhead of multiple forward passes per step is not amortized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQyWn4E2HI", "reviewer": "Reviewer_cxUA", "summary": "This paper introduces PUNT (Parallel Unmasking with Non-influence Tests), a training-free sampling algorithm for Masked Diffusion Models (MDMs) that enables efficient parallel token generation while maintaining output quality. The key idea is to select subsets of masked tokens that are approximately conditionally independent, thereby allowing them to be decoded simultaneously without introducing dependency errors. PUNT implements a divide-and-conquer procedure that identifies such token sets using O(log |M|) model evaluations per iteration, relying on a contextual independence test based on KL divergence between conditional distributions.", "questions": "1. The paper proposes $\\epsilon$ as a fixed hyperparameter. Have the authors explored using a dynamic $\\epsilon$? For instance, a schedule that starts with a small, strict $\\epsilon$ (to establish the high-level structure, and then increases $\\epsilon$ in later steps to be more aggressive and rapidly fill in local details? \n2. How sensitive is PUNT’s performance to the confidence ordering strategy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is built on a very clear and reasonable motivation. It correctly identifies the fundamental conflict in parallel MDM sampling: the desire to unmask high-confidence tokens versus the necessity of conditional independence between simultaneously updated tokens. PUNT's approach of directly testing for and mitigating this \"inter-token interference\" is a sound and principled alternative to purely confidence-based heuristics.\n2. PUNT demonstrates a clear Pareto improvement on long-sequence and instruction-following tasks like IFEval. The observation of an emergent hierarchical generation strategy (Fig. 2) is a significant and interesting finding.\n3. The connection between independence stability and attention sparsity offers an intuitive, architecture-aware justification of the method’s assumptions.", "weaknesses": "1. Experimental validation is limited. The current experiments mainly demonstrate advantages on two benchmarks (IFEval and MT-Bench), while the improvement on MT-Bench over the Dilated Sampler is marginal. On short-answer tasks (§4.2), PUNT shows no clear advantage—likely due to higher per-step complexity. The evaluation lacks additional baselines such as APD (Adaptive Parallel Decoding, arXiv:2506.00413) or other few-step DLM planners that could better contextualize PUNT’s trade-offs.\n2. Definition 3.2 relies on a single, fixed sequential order ($X^i$ given $X_{<i}$). The algorithm implements this order by sorting all masked tokens $M$ based on their initial confidence (at the start of the step). This static, confidence-based ordering may be suboptimal, as token confidences are likely to change once other tokens are (tentatively) revealed.\n3. The actual implementation (Sec 3.3, Alg. 1) appears to use a much stricter test than the sequential one defined in Definition 3.2. It performs a batched test, checking all tokens in the \"test\" set $S_1$ for dependence on the entire \"anchor\" set $S_0$. This is stricter than Eq. 2, which only tests a token $r_i$ against preceding tokens $R_{<i}$(31). Does this simplification, made for the sake of parallel computation, lead to over-pruning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)