Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
Authors: Satoshi Hayakawa
Yuhta Takida
Masaaki Imaizumi
Hiromi Wakaki
Yuki Mitsufuji
Venue: Accepted by TMLR ()

Abstract: Masked diffusion models have shown promising performance in generating high-quality samples in a wide range of domains, but accelerating their sampling process remains relatively underexplored. To investigate efficient samplers for masked diffusion, this paper theoretically analyzes the MaskGIT sampler for image modeling, revealing its implicit temperature sampling mechanism. Through this analysis, we show that MaskGIT is asymptotically equivalent to a choose-then-sample (CTS) formulation, instantiated as the “moment sampler,” which explicitly separates index selection from token sampling. This CTS reformulation is essential: it yields unbiased token sampling and exposes an algorithmic design space for index selection, both of which are inaccessible in MaskGIT’s original formulation. Regarding token sampling, we reveal that MaskGIT implicitly adopts a low-temperature sampler, which explains why MaskGIT often degrades with more sampling steps. The CTS reformulation of MaskGIT allows to fix the temperature sampling to ensure unbiasedness. We also improve the index selection in CTS through two key innovations: a partial caching technique for transformers that approximates longer sampling trajectories without proportional computational cost, and a hybrid approach formalizing the exploration-exploitation trade-off in adaptive unmasking. Experiments in image and text domains demonstrate our theory as well as the efficiency of our proposed methods, advancing both theoretical understanding and practical implementation of masked diffusion samplers.

Published in Transactions on Machine Learning Research (04/2026)
Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion
Satoshi Hayakawa Sony Group Corporation
hayskaun@mist..ttttoky...oo
Yuhta Takida Sony AI
yuta.takidalsony.ccco com
Masaaki Imaizumi
imaikaum/@lgeeco...tooog.oooo
The University of Tokyo
Hiromi Wakaki
hiromi, whromii.wkkkki@sony.coom
Sony Group Corporation
Yuki Mitsufuji
Sony AI, Sony Group Corporation
Reviewed on OpenReview: https: optp:/vveeprrveee...mmmmm net/ forum? id= mk WW68121g
Abstract
Masked diffusion models have shown promising performance in generating high-quality sam- ples in wide range of domains, but accelerating their sampling process remains relatively underexplored. To investigate efficient samplers for masked diffusion, this paper theoreticall analyzes the MaskGIT sampler for image modeling, revealing its implicit temperature sam- pling mechanism. Through this analysis, we show that MaskGIT is asymptotically equiv- alent to a tasssetthhn-samappe (CTS) formulation, instantiated as the "moment sampler," " which explicitly separates index selection from token sampling. This CTS reformulation essential: it yields unbiased token sampling and exposes an algorithmic design space for index selection, both of which are inaccessible in MaskGIT's original formulation. Regard- ing token sampling, we reveal that MaskGIT implicitly adopts low-temperature sampler, which explains why MaskGIT often degrades with more sampling steps. The CTS reformu- lation of MaskGIT allows us to fix the temperal ure sampling to ensure unbiasedness. We also improve the index selection CTS through two key innovations: a partial caching tech- nique for transformers that approximates longer sampling trajectorics without proportional computational cost, and hybrid approach formalizing the inpoaaation-xpliitattio trade- off in adaptive unmasking. Experiments in image and text domains demonstrate our theory as well as the efficiency of our proposed methods, advancing both theoretical understanding and practical implementation of masked diffusion samplers.
1 Introduction
Generative models have witnessed remarkable progress in recent years, with diffusion models SShh-DDickstein et al. 2015) emerging as a dominant paradigm across various domains including images (Ho et al. |2020 Dhariwal & Nichol 2021 audio [Kng et al. 2021) and video (Ho et al. 2022). While continuous diffusion
models have garnered significant attention, discrete diffusion models (Austin et al. 2021 Campbell et al.
2022 Gu al. 2022 Lou et al. 2024) offer compelling advantages for inherently discrete data, such
tokenized representations in all the above domains.

---

Published in Transactions on Machine Learning Research (04/2026)
Among discrete diffusion approaches, masked diffusion models (Sahoo et al. 2024 Shi et al. 2024 Ou
et al. 2025) have demonstrated exceptional performance, particularly in generating high-quality samples in language domains. However, their sampling process remains computationally intensive, requiring hundreds of function evaluations. Recent efforts to accelerate sampling in discrete/maste /maked diffusion models have explored various directions, including optimized scheduling (Park et al. 2025), distillation (Deschenaux &
Gulcchre 2025 Hayakawa et al., 2025; Zhu et al. 2025), and adaptive token selection
(Kim et al. 2025;
Ben-Hamu et al. 2025
One of the most notable masked diffusion samplers is given by MaskGIT (Chang et al. 2022 which has shown that impressive image generation can be achieved with as few as 8-12 sampling steps, significantly fewer than the hundreds typically required by discrete diffusion models. This efficiency has inspired subsequent
work in image (Lezama et al. 2023 Besnier et al., 2025), audio (Garcia et al.) 2023 Comunità et al.) 2024). and language domains (Zheng et al. 2024 Nie et al. 2025). However, the method is heuristic and not well understood in theory, and it often shows degraded performance when increasing sampling steps in image generation (e.g., Gat et. al. 2024: this paper's Figure 3 or severely reduced diversity in language generation (Figure 5]. Moreover, MaskGIT's sampling process entangles its token sampling and index selection, thereby limiting its compatibility with various existing index-sclection criteria (Xiang et al. 2023 Zheng ct al. 2024
Besnier et al. 2025 Kim et al. 2025). In summary, although the MaskGIT sampler has a promising performance in image generation, there are two problems to solve in the MaskGIT sampler, especially when applying it to language and other domains: (i) reduced diversity in many-step and/or low-temperature
sampling, and (ii) limited flexibility of the sampler to incorporate other index-selection criteria.
In this paper, we address these problems through theoretical analysis of MaskGIT. We show that MaskGIT is asymptotically equivalent to a choose-then-sample (CTS) formulation (instantiated as "moment sampler"), which explicitly separates index selection from token sampling. This reformulation allows us to address problems (i) and (ii) independently: for (i), we reveal that MaskGIT implicitly performs temperature- based token sampling, which accounts for the diversity degradation, and the CTS formulation enables unbiasod sampling by simply removing the temperature; for (ii), the decoupled structure naturally accommodates various index-selection criteria that were incompatible with MaskGIT's entangled process (see Figure D).
For practical improvement of masked diffusion samplers concerning point (ii), we then introduce two key techniques specifically tailored for choose-then-sameg methods. First, we propose partial caching technique for transformer- based models that effectively approximates the sampling trajectories with more steps without proportionally increasing computational cost, unlike the MaskGIT sampler that requires recomputation of all positions at each step. Second, we formalize the ipploration-ccplotattin trade-off in adaptive unmasking of masked diffusson sampling, leading to a hybrid approach that combines the strengths of exploitation- methods (Kim et al. 2025 Ben-Hamu et al. 2025) with exploration-oriented techniques such as Halton scheduling (Besnier et al. 2025).
Our contributions are illustrated in Figure 1 They can be summarized as follows:
In Section 3. show that MaskGIT implicitly performs temperature sampling and is asymptotically equivalent to a CTS formulation instantiated as the moment sampler (Theorem 2 providing a precise decompasition into index selection and token sampling, The CTS reformulation yields an insight that MaskGIT's degradation at large step counts can be attributed to its implicit temperature sampling; removing temperature leads to unbiased sampling in the many-step limit (Proposition 3)
In Section 4 leveraging the CTS formulation, we introduce two algorithmic improvements in in- dex selection that are not expressible in MaskGIT's original sample-then-chooss form. One is a partial caching technique for transformer- based models that approximates more steps without pro- portionally increasing computational cost. The other is through our formalization of the exploration- exploitation tradc-off in masked diffusion sampling, developing hybrid approach that balances these competing objectives.
In Scction 5 experiments on image and language generation validate the theoretical equivalence, isolate the dominant role of temperature, and demonstrate improved officiency-quality trade-offs for CTS- samplers.
2

---

Published in Transactions on Machine Learning Research (04/2026)
MaskGIT sampler
entangled
choose by
sample
sample confidence
marginal
distributions
(param:
vocabulary
i Main result: Asymptotic equivalence (Theorem 2) when 1 1/a
CTS reformulation (Moment sampler) disentangied
choose by moments
sample w/ temperature
position
(param: at)
(param:
Issues of MaskGIT sampler:
(i) Reduced diversity
Limited flexibility
(ii) Design space for index selection
(i) Revealing implicit temperature sampling
Partial caching (Section 4.1)
Debiasing token sampling (v 1) (Proposition 3, Section 3.2)
Solutions
Exploration-exxpootttion (Proposition 4, Section 4.2)
Explaining MaskGIT's degradation (Section 3.3)
Figure 1: Overview of our contributions. We show that the MaskGIT sampler is asymptotically equivalent choose-then-sample (CTS) formulation (Theorem 2). instantiated as the moment sampler, which decouples index selection and token sampling. This CTS reformulation yields an unbiased sampler when temperature is removed (Proposition 3) and provides a principled explanation for MaskGIT's degradation at large step counts (Section 3). Moreover, it opens an algorithmic design space for index selection that enables partial caching and hybrid bi-criteria mothods (Section
Our work not only advances the theoretical understanding of masked diffusion samplers but also provides practical techniques to enhance their efficiency, toward better modeling for discrete tokens.
2 Preliminaries
Let Qdata be the data distribution over the product space X' SD, where S finite set of tokens and D is the dimensionality /legthh of the data. Each state X can be represented
sequence of tokens x (2:) with E S for each i E [D), where [D] = (I..... D}. For any set of indices T C [D], let write Ø1 = (zi)ier- For I,J c [D], let (1)())))=)) slightly abuse the notation to represent the case T= {i} to write () Finally, arglopkirr {us) is the top-k indices of sequence (a,)er rearranged descending order.
P(yr = x, yJ = æy) with Adata- Let us also
2.1 Masked diffusion models
In masked diffusion models, we augment the vocabulary with a special mask token M & S. The forward process (t) E (S (M)) with SD M x00) = x ~ Qdata gradually replaces tokens with mask tokens according to predefined schedule, eventually reaching æ(1) M). For generation,
simulate its
backward process starting from fully masked sequence. While the time-dependent formulation is a natural consequence of general discrete diffusion modeling, recent studies (Ou et al. 2025 Zheng al. 2025 pointed out that, under commonly used forward masking models, the conditional probability P(x(0) does not depend on L. Thus, for simplicity, we assume that a given pretrained model is time-independent and tries to approximate the conditional distribution of qr) (1)_(()(æ))) for I, C [D], and æ ~ gdata Then, our unmasking process given by / Jo C C Jn [D] to iteratively sample ÆJA.J approximately from J.A.2 1/2e (+(x)_,,) for & = 1,.... n. For efficient sampling, we prefer using smaller n, the number of unmasking steps, which requires unmasking of multiple token positions at each step.
xx)) y)
3

---

Published in Transactions on Machine Learning Research (04/2026)
Since the probability distributions over S with large |S] or [Z] are intractable, directly modeling q/l for T containing multiple token positions is inefficient. A common workaround is the product modeling Pny == Ilke (())) which only approximates one-token marginals as PalJ R qij.. In this paper, we investigate efficient samplers for such product models without any retraining or additional components.
2.2 MaskGIT sampler
We review MaskGIT, which is a pioneering method of post-hoc efficient sampling for masked diffusion [Chang et al. 2022). To simplify the analysis, let us just consider one step of unmasking. Suppose we are given N probability distributions PI...., PN (corresponding to PiJ. for & J the previous section, and so N D -J]). We would like to unmask token positions from [N]. Given the temperature parameter a >> 0, the MaskGIT sampler (Chang et al. 2022) is given follows:
(MG1) Independently sample x; pi and standard Gumbel noise & for each i E [N].
(MG2) Choose (i....., is) = & (log (1r,) + a
(MG3) Return the indices i,,.... is and samples Th....,;
We formalize this sampling scheme in Algorithm ! in the appendix. This sampler works surprisingly well in image modeling when the number of sampling steps is small (Chang et al.) 2022 Li et al. (023). However, it also observed that its performance degrades as we increase the number of steps (Gat et al.) 2024 Ren et al.] 20255] see also Figure B). In the following section, we investigate this algorithm and derive an asymptotically equivalent sampler that is more flexible and interpretable; the analysis also partially explains the above performance decay.
3 Choose-thene trooos-theen-smple reformulation of MaskGIT sampler
We analyze the MaskGIT sampler in this section. Let us start from general result on the Gumbel-top-k sampling, which is used in (MG2) of the MaskGIT sampler. As a top-k generalization of the Gumbel-max trick [Maddison et al. 2014), the following result is known:
Proposition 1 (Gumbel-top-k trick, Kool et al. 2019 Eq. 18). Suppose we are given p1...., E R and ii.dd standard Gumbel noise f1,.... EN. Let (ii,.... , [N] with é S k, we have (i it where I=-1== (i......
(
= argtop {1z: + E.}- Then, for distinct indices === - = i--1) exp(ute)/ exp(p..
As mentioned in the original paper, it is mathematically equivalent to the size-k sampling without replace- ment with logits μ.. While it also reveals the value of P(i = i ig), particularly interested in the conditional form in the proposition.
Let us consider the case = []] for simplicity. Let (ii,..., a = k (1,;)((;;) + a& in the MaskGIT sampler (Algorithm line 3). Since it is equivalent to consider the Gumbel-top-k sampling for #4 = a- log (o:)) from Proposition we have
Pie (xice 22/a
P(1 = i i = i1. .1......4... S.....GG...... ==== (1
EiEMyyy pr(z;)1/a
for each E < ht When N- pe(x;) should be approximated by its expectation because of probability concentration:
- k is large (i.e., there are many masked positions), the sum of independent terms
Σ
p((x;)
Σ Exoon pr(o:)
Σ Ep(s))+1aa
N
N
VE(NiVe
While not mentioned Chang al. 2022
this "Gumbel temperature" is usod in their official implementation (https:// typical value ranges around 1.0 10.0 with additional step-dependent, scheduling
github. con/google----eeoooooooooooooeee rwwearcheeeeeeeee research
maskgit/ t/p.
Besnier Chen 2023 L.
2023
Comunità al. 2024b.
4

---

Published in Transactions on Machine Learning Research (04/2026)
To see its quantitative version, applying Bernstein's inequality (e.g.. Boucheron al. 2013 to the independent random variables pı(x;)4//a E [0. 1] yields, for 1 1>> 0,
Corollary 2.11)
t 2( s 2exp ( + t/3) 2t
E WEINTVAL1
P((z))
E
E pi((x) Va]
WEINTNAL1
where a² is the variance of DiEINTVe (1, Since 0 S (r())) & 1, it is upper-bounded o²
E [I(((x)) S Eie(NIVe- E B[pr(x:)V/]] By substituting this upper bound and letting /
c E LiEENN
E (xp)(x)))/)) for an c <> in the Bernstein estimate, we can quantify the concentration of (2
as follows:
P Si (1;))
26
2xxp
3e
Σ
E E(p((a))V/a))
EIEINMML, 1 (p)
SE(N]AT k + +
3(N
< 2 exp
8|5|1/a
where the second inequality follows from N - - N (8- 1) and the lower bound of E [p.((;)) using the vocabulary size |S], shown by (17 in the appendix. Although the actual proof is more involved, this concentration is at the heart of proving our main theorem (Theorem
Since Exes (x +1/a the (1+1/a)-th power of (1+1/a)-norm of when regarded as a vector in gls), we can simply write EiEINTAL // EiEN peppllll 11 Under this approximation (including the same approximation lor smaller f), we can derive
Pie (x 1411/
P = ie 12 i1 i1....,, ......, = ie-1)
(3)
EvENN 1p Ind 241/a
where the right-hand side makes a probability distribution over S x ([N] \ Te-1). The derivation of choosing indices i,,..., with the right-hand side of (3)
3 is
deferred to Appendix B.I. From Proposition [
(summed over 2i, S) is equivalent to Gumbel-top-k sampling with μ4 logllp:// index i, the sampling distribution for x; is proportional P. 1-1/a
1+1/a Then, given the
3.1 Moment sampler
Based on the above analysis, the moment sampler, our alternative sampler for approximating MaskGIT, is formulated follows:
(MM1) Let (i...... i = argtopkie(N flog Ip: ll 1/11/
1+1/a + &.} with i.i.d standard Gumbel (5:)
(MM2) Independently sample D. with p. P; 1+1/a for each i E (f1,...,444
(MM3) Return the indices i1,..., in and samples Tig
Since the exponent + 1/a in MM2 is a potential source of sampling error discuss later in this section, we also consider the "unbiased" version of second step, i.c., simply sampling from P.. To cover both of these options of sampling from x pž with exponent 7 + 1/a and 1, we formalize the moment sampler in Algorithm 2 with a general exponent 7 in (MM2) Tn the experiments in Sect 5 = + 1/a defines Moment, whereas the choice 7 = provides U-Moment. See also Figures Cand in the appendix for comparing the MaskGIT and moment samplers in actual implementation using Pyrch...
Mathematically, prove the following:

---

Published in Transactions on Machine Learning Research (04/2026)
Theorem 2 (Moment sampler approximates MaskGIT in the N > k² regime). Let PI.....PP .N be probability
distributions over a finile set S. For k E [N] and > O, lel PMaskGTT be the oulpul distribution of the corresponding MaskGIT sampler (Algorithm D i.e., the distribution of (i1,..., .i)) and (Ti, ) over [N] x S Similarby, let Pmoment be the output distribution of the moment sampler (Algorithm with 7 = 1+ 1/a). Then, we have
1//S///
drv (Pmomentt PMankGIT) $ 5V
1 1
log"
N
k2
where drv denotes the total variation distance and log 1x)) == log(max x}) for x E R.
For a formal argument, see Appendix C.6 and Theorem Since /8/2/a is constant and N/k is approxi- mately the number of steps, gets tighter when #(steps) > #unnmasked tokens per step). In this regime, MaskGIT is approximated well by the moment sampler, which samples from different (or biased) distribu- tion compared to the true marginal because of the exponent 1 the degraded behavior of the MaskGIT sampler we increase the number of steps (see also Section 3.3 Moreover, even outside the a above regime, we empirically observe that the moment sampler shows fairly close behavior with MaskGIT in terms of evaluation metrics (Figures 3 and Leftt)
1+1/a in (MM2) It thus partially explains
From Theorem 2 we can approximately decompose the MaskGIT sampler into index selection and temper- ature sampling (Figure) 1. This allows better understanding: for instance, in our experiments, temperature sampling is the dominant factor as we can mostly replicate the performance of MaskGIT even if omit the index selection of moment sampler (Section 5). Also, from practical viewpoint, we improve the moment sampler by techniques not applicable to MaskGIT (Section 4h.
3.2
thoose-then-samee algorithms
Arguably the most notable difference of the moment sampler from MaskGIT is that choose the indices to unmask before sampling token in each position. Let us call this strategy choose-then-sample (CTS) in general, which we formalize and analyze below.
Given the set of currently unmaskod indicos and sample æ,, a step of CTS algorithms over S with inverse lemperature 7 > 0 can be formalized as follows:
(CTS1) Sample J c [D] I for where to unmask. Its distribution is denoted as T(111, x1).
(CTS2) Σj ~ Pj with pj Pyjr(-lay)* for cach j E J.
We also present the whole algorithm as pscudocode in Algorithm 3 in the appendix. The unmasking position distribution is usually determined from the information regarding (151(() (a)))) such entropy and probability margin (Xiang et al. 2023 Kim et al. 2025).
It is worth noting that, unlike the "sample-then-chet cchoosc""cccc MaskGIT, CTS algorithms without temperature (i.e., 7= 1) return the correct distribution as we increase the number of steps, provided that its marginals are accurate. We formally prove this in Proposition 3 It is also mentioned by Ben- Ben-Hamu et al. 2025 Section 5) without a formal proof, and we prove it for completeness in Appendix C.1
Proposition 3 (One-by-one CTS algorithm is unbiased). In a CTS algorithm with 7 = 1, let us further assume that J ~ ,(-1T, ær) in (CTS1) is always a singleton set (i.e., |J] = 1) and Paj = qs holds for each I I [D] and E [D] \ Then, we have x qdata for the generated sample æ.
We next discuss the significance of CTS reformulation beyond providing an alternative to MaskGIT.
3.3 Why CTS reformulation essential
Theorem 2 establishes an asymptotic equivalence between MaskGIT and the moment sampler. Importantly, this result goes beyond providing an alternative implementation: converts MaskGIT's sample-then-chee choooe mechanism into an explicit CTS formulation, in which index selection and token sampling are decoupled.
6

---

Published in Transactions on Machine Learning Research (04/2026)
This separation allows us to analyze correctness independently of ordering: in Proposition show that CTS algorithms without temperature (r are unbiased in the one-by-one extreme, provided that one- token marginals are accurate. This observation is difficult to express in MaskGIT's original formulation, where token sampling and index selection are intrinsically entangled.
The CTS view also provides a principled explanation for MaskGIT's degradation as the number of steps increases (Gat al., 2024 Ren al., 2025b). Under the moment-sampler approximation, MaskGIT virtually samples from distributions proportional to P; 1+1/a rather than the estimated marginal introducing a persistent bias that accumulates across refinement. steps. This interprotation isolates temperature as the dominant factor governing MaskGIT behavior, which we verify empirically Section S
Beyond these theoretical implications, the CTS reformulation opens an algorithmic design space that is inaccessible in the original MaskGIT formulation. Because index selection is explicitly decoupled from token sampling, one can introduce ordering-level improvements without interfering with token sampling, such the partial caching approximation (Section 4.1 and the hybrid ordering strategy (Section 4.2)
4 Towards more efficient choose-then--ampp algorithms
One benefit of CTS algorithms pointed out by Zheng et al. (2025 is that avoid sampling from N categorical distributions over S, which can be costly if N and S are large. In this section, we further provide two techniques for enhancing CTS algorithms.
4.1 Partial caching approximation with transformers
Intermediate output
input
Q
ranked P:
KT
X L
already unmesked
sample
[M]]
demyctffy
already unmasked
demusitff
chosen
3rd
indices
MasleScT
[W]]
4th
[M]
not chosen
not chosen
Value caching
Key caching
V
output
L updated for e
KT
already unmasked
demyssfffy
sample
demustiff
Mosksgiff
unmasked
MasksSTT
[W]]
sempler
Figure 2: Illustration of partial caching approximation applied to an L-layer transformer, where softmax(:/ /Vd),, with de being the dimension of key and query vectors.
In masked diffuusio modeling, it is common to use a bidirectional transformer for computing logits at each position, where the standard kay-value (KV) caching for transformers with causal masking (Katharopoulos et al.] 2020) is not applicable. However, CTS mothods, we can use a similar idea to KV-caching for virtually increasing the sampling steps without proportionally increasing the cost.
Consider CTS unmasking step. Let [D] be the set of all the indices and U [D] be the set of already unmasked positions with values æv. To compute (. for E [D] \, U, we input D tokens to the transformer: at each position E U and M at all the other positions. In ordinary CTS methods, we then sample 1 ~ (U, the set of indices to unmask at this step, where π can depend on the inferred (pqu((laa)) (-Izu)) Finally, we sample I; ~ (+(o)) for each E 1.
Our partial caching method can incorporate arguably finer-grained token dependencies within First, we cache the key-value vectors as in the standard KV-caching. Then, divide I into two A, B C with
7

---

Published in Transactions on Machine Learning Research (04/2026)
AUB and An = 0. For i E A, just sample 2; peu(()xu) We approrimate sampling from PiuuA(: (aUUA) ((auuA)) for remaining indices E B with low cost as follows:
Run the transformer only at positions E 1, with the imput æ, for E A and for i e B.
As key-value vectors positions E 7, we use the cached ones.
It does not fully compute the effect of input change (from M to at each position E A) over positions other than AUB. However, it approximates the attention change for positions i E B that directly comes from 2A: thus boosting the empirical performance (Figure 4.. The second step operates with /l/D times computation of the original inference, and the empirical performance gain outweighs its additional computation. practice, we can determine the partition AUB by introducing an ordering. For instance, we let as the first half of in the order of confidence or preference. Figure 2 shows an example of such two-stage sampling process. See Section G for comparison with other caching techniques for discrete diffusion.
4.2 Balancing exploration and exploitation
In this section, we decompose the error of a CTS algorithm and see there are two types of existing algorithms depending on which part of the the error to optimize. Crucially, the CTS formulation makes such a decomposition possible the level of index selection: since ordering explicit, we can reason about how the choice of T trades off different sources error without entangling with token sampling. Motivatod by the crror bound, we design hybrid (bi-criteria) ordering that explicitly balances terms corresponding to exploitation and exploration.
Proposition (KL decomposition for two-round CTS). Consider two-round CTS procedure on the index set [N] with the ground-truth distribution 9. In the first round, size-k subset C [N] is chosen, and tokens at eiel are generated 24 qs, independently to each other. Then, the remaining tokens generated independently Ij ~ ()r(((as) for E [N].. Let be the resulting distribution of this two-round procedure. Then, we have
DKL(gllp) = DKL
II iE!
+ Exrrar DKL ()qp(1))))
Il ig!
Σ IET
(g))
(- H(qg)(1<<))) ((tp)/)))))aaaaa(((((((((a)aa)))
Egyoag
((ga(((1z))
(4)
TET
jE[NJ
(a) explostaaiin
(b) spatial dispersion
(c)
exploration
where H() denotes the entropy of probability distribution and s0-11") for I' c [N] is distribution over the subsets of I' defined as ((J[ ( 1)-
The formal proof of Proposition 4 given in Appendix C.2 After surveying existing approaches, propose
"hybrid" approach balancing the optimization of different terms in
Exploitation. Most of the existing adaptive unmasking schemes Xiang et al. 2023 Zheng al. 2024
Kim et al.) 2025) just focus greedy minimization of (4)), if not explicitly based on entropy. Let us call this strategy exploitation. Term (4) is utilized by [en-Hammu et al. (2025) for justifying adaptive selection of k, through its lower bound max,er H (q.), but is not used for selecting itself, which chosen through exploitation given k.
Exploration. In contrast to exploitation methods, Halton-Mask (Besnier et al. 2025 tries to maxi- mize 4b) by using the two-dimensional Halton sequence (Halton 1960) for image modeling, which prevents the conditional entropy Emgrog H ({g))((=))) from becoming small (see Besnier et al. 2025| Section B). While not discussed, the use of low-discrepancy sequence including Halton sequence also aligns with the

---

Published in Transactions on Machine Learning Research (04/2026)
minimization of c)). Let us call exploration the strategy of using I whose indices non-informative to each other and informative as set to the rest of indices.
Proposition suggests natural trade-off between exploration and exploitation: selecting 1 greedily by confidence as in exploitation methods tends to reduce 4a) but may worsen b) and (c)), while spatially dispersed choices (exploration) do the opposite. Our hybrid (bi-criteria) ordering natural consequence from the above observations: we allocate a fraction of the budget an exploitation-orieneed ordering and use the remainder for exploration-oriented ordering.
Proposed method: Hybrid approach. Let i (2...)) and = (21.22....) be two orderings of indices given by different strategies. To determine n indices lo unmask in the next step, simply take the first m (< 1z) indices from and follow the ordering of for the rest to make merged ordering k. Here is an example in the case n = 4 and m = 2:
(2.3,6,5, 1,4,) j - (4,,,,,,,,6
k: == (,,,6,
where underlined indices are chosen for the next unmasking step. In our experiment in Section 5.2 we use the Halton sequence (exploration) for i and moment- noment-based ordering (exploitation) for j. We can also apply the caching in the previous section by using the merged ordering.
5 Experiments
We evaluate our theory and proposed methods on unconditional generation tasks in image and text domains. Additional experimental details are provided in Appendix D The primary objectives of our experiments differ across domains, In the image domain (Section 5.1). where MaskGIT already demonstrates strong perfor- mance, our goal is validate our theoretical analysis (that the moment sampler approximates MaskGIT; Theorem and to isolate the role of implicit temperature sampling, rather than to pursue further perfor- mance improvements. In contrast, in the language domain (Section 5.2 evaluate the practical impact of CTS-based design choices efficiency and quality-diversity trade-ofis.
5.1
Image modeling
In our experiment in the image domain, adopted MAGE (Li et al. 2023 ViT-B model) the pretrained masked diffusion model. MAGE can be regarded as masked diffusion model over VQGAN (Esser et al, 2021b tokenizer space, trained on ImageNet 256 x 256 (Deng et al. 2009) for unconditional generation. Following the original implementation of MAGE, we employ Gumbel temperature of a(1 - n//N) for the n-th step out of steps, where a is the global temperature parameter, and we adopt the cosine unmasking schedule (Chang et al. 2022). The schedules are shared within the tested methods. We compared MaskGIT (Algorithm ! and Moment (with y = + 1/a in Algorithm without temperature sampling in the final step) along with the following:
Temp: We only conduct temperature sampling in Moment, i.e., replace the step (MM1) with uniformly
random selection of indices.
Random: Vanilla discrete diffusion sampler conditioned with fixed number of unmasking indices. It is equivalent to MaskGIT with a + co or at-random index selection.
Halton: Index ordering is given by a fixed two-dimensional Halton sequence as Besnier et al. 2025). There is no temperature involved in this sampler.
Figure 3 shows the comparison of these five samplers combined with three choices of global Gumbel tem- perature: a e {3.0, 6.0, 12.03 (a = 9.0 omitted to avoid making the figures too dense). Additionally, Table D shows the relative difference of MaskGIT and other samplers, in terms of the ImageNet FID perfor- mance. We can see that Moment shows similar performance MaskGIT, supporting our theory. A surprising finding is that we almost replicate the performance of MaskGIT using Temp without adaptive ordering,

---

Published in Transactions on Machine Learning Research (04/2026)
which suggests that the performance of the MaskGIT sampler with MAGE primarily stems from its implicit temperature sampling rather than the confidence-based ordering. We emphasize that these image results are not intended to demonstrate improvements over MaskGIT. Instead, they serve to validate our analysis by confirming that (i) the moment sampler clearly approximates MaskGIT and (ii) sampling temperature largely determines MaskGIT's behavior in this regime. Consequently, we focus on practical gains in the language domain.
Table 1: Mean relative difference (%) of ImageNet FIDs against reference MaskGIT samplers across 8, 16, 32, steps unconditional experiment with MAGE. Given reference sequence (Ia)s, mean relative difference of sequence (yi); was computed as average of w1/zz/. Difference under 10% is bolded. Moment and Temp closely approximate MaskGIT with the same temperature.
Ref. MaskGIT temperature
MaskGIT
Moment
Temp
Random
3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
a = 3.0
67.1
37.8
50.4
4.1
38.5
47.3
5.7
36.4
47.5
a= = 6.0
121.9
77.0
25.1
83.7
4.4
20.0
74.3
3.1
21.6
a 12.0
113.9
106.8
24.5
113.4
27.6
4.3
98.8
24.0
3.0
In unconditional experiments with MAGE, we also tested our caching method Figure Left). Moment+Cache applies the caching technique in Section 4.1 to Moment for creating an "intermediate" " step at each sampling step (sce Appendix D.2 for details). For cach sampler, a E {3.0, 6.0, 9.0, 12.0} for each number of sampling steps N E {8, 16, 32, 64), against the average sampling time per batch. The experimental result suggests that, while the performance gain might not be large enough to make the latency overhead negligible, the partial caching indeed gives some performance boast.
plotted the best FID among
As a supplementary experiment, we compared these samplers in class-conditions setting, using MaskGIT- PyTorch (Besnier & Chen (023) as pretrained masked model. The relative differences are reported Table the appendix. In this setting, Moment again well approximates MaskGIT with the same temperature, while Temp is not necessarily aligned well with MasKGIT.
5.2 Language modeling
In the language domain, we focus on evaluating the practical consequences of the CTS reformulation, since diversity degradation caused by temperature sampling is more pronounced and efficiency gains are more meaningful in this setting. We used SDTT (Deschenaux & Gulcehre 2025 small model with KL diver- gence target) as the pretrained model, which was obtained after seven rounds of distillation of the MDLM model (Sahoo et al.] 2024) trained on the OpenWebTText dataset (Gokaslan & Cohen 2019). SDTT is a masked diffusion model over the space of GPT-2 tokenizer (Radford et al. 2019). In the experiments, we adopted the linear unmasking schedule (i.e., unmasking the same number of positions at each step) and the same Gumbel temperature schedule as in Section 5.1 As evaluation metrics, we adopted Generative Perplexity (with GPTT-2large) and Entropy, which are in a trade-off relationship, following recent literature GGat et al. 2024 Zheng et al. 2025
We first compared MaskGIT, Moment, Temp, and Random as described in the previous section. We confirmed the tendency that the temperature roughly determines the performance; while the methods with temperature sampling attain lower (= better) Generative Perplexity (Figure g Left), their generated sentences have extremely lower Entropy (Figure 3 Left), thus harming the overall quality. To mitigate this issue, we introduced U-Moment (U for "unbiased"). where we only conduct index selection and omit the temperature sampling in Moment, i.e., the case of 7 = 1 in Algorithm 2 In the experiment with "unbiased" met hods (Figure(3) Right), Halton and U-Moment showed contrastive behaviors, where the former gave better trade-off with fewer stops, and the latter worked well with more steps. By merging these two into Hybrid as explained in Section 4.2 we obtain uniformly better trade-off compared to Random (sce Appendix D.4.2 for the details of Hybrid).
10

---

Published in Transactions on Machine Learning Research (04/2026)
50
MaskGIT
Halton
90
Moment
a-3.0
Temp
80
40
a-12.0
Inception Score
30
60
50
20
MaskGIT
Halten
Moment
a-3.0
Temp
q=6.0
10
30
Random
a= 12.0
20
40
50
60
20
40
50
60
Steps
Steps
Figure 3: Fréchet Inception Distance (FID, 1) and Inception Score (1) against the number of steps for various samplers with MAGE. Both metrics computed by 50,000 generated images. We that Moment closely approximales MaskGIT with the same lemperature in both metrics; their quantitalive approximalion precision is given in Table 1
MaskGIT
Random
Moment
120
Random+ Cache
Moment+ Cache
Generative Perplexity
Hybrid
16
100
Hybrid Cache
FID
80
12
40
2
2
2
2
22
2
2
Sampling Time (seconds)
Sampling Time (seconds)
Figure 4: Average performance gains of our proposed samplers against sampling time per batch (on A6000 GPU). Fach shaded region shows the standard deviation over three trials. (Left) FID of samplers applied to MAGE. (Right) Generative Perplexity (L) of samplers applied to SDTT. Especially, Hybrid achieves approximately 2x speedup to achiove the same Generative Perplexity as Random.
5.5
U-Moment
120
Generative Perplexity
5.0
Random
100
Random+Cache
Hybrid
Entropy
4.5
Hybrid+Cache
80
4.0
MaskGIT
a-3.0
3.5
Temp
Moment
0=6.0
am9.0
40
Random
a=12.0
20
24
25
2'
5.20
5.25
5.30
5.35
5.40
Steps
Entropy
Figure Language experiments. Each plot was computed by 1,024 generated sentences with 1,024 to- kens with number of steps in {8, 16, 32, 64, 128, 256). (Left) Entropy (t) against the number of steps for lemperature-based methods. We can that temperature sampling significantly harms Entropy, one can also see qualitatively in Appendix E.5.2 (Right) Trade-off between Generative Perplexity and Entropy. Hybrid i samplers improve the trade-off uniformly over Random across different numbers of steps.
11

---

Published in Transactions on Machine Learning Research (04/2026)
By using Hybrid and/or +Cache, we can improve not only the trade-off (Figure Right) but also sampling efficiency in terms of computational time (Figure 4 Right), leading to 1.5-2x acceleration. It shows the effectiveness of our approaches in Section 4 Note that, while the improvement of Hybrid seems consistent, the efficiency gain of +Cache can depend on GPUs (Appendix D.2.1
To examine whether these trends persist beyond the (-TTa-small) model, we further evaluate Hybrid and related CTS variants on (i) a larger SDTT model, (ii) an SDTT model augmented with an expanded architecture by the Di4C distillation (Hayakawa et al. 2025), ; and (iii) LLaDA-SB (Nic et al.) 2025) in the appendix. As reported in Appendix E.3 Hybrid consistently improves the perplexity-entropy trade-off over Random and U-Moment in both scttings, with comparable or larger efficiency gains. These results suggest that the benefits of our proposed methods are not limited to a particular model size or training procedure.
6 Related work
We overview relevant studies on efficient mamplingmmdelling
modeling and theoretical analyses for discrete diffusion.
Efficient sampling schemes for discrete diffusion. Several efficient sampling schemes have been pro- posod for speeding up discrete diffusion. For general discrete diffusion models, not limited to masked diffu- sion, Park et al. (2025) proposed non-adaptive time-schedule optimization, and Ren et al. (2025b) proposed discrete diffusion analogues of high-order ODE solvers. They can be combined with our method in principle.
For masked diffusion setting, Chang et al. (022) proposed the MaskGIT sampler, which has been theo- retically analyzed in this paper. LLaDA (Nie et al., 2025) utilizes the MaskGIT sampler in sampling from diffusion LLMs. Besnier et al. (2025) adopts the two-dimensional Halton sequence for selecting unmasking positions in masked image modeling (exploration). There have been also a few top-k sampling (exploitation) methods including entropy (Xiang al. 2023 confidence (Zheng et. al. 2024), and probability margin (Kim) et al. 2025). Additionally, [en-Hamu et al. (2025) recently proposed an "entropy bounded" unmasking pro- cedure that adaptively determines the number of unmasked indices in combination with any top-k sampling criteria. See also Section 4.2 for how these methods can be understood from the viewpoints of exploration and exploitation. Finally, the first-hitting sampler (Zheng et al. 2025 demonstrated that the usual masked diffusion sampler can be accelerated by replacing it with an equivalent
CTS sampler.
Regarding the exploration-exploitttionn trade-off, |Wei et al. 2025 and Lee et al. 20255 propose dynamic sampling approaches for diffusion language models that go beyond simple exploitation by incorporating token- level and trajectory-level uncertainty, respectively. Fu et al. (2025) independently provide an information- theoretic lower bound on the number of unmasking steps for confidence-basee parallel sampling and propose Exppore-Then--xppoos Then-Exploit (ETE), which targets high-entropy tokens through beam search to trigger "cascades" of confident predictions; they address the exploration-explootttioo trade-off in adaptive unmasking of block diffusion AAriola el al. 2025) differently from ours in both theory and method, complementing our formal- ization in Section 4.2
Efficient modeling of dimensional correlation in discrete diffusion. In contrast to the previous section, there are also methods that require additional training of models for efficient sampling in discrete diffusion. For general discrete diffusion, Hayakaaa et al. (2025) proposed "mixture" modeling to distill dimensional correlations learned by many-step teacher models, and Chen et al. (025) pointed out that quantum circuits can realize a one-step diffusion sampler.
For masked diffusion, Lezama et al. (2023) proposed the discrete predictor-corrector sampling, which is essentially based on learning an additional model that determines which token to unmask (see also Peng
et al.
2025). There are two methods exploiting pretrained autoregressive model to recover dimensional
corrolations:
one is based on copula (Liu et al. 2025), and the other conducts importance sampling (uu
el al.
2025
similar to speculative decoding in autoregressive models (Guo & Ermon, 2025). Finally, Zhu
et al.
2025
distill pretrained masked image generation models into a one-step sampler.
Caching methods for masked diffusion. diffusion models (Arriola et al. 2025 Ma et al. 2025 They both focus on caching the KV information from
There are a couple of works on applying KV-eache to masked
12

---

Published in Transactions on Machine Learning Research (04/2026)
past tokens for long-term efficiency, whereas our method in Section 4.1 caches past (already unmasked) and future (not unmasked in this round) tokens for stepwise efficiency.
Thcorctical analyses of discrete diffusion. While this paper primarily studies the thcoretical approx- imation of MaskGIT with a CTS variant, there exist theoretical studies on discrete diffusion samplers that are particularly relevant to our Proposition
Ren ct al. 2025aa Theorem 3.3 & Corollary 3.4) derive Girsanov transform for discrete-state Markov processes and a subsequent KL-divergence estimate, which can be seen as a generalization of Proposition 4 in a continuous- time regime, whereas the main focus of our bound 4 is an interpretable error decomposition into exploration and exploitation rather than a tight convergence analysis.
Convergence guarantees for discrete maakeed diffusion models have been studied for various samplers includ-
ing 7-leaping (Campbell et al. 2022 Ren al.] 2025a Liang et al. 2025a), score-based denoising (Zhang
et al.
2025
Conforti et al.] 2025 Euler (Liang et al., 2025b; 2026) and its higher-order variants (Ren
ct al. 2025b,, uniformization and its variant (Chen & Ying 2024 Huang et al. 2025a). the first-hitting
sampler (Zheng et. al. 2025 Liang et al. 2026), as well as other sampling strategies (Huang et. al.] 2025b;
Dmitriev et al. 2026) Wan et al. 20255 also study convergence within the framework of discrete flow
matching (Gat et al. 2024).
Another line of work focuses on the statistical error of discrete diffusion models in the context of learning
theory (Srikanth et al. 2025 Wakasugi & Suzuki 2025
While these studies generally discuss the convergence of samplers models to the data distribution, our aim is to demystify and approximate the MaskGIT sampler, which is inherently biased (as discussed in Scction 3.3 but useful in practice. That said, their thcoretical rigor can be informative for designing better sampling strategies; lackling the exploration--xpllitaaion tradeoff (Section 4.2 using such tools could be an interesting future direction.
7 Conclusion
In this paper, we provided a theoretical analysis of the MaskGIT sampler for masked diffusion models, re- vealing its implicit temperature sampling mechanism. Through this analysis, we showed that MaskGTT is asymptotically equivalent to thoose-then--ample (CTS) formulation, instantiated as the moment sampler, which explicitly separates index selection from token sampling. This CTS reformulation yields interpretabil- ity and flexibility that enables efficiency techniques that do not arise from MaskGIT's original sample-ther then- choose formulation. We further enhanced the efficiency of general CTS algorithms through two techniques: a partial caching approximation for transformer-beee models and a hybrid approach that formalizes the iitooratioo-epplitaation trade-off in adaptive unmasking. Our experiments validate the theory in the image domain and demonstrate the practical benefits of CTS- based design choices in the language domain.
While our methods are post-hoc and can easily boost the efficiency of pretrained masked diffussin models whose outputs consist only of one-token marginals, more fundamental "modeling" approaches in Section t are needed to accurately capture the complex distributions of high-dimensional discrete data. Theoretical understanding and practical improvement of such methods will be important future work toward realization of more efficient. discrete generative models.
Broader Impact Statement
While this work mainly presents theoretical contributions, the proposed methods could be applied to gener- ate misleading or harmful content if used with inappropriately trained models. Furthermore, as our methods modify the sampling distribution, existing biases in pretrained models may be amplified or altered in un- predictable ways. We are committed to responsible use of these techniques and recommend implementing appropriate content. filtering mechanisms in practical applications.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Acknowledgments
We are grateful to Naoki Murata, Joan Scrrå, and the anonymous reviewers for their constructive feedback, which significantly improved the paper.
References
Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion lan- guage models. In Proceedings of the 13th International Conference on Learning Representations, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Damiel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Tn Advances in Neural Information Processing Systems, volume 34, pp. 17981 17993, 2021.
Heli Ben-Hamu, Itai Gat, Daniel Severo, Niklas Nolte, and Brian Karrer. Accelerated sampling from masked diffusion models via entropy bounded unmasking. In Advances in Neural Information Processing Systems, volume 38, 2025.
Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. arXiv preprint Xrx::23310. 14400, 2023.
Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthicu Cord. Halton scheduler for masked generative image transformer. In Proceedings of the 13th International Conference on Learning Representations, 2025.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: Nonasymptotic
Theory of Independence. Oxford University Press, 2013.
Andrew Camphell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Advonces in Neural Information Processing Systems, volume 35, pp. 19266-28277 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Chuangtao Chen, Qinglin Zhao, MengChu Zhou, Zhimin He, and Haozhen Situ. Overcoming dimensional factorization limits in discrete diffusion models through quantum joint distribution learning. arXio preprint srXiv:2505.0515) 2025.
Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact implementation through uniformization. arXw preprint arXio:2402.08095, 2024.
Marco Comunità, Zhi Zhong, Akira Takahashi, Shiqi Yang, Mengjie Zhao, Koichi Saito, Yukara Ikemiya, Takashi Shibuya, Shusuke Takahashi, and Yuki Mitsufuji. SpecMaskGIT: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond. In Proceedings of the 25th International Sociely for Music Information Retrieval Conference, pp. 420 428, 2024.
Giovanni Conforti, Alain Durmus, Le Le-Tuyet-Nhi Pham, and Gael Raoul. Non-asymplotic convergence of discrete diffusion models: Masked and random walk dynamics. arXiv preprint arXiv:2512, 00580, 2025.
Thomas M. Cover and Joy A. Thomas. Elements of information theory. John Wiley & Sons, second edition, 2006.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. Teee, 2009.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Justin Deschenaux and Caglar Gulcchre. Beyond autoregression: Fast LLMs via self-distillation through time. In Proceedings of the 13th International Conference on Learning Representations, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Newral Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Daniil Dmitriev, Zhihan Huang, and Yuting Wei. Efficient sampling with discrete diffusion models: Sharp and adaptive guarantees. arXiv preprint orXw:2602.15005 2026.
Abhimanyu Dubey; Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models, arXiv preprint arXiv:2407221783 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 12873 12883, 2021.
Hengyu Fu, Baihe Huang, Virginia Adams, Charles Wang, Venkat Srinivasan, and Jiantao Jiao. From bits rounds: Parallel decoding with exploration for diffusson language models. arXiv preprint or 2111:2351..21103, 2025.
Hugo F Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. VampNet: Music generation via masked acoustic token modeling, In ISMIR 2023 Hybrid Conference, 2023.
Itai Gat, Tal Remez, Neta Shaul, Felix Krcuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In Advances in Neural Information Processing Systems, volume 37, pp. 133345 133385, 2024.
Aaron Gokaslan and Vanya Cohen. OpenWebTeextCrppss 2019.
Openwebtext corpus.
http://SkyLion007.ghh 1o/
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for lext-lo-image synthesis. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Gabe Guo and Stefano Ermon. Reviving any-subset autoregressive models with principled parallel sampling and speculative decoding. arXiv preprint Xrv::2250 222.5.4000200.6 2025.
John H Halton. On the efficiency of certain quasi-random sequences of points in evaluating mulli-dimennional integrals. Numerische Mathematik, 2:84-90 90, 1960.
Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi, Hiromi Wakaki, and Yuki Mitsufuji. Distillation of discrete diffusion through dimensional correlations. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. Advances Neural Information Processing Systems, volume 33, pp. 6840 9840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, volume 35, pp. 8633 8646, 2022.
Xumpeng Huang, Yingyu Lin, Nishant Jain, Kaibo Wang, Difan Zou, Yian Ma, and Tong Zhang On the 6-free inference complexity of absorbing discrete diffusion. arXiv preprint arXiv:250..2 21835, 2025a.
Xumpeng Huang, Yingyu Lin, Nikki Lijing Kuang, Hanze Dong, Difan Zou, Yian Ma, and Tong Zhang, Almost linear convergence under minimal score assumptions: Quantized transition diffusion. aixiv preprint 271. 2025b.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International conference Machine Learning, pp. 5156 5165. PMLR, 2020.
Jacyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. In Proceedings of the 42nd International Conference Machine Learning, 2025.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiflWave: A versatile diffusion model for audio synthesis. In Proceeding of the 9th International Conference on Learning Representations, 2021.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top- k trick for sampling sequences without replacement. In Proceedings of the 36th International Conference on Machine Learning, pp. 1499-3508 2019.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends0 in Machine Learning, 5(2 3):123 286, 2012.
Sanghyun Lee, Seungryong Kim, Jongho Park, and Dongmin Park. Lookahead unmasking elicits accurate decoding in diffusion language models. arXiv preprint ar xrXv::2511.0550 2025.
Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Discrete predictor- corrector diffusion models for image synthesis. In Proceedings of the The Eleventh International Conference on Learning Representations, 2023.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: Masked generative encoder to unify representation learning and image synthesis. Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 2142-2152, 2023.
Yuchen Liang, Renxiang Huang, Lifeng Lai, Ness Shroff, and Yingbin Liang. Absorb and converge: Prov- able convergence guarantee for absorbing discrete diffusion models. In Advances in Newral Information Processing Systems, volume 38, 2025a.
Yuchen Liang, Yingbin Liang, Lifeng Lai, and Ness Shroff. Discrete diffusion models: Novel analysis and new sampler guarantces. In Advances in Neural Information Processing Systems, volume 38, 2025b.
Yuchen Liang, Zhiheng Tan, Ness Shroif, and Yingbin Liang. Sharp convergence rates for masked diffusion models. arXiv preprint arXiv:2602.220 22505, 2026.
Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. In Proceedings of the 13th International Conference Learning Representations, 2025.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dKV-cache: The cache for diffusion language models. In Advances in Neural Information Processing Systems, volume 38, 2025.
Chris J Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances in Neural Information Processing Systems, volume 27, pp. 3086 3094, 2014.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In Advances in Neural Information Processing Systems, volume 38, 2025.
Jingyang Ou, Shen Nic, Kaiwen Xue, Fongqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In Proceedings of the 13th International Conference Learning Represenlations, 2025.
16

---

Published in Transactions on Machine Learning Research (04/2026)
Yong- Hyun Park, Chich-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. In Proceedings of the 13th International Con- ference on Learning Representations, 2025.
Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Avishek Joey Bose, Alexander Tong, and Pranam Chatterjee. Path planning for masked diffusion model sampling. arviv preprint arXie:2502.088: 08540, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Llya Sutskever. Language models are unsupervised multitask learners. OpenAT blog, 2019.
Yinuo Ren, Haoxuan Chen, Grant M Rotskoff, and Lexing Ying, How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. In Proceedings of the 13th International Conference on Learning Representations, 2025a.
Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. In Advances in Newral Information Processing Systems, volume 38, 2025b.
Subham Sekhar Sahoo, Marianne Arriola, Yair SchifT, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Tn Advances in Neural Information Processing Systems, volume 36, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neurol Information Processing Systems, volume 37, pp. 103131-103167, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 36th International Conference on Machine Learning, pp. 2256 2265, 2015.
Aadithya Srikanth, Mudit Gaur, and Vaneet Aggarwal. Discrete state diffusion models: A sample complexity perspective. arXiv preprint arXiv:2510. 1005540.. 2025.
Shintaro Wakasugi and Taiji Suzuki. State size independent statistical error bound for discrete diffusion models. In Advances in Neural Information Processing Systems, 2025.
Zhengyan Wan, Yidong Ouyang, Qiang Yao, Liyan Xie, Fang Fang, Hongyuan Zha, and Guang Cheng. Error analysis of discrete flow with generator matching. arXiv preprint ar Xrxi:2500.22 215500922106666 2025.
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang. Accelerating diffusion large language models with slowfast sampling: The three golden principles. arXiv preprint 2rXiw:2506.108/ 2025.
Jinxi Xiang, Kuan Tian, and Jun Zhang. MIMT: Masked image modeling transformer for video compression. In Proceedings of the 11th International Conference on Learning Representations, 2023.
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Energy-based diffusion language models for text generation. In Proceedings of the 13th Interna- tional Conference on Learning Representations, 2025.
Zikun Zhang, Zixiang Chen, and Quanquan Gu. Convergence of score-based discrete diffusion models: A discrete-time analysis. In Proceedings of the 13th International Conference on Learning Representations, 2025.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yi Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffu- sion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In Proceedings of the 13th International Conference on Learning Representations, 2025.
17

---

Published in Transactions on Machine Learning Research (04/2026)
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In Proceedings of the 1st Conference on Language Modeling, 2024.
Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. Di[MO: Distilling masked diffusion
models into one-step generator. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18606-18618, 2025.
18

---

Published in Transactions on Machine Learning Research (04/2026)
A Algorithms
In this section, we itemize the algorithm pscudocodes of the MakGIT sampler (Algorithm D. Moment Sampler (Algorithm 2 and the general form of choose-then-sample methods (Algorithm 3 To demonstrate the actual implementation of the moment sampler and highlight its differences from MaskGIT, we give their example PyTorch implementations in Figures and
Algorithm One-round of MaskGIT sampler: OnnRRonndMaakkCT7((()))ss......
Require:
(Pi)er: Family of probability distributions over S, with [T] N ke E [N]: Number of indices to choose
a > 0: Gumbel temperature
Ensure:
(f1,..., (x) E Th: distinct indices
(z1,) 8%; Sampled lokens
Independently sample standard Gumbel noise & for each i Independently sample pi for each i E
3: (ii,..., ix) & argtopkier (log + aςi}
4: return (f1,...,4a) (ri,)=z
Algorithm One-round of Moment sampler: OneRooundMomenn((()))r... k a,,)
Require:
(pi)er: Family of probability distributions over S, with |4| N
1: E [N]: Number of indices to choose
a > 0: Gumbel temperature
? > Inverse sampling temperature (set 7 1 + 1/a when approximating MaskGIT)
Ensure:
(f1,..., ix) E I#: distinct indices
(Ri,)=z E Sh: Sampled tokens
1: Independently sample standard Gumbel noise & for each i E
2: (i1...., 1 + ar (log(E (x(x) + {r} 3: Independently sample 2i, ~ PP 17 for each e [k]
4: return (i1,...,i4), (x;,)
Algorithm 3 General choose-then-sampe algorithm
Require:
a.)): Distribution nonempty subsets of [D] conditioned by and æ
Pj (+arr) Dimension-wise denoising model for any j. I, æ
7 > 0: Inverse sampling temperature
Ensure:
x SD: Generated sample
Initialize 5-00
while & [D] do
Sample (1-11, 2r)
Ignore Wg 0 when ZrU has been determined far
Pytr( Ør)"
Sample 2y
for each E J
x17
I+/
6: end while
7: return
19

---

Published in Transactions on Machine Learning Research (04/2026)
def maekgit__ampler (nodel, batch_aize, seq_len, aue_atepa, mask_id, alpha):
MaskCIT sampler with multiple rounds
Args:
model: Prediction model betch_size: Batch size
seg. len: Sequence length (D)
sum steps: Number sampling steps
10
mask
token
the vocabulary
alpha: Gumbel temperature for top-k
18
13
Instsalsze: all tokens are masked x torch. torch.full (bll((bbac_eeize, seq_len), maek_id)
14
15
Compute (assume sum_steps is divésor of seq_ ten)
16
k seq_len // stepe
17
18
for t in (an_ssmu_stepaa
19
# Get probabsisty distridution p_ at each posstion logits = (x)
log_probs- = torch. Borch..o__ofttaxx1gge
dim=-1) # log p_t
Step 1: Sample standard Gumbel noise for each
gumbel_nost noise torch dis.rhiittrrbbionss Seterrbbbtionn..mmeee..... Gumbel (0, 1) sample( (batch_size, sen]
Step Sample #_á from p. for cat cat_dist torch. distributions. Categorical (logitteggg) samples cat_dist. cat_dist.sample(( aai's
29
30
31
32
Step 3: Select top-k indices ($_1, .... k_k)
53
Get log P_ p_$(=_t) () for sampled takens sampled_log.prott = torch gather( log_proba, dim=-1, uuupaeseaaallssuuuuuueeete1 >.queeze(11) # log (x_s)
34
36
36
# Calculate confidence with Cumbel mosse confidence sampled_log_proot + alpha gumbel_noise
Enclude already unmasked positions from top-k desermination confidence (x mask_id)] float float ('-inf')
Get top-k indices with highest confidence unmask)
45
indices to_ unmask topk( confidence, k, largest-True
46
47
48
49
Create boolean mask for indsces to unmask
50
unmask torch. sercc..zere__i. Barch.zeros_likeGe lxke(x, dtype-torch, bool) unmask acatter_ (1, saaiicsstt_uumaaeee True)
01
52
58
Step 4: Update only the selected posétions [unmask] samplea [unmaak]
54
return
Figure 6: Example PyTorc implementation of MaskGIT sampler. For simplicity, unmask the number of positions at each step, and the Gumbel temperature is held constant throughout sampling itera- tions. The "Steps" in the comments correspond to those in Algorithm I and the highlighted lines show the differences from Moment sampler in Figure 7
20

---

Published in Transactions on Machine Learning Research (04/2026)
def moment_ sampletr_ammleeeeddee (model, batch_size, seq_len, nun_steps, id, alpha, gamma):
Noment ler with multiple rounds
Args:
model, alpha: Same maskgit_sampier
(alpha assumed positive for simplecity)
gamma: Inverse samplèng temperature
10
(set gamma 1/aipha approzimating MaskGIT)
non
12
Inétialsze: tokens masked
13
x torch. full (batch_aize, seq_len), maek_id)
14
Compute (assume num_ steps is devssor of seq_ ten)
15
k Beq_len // stepe
16
17
for t in range (oun_ssteppp))
18
# Get probabsisty distridution p_ at each posstion logits = mx)el(x) log_probs norch.log_softmxx11eg
19
dim=-1) # log p_t
Step 1: Sample standard Gumbel noise for each gumbel_noist soceell__sse___ee terch distributions Gumbel(O, 1) sample (batch_aize, seq_len)
Step Select top-k indices (1_1, ...., k_k) Compute log(sun, a P. (=) ((+1/alpha)) for each moment, scores (
29
30
1/alpha) log_probs, dimm-1
31
52
53
Calculate confidence with Gumbel noise confidence noment_acores gumbel_noise
34
36
36
# Exclude already uamasked poastions from top-k determsnation confidence ((x I= mask_id)] float Eloaa('-infs ('-inf')
# Get top-k indices with highest confidence (to unmask) indices__oe unmask torch. topk( confidence, Targest-True
Step Sample a e[t) from P_ [_f(_ell) "gamna for each ell Entract log-probs only at posstions unmask ([batch_ size, batch_indes indices torch. (bange size)) unsqueeze(1) expand(-1, selected loteptedd_og_pobb- log_probs
45
47
indices, unmask]
48
49
Somple le from the distridutson proportional to P_ ((l) "gamma cat cat_dist torch, distrributiinn cwt.rbbbrrdine.ccttggor..... logits=ganma gelected_log_preet
50
51
52
58
samples dit_.ist. sample)
54
Step Update only the selected posétions
[batch, [batch_indices indices, indices_te [occcssst__umakk samples
return
Figure 7: Example Py'Torch implementation of moment sampler. For simplicity, we unmask the same number of positions at cach step, and the Gumbel temperature is held constant throughout sampling itcrations. The "Steps" in the comments correspond to those in Algorithm and the highlighted lines show the differences from MI sampler in Figure
21

---

Published in Transactions on Machine Learning Research (04/2026)
B Formal and informal derivation of moment sampler
For notational simplicity in the proofs, let B = 1 + 1/a throughout this section.
B.1 Informal derivation of moment sampler
In this section, we derive the moment sampler approximation 3 from approximation to we have
and
2
First, by applying the
Pi, 1/a
(P ie 17 in G-1 1
(5)
EveNNVa 11:
for each I,...,k. Then, by multiplying this for t = I,...,e l- 1, we obtain
Pi, (Ir,) )1/2
= - P( ......,. 12-1= (x:)
Σ EEEN
Note that the right-hand side is independent of (3i))e(N][e so we can replace the conditioning on (c:) by (ai)ketes By marginalizing out x, with E Ie-1 from this, we have
P( = i...... = ie xss x=,)= Σ
Σ
P( = i1..... = de (x)) (P xs,)
Zit
Fig-1
sy Pie (Zi,) Pie (2it))
Pie (tiee
ExEINRA I1pil13
lppi 1p:
Die (xi )
=:Co-1 (Ca-i((amnde-1))
Therefore, we have
P(ù .i. = is xze P(ate))
Pl = ie i1,...,,1211 = =
P(i .,,..., .//11 = ic-1)
Ch.ilh......ee (8-11)
Pi, (r
Pi, (I )3
x Σ
P(i i1, ..,... 82-1 i E 11p:11
EiE(NTVer l1p:11@
Since the final right- heaht-handdao side actually normalized probability distribution over S ([N) \ Ie-1), we have derived
B.2 Strategy for formal proof of Theorem
Formally, we work under the following setting:
Setting A (MaskGIT sampler). We are given 0 and N probability distributions PI.....PN over finile set S. We sample I; ~ P: and standard Gumbel noise & independently for each i E [N], and let (ii...., i = (
+ afi} with k E [N].
Corollary 2.11) to the sum of pa(x,) E (x, Since the summand is always within [1, 1]. for 2 0, we have
Let us start by applying Bernstein's inequality (see, e.g., Boucheron et al. 2013
N P
N
(x) S s-t S <exp (- 2( + +t/3) +t/3))
Σ
P. (x. 1/a Σ
(6)
22

---

Published in Transactions on Machine Learning Research (04/2026)
where o² : == B (p.(ax))// - -B(((t;)a Since S pi(x;) I 1, we have
E p=( l/ (x(1)) $
E p((
(7)
Now, recall that E ( lppil & for B 1 + 1/a. By letting t = > lleil''s for some 0 S é S 1 and combining
and 7 we have
2 Σ - |lppi )
P
Σ p.(x,)
s (1-c) e)
Σ
S exp
2 2 1=1 I 1 + E 1ppi 1//33
==1
62
E N 111.111113)
exp
2(1 + e/3)
3c2
S exp
11pa112
8
To utilize this estimate for our analysis of MaskGIT sampler, we prove the following proposition. The proof is given in Appendix C.3 Proposition 5. Under Setting let [N]c denote the set of all the subsets of [N] with cordinality less than k. Then, for E 1], we have
(b: - 1)81//
Σ KEN
pa(x;) 1/a
1-
Σ NNMM
1p:ll
for all I E [N]ck
(9)
N k +1
with probability at least 1 - exp
(-3
We use Proposition 5 prove the following assertion, whose proof is given in Appendix C.4
(k-1))8)1/a holds. For E [0, 1], let us define Z, c S
Proposition 6. Under Setting A assume e+ N-k+1 the set of (2 E SN that satisfies the following incquality:
P( i1. o = is (x:) = (2))
(k -k-1)/88///
-k
Pie (zi
N N
Eie(NIV 111ii1/8 B
where 1--1 == ig- Then, have P((x)) E Z.) > 1 exp (- (1)1//
We next compute the probability distribution of the moment sampler.
Setting B (Moment sampler). We are given a > 0 and N probability distributions PI...., PN over a finite set S. Let 3 =11+10 1+1+1/
E [N], and let 1/aa. We sample y/4 ~ p, 11 and standord Gumbel noise qi independently for each sI (logl p: m with k [N].
Proposition 7. Under Setting B for each distinct indices i is E [N] and (not necessarily distinct)
Zig Zi, E S, we have
F(j = i......55 de = ik. (B52 )a. = (24, )
Pi, (2i,))
(10)
EVENTT 11p:11
where Te-1 (i1,... ic...))
23

---

Published in Transactions on Machine Learning Research (04/2026)
The proof is given in Appendix C.5
Let us finally prove Theorem
The following is its restatement under the above settings.
Theorem 8. Under Setting A
for each i = (i1,-.., ix) E [N]k and ) St, let
PMaskGIT (i, (z4, (1,))=))) == IP(11 = i1. . in: (#ig )ē=ı (E )
Pmoment (ē, (Zte )==) == IP j i J in. (15; )ē=1 (2 (=1e)==1)
Then, as probability distributions over [N]* st, we have
1:2 12/S/1/a
N logg
(Puomentt PMuskGIT S 5bV N
(1+ ++y1os
where log (x) == log x1) for E R.
This main result is proven in Appendix C.6
C Proofs
C.1 Proof of Proposition 3
Proof. Let In = and inductively (and randomly) define
~ (-1101--1), Ton ~ Pa.lua- ( (-12) I, == In
for = I,...,,. D, where we abuse the notation to simplify {an} i into on т. Note that (o1,..., oD is a random permutation of (1....,D) D). Let us write p(æ, or) be the joint distribution of x and a = (a,)n= (the latter constrained to be a permutation). Then, we have
D II
p(a,
n (on 11,, Ponlnnn (No I (.1,-)))
n=1
PD (1 ( ([,):))))_>) mmm
II naml
Ponlf (Fon (oon)))))))) Idata (x)
II a (l))))
from the assumption Pşu = 4zlr. Thus, it suffices to prove that Σ. I. v(on|<)n--)) 1. To this end, let us prove the following for le 1..... D by induction on l (the case k D what we would like prove ultimately):
Σ
II
(o.,|,1, 1) ,)= =1
for any T with |/ = D k and ætp #
(11)
TD-K+T.-... DD D=D-
It is clearly true for & = 1, since it is just the total probability sum of the probability distribution Tt a-1101-11) For 2 2, from the induction hypothesis, we have
D
D II
Σ
II
< [0)1)))-1)
Σ 7(0D-1+1111--)
Σ
GD- n=D- -=
ap-
-D- 7 D+11
=1 by induction hypothesis
Σ
a(oD-1t1l11 )) =1
ap-k-t
Therefore, the proof has been completed.
24

---

Published in Transactions on Machine Learning Research (04/2026)
C.2 Proof of Proposition
Proof. By using the chain rule of KL divergence (Cover & Thomas 2006
Theorem 2.5.3), we have
De (Ip) (2 || P1) + Ber DKL (1 p=-p(-(a)))]
(12)
Let first consider the KT divergence between or and pt- First, we which shows the first inequality in
have
II (a log (r1)
DKL(ar pT) Emywas
log
ViEl
Brr,ngg |- log qe(x;:]] E (-lggan gr
iEl
Σ ((q))
(13)
Next, by using a permutation (o1......0, as follows:
0%) of I, the remainder term of
13 (entropy qı) can be rewritten
E Bay |- (rg
= Σ
II
20,1,, (Far [1 log 2or (For (x-1e1))
Bey j=l
Σ
Ue Ue-1 (oulle) (Zor [x log Ro, (For (11e)
1=1 sonfor
Emy - Eus, gos le-1 [- log qoelTe (Foz [x16-
Exi ~ (177
(14)
Let us consider laking the average of the right-hand side over all the permutations. With the uniformly random permutation (01...., for each E I, l with or = i takes the uniformly distribution over [k]. 4e-1 (conditioned by then takes the uniform distribution over all the possible size-(f- 1) subsets of I \ {i). Therefore, if we write such an as f = o a=1()))))))))))))))))) (t), the probability that J JCI {i} is chosen as I, I can be computed as
(
1 |J
k
*(J] (O))
Fo 1(0--==17)
where to is the probability distribution defined just before
By applying this to (14) we have
Exjoy lo log (x()) Σ>
(L [ [((g)(()
iE!
Combining it with
13
we obtain
DKL(gr [pr)) = EH(a: -
-
E, E) (1(
(15)
iEl
ie!
Finally, for the remaining term, suffices to prove
Egyoyi (DkL(grir((11))) (1pr()((1))))))) s Exjogg
Σ H(gHr(+(z1)
SEINIV
This can be proven by using the positivity of entropy and modifying replaced by T. Thus, the proof is completed.
with q replaced by q.1 and I
25

---

Published in Transactions on Machine Learning Research (04/2026)
C.3 Proof of Proposition 5
Proof. Let Qr be the event under which we have Σ Pi(o;) > (1 - e) Ei 11:e1..
From S
we have
P(Q,) 2 1 exp(- 3c2 8
(a=alllil))) For any Ie <M,, under Ωe we have
N (i(
Σ NE[N]U
pi(xi)
Σ
1/
RE1
> (1 6)
Σn((os)
KET
2 (1 IIp:
Σ
(16)
NNIN
EE/
Now we want to estimate the ratio between Eier (a and EieNnyylleills inequality, we have
Let S = ::=)). From Hölder's
8 25 ( (1
1p:1 = S. S
Σ [i(x))
= = =S-1/a
(17)
#ES
#ES
By using this, we obtain
Eiel (
(k- (k-
-k- 1)51
EiENNTY/ lpill (N k 1)5-1/a
-
By applying this
16 under De, we have
pa > (1 (1-e- (k- (k-1)5//a 1)51/a
Σ
Σ 1lpil1
N - k +1
KE(NN
Moreover, by using 17 we have
3x2 N
322
P(O.) 2 1 - exp
1p: 11
2 exp
(-5 NS-1/00
which completes the proof of the proposition.
C.4 Proof of Proposition
Proof. Whether or not 99 holds only depends on the actual values of (x)) Thus, we can define the set
Z C SN such that g is satisfied if and only if (I:) Z% Now, let (2) E 2 From have (recall S = (S)
and ( we
P(i i i (x.) (2))
(112 ie -f. 12-1 i (I.) = (2)))
8=1
(or 1/a
CEINTIUL De(
t=1
Die )
1 1-c-
11-11
EvENNTVe 1
N-K+1
(k - 11)5///
-k &
Pi (I;, )
c
II
- k k+1
EiEAMT 11p.l1/
E=1
26

---

Published in Transactions on Machine Learning Research (04/2026)
We thus have Ze, which implics Z C Ze. Therefore,
P((z)) Z.) 2 (((x)))== E 23) 2 21-2000 exp (- 21 1/0
follows from Proposition 5 and the definition of Z/
C.5 Proof of Proposition
Proof. From the independence of yi and (iz) in Setting we have
P(Ji h ................................ Hh ik. (U37 Yi (2i, yim1) = P( = ......, = in) P(yvie) )== = (2se (2se) 15 = i1, -k.
= it)
P(1 i........ 11 = in) II P3le 24e)
P(i = h,..... . ix) II Pie (Zie) )
P l1
Now, we have
[p: l1
PÚi .,,..., jk ix)
Lie |1::1
by letting He = 1ogll4411 in Proposition
1
We obtain
10 through these two identities.
0
C.6 Proof of Theorem
g
Proof. Let e S By introducing extra variables, let define
PMaskGrr 2) (( in ... 1 i (x:)) (2)) I = i ...
Pmoment (i, = Pjt i1
(Wr))= = (zw) (Z.) = (Z))ENNN))))
where we suppose and y; are sampled independently. These define probability distributions over [N] SS From the independence and Proposition 7
we have
Pinoment = Pinoment (i, (Zie) P ((E))EEENN = (Zi)EENNI)A))
Pie (2 )
II P=(2,).
(18)
EiENNV 1p:11/ kE[N]U
(k-1)
and recall the set 2, defined in Proposition 6 definition of E,, we have (recall SS)
Suppose 0 c 1, + N-k-1
For z E Ze. From the
(M (ii
i1 ....... 2/ is
(2:) N (2) PP ((T)) (x)
(k 1)5t/a
-k k
Pie (Zs,)
1//a 2/a
N
N - k ++1
{=1 SE(NNNV /e_, 11p.11
1=1
(k 1)5t/a
-k
Pie (Ese 1//a Pie (Zie)
II p((#))
N k ++1
EIENNRAe 11p:1 VE(N]M
(k - 1)5t/a
Pmoment (i,z),
(19)
- +1
27

---

Published in Transactions on Machine Learning Research (04/2026)
where we have used (18) in the last equality. Let us next bound the total variation distance between PMaskIIT
and Pmoment- Let us denote (a)+ == max(0,a) a). In general, for probability distributions p and over the same finite set X, we have
E((l)) - a(x))++ - [(((z) - -((()))((z))+ = -((())-(())) =
-())== 0.
ZEX
ZEX
FEA
Thus, for the total variation distance, we have
1 Elr(1) - c(x)
dTv(P,
2 ZEX
Σ (p(x) - a(x))=++ + E(((x) - p(x))+
= =><<> (p(x) - ((x)))+
2
MEE
ZEX
NEX
By using this, we have
drv (mmoment: PMaskGIT)
Σ
(iMaskGrr(i,z) Pmoment (f) #ESM
1E[N]
Σ
Σ
(PMAskGIT(1, (1, Pmoment (2, 2))+ +
Σ
(fMackGrT(i, z) - Pmoment (1, 2))
&E|N] #E2,
BE[N]' ZEZ.
(1-c- (k- - 11)5///// -k
Σ Σ
-1)
Pmmoent (2, x) +
PMackGIT
N - k+1
VE NN ZEZ,
iE(N Z62.
(k- (k-1)/ + -1) ((( & 22).
(20)
N - 1
By applying Proposition G we obtain
(1----
(k (1))//
-b
drv ( S
k+1 11-e-N-k+1))
-1
+ exp
N k
For 5
1
we have
2k
8
2k5
(1 (1- = = =1
8 < +
1
1+
1
1
2k
n=0
Since (k 1) 24-1
S 2 S 1/2, we have
2k:5
k
2k8
(1-8) -)* 1
Σ
2k -
71
2h
2k8
(k 1)n-1
2kd
n-1
< + k:
2k
n!
2k:
-
2k8
80
2k-8
m
S 51+ + k
1)
2k
m!
21:
21 Ve
1+
+ 4k6.
2k
Thus, assuming +
(R-1)st/a
1
we have
N N-k+1
4k
(k - 1)51/a)
dry (Pmoment PMackGIT) 4hh
c
+ eexp
- 2N 1/a
(21)
N - -
28

---

Published in Transactions on Machine Learning Research (04/2026)
Since the total variation distance is always bounded by actually 21 holds without the posed assumptions
on c+ (k-1) N-
Note that, when N < 1/, the upper bound of drv (Pmoment: PMaskGTT) becomes larger than 1, which holds trivially true since drv is bounded above by 1. So, it suffices to prove the desired inequality with log
instead of log* under the assumption k/ Under this, by letting e
S1/a
log k2817
21
we have
drv Penoment PMaskGIT)
4 $1/0
N 1251/0
4k(k - h +
1)84/a
12
S 4kg
log
(22)
N
We need to make sure 1, but the above bound is valid even when >> 1, again because of the boundedness of drv. From N 2 h ///////0 2 k2, we also have
4k(k 1)8²/a
1 B-k 4S4//
4h2 51/a
1251/0
45
4
++
N
N
N
N
By applying this to
22
obtain
4 S
N
k281/a
dr (Fmoments PMaskGIT) S 4
log
+ +5V
3 N
12
k2S1/a
64
5+
log
N
3
k251/a
1221//a
I 5V
1
log
(23)
N
S1/0
Finally, by denoting I -.1,...,
in), we have
dTv(Pmoment, Paaiomment,MMskGrry
2 Σ
Σ
(PMustGrr(i (2i)ier) Pmoment (ś, (2:)ee))
M (EtleCrSSk
Σ
Σ
Σ (sberESN-t
(pMaskGrT (i, (zi)kez, (2z)ser) Pmoment (ł, (zi)ie,, (2))
* (si)serSS*
5 Σ Σ
Σ
|MMasKCTT( (zi)ier, (zz)iær) - Pmommnt (1, (zi)iers (2))aær))
* (S (S))
= dry (Pmoment PMaskGrT).
By combining this with
23 obtain the desired conclusion.
D Additional experimental details
D.1 Sampling schedule
Let D be the number of positions (so SD) and N be the number of total sampling steps. Let Jn C [D] be the (random) set of indices that are open after the n-th step, i.c., @ = Jo c J1 C ... C JN Jv [D]. Let In := -n \ Jn-1 for n = 1,..,, N, which is the (random) set of indices we unmask at the n-th step. In all the experiments, the cardinalities [Jal and [Ta (= Jn (J--1l) are predetermined by the unmasking size schedule such as:
Cosine schedule: [Jal round (c })))
29

---

Published in Transactions on Machine Learning Research (04/2026)
Uniform schedule:
| round(D .)).
Here, "round" means integer rounding. We adopted the cosine schedule for image and the uniform schedule for language.
Let us now consider specifically the n-th sampling step out of N steps. Given the demoising model we use the marginal distributions (PaJa-:)ieDD)yu.. mask in this step, determined by the unmasking size schedule. Then, for the sampler MaskGIT, we use OncRoundMassCIIT(PP... (PP) ))e(p)), 1: k. an) from Algorithm t determine In and ø/,- Here, on is the Gumbel temperature for the n-th step, which is scheduled as On a(1 n/N) following (Chang et al., 2022,, where is the temperature parameter of the method presented in the figures (e.g., Figure 3). We use the same (n, for Moment (Algorithm g and its variants given the parameter Nole that, in the final step (n N) of Moment or other semperature-sampiie
for this step. Let & be the number of indices to un-
methods, we omit the sampling temperature (or take ON + oo), in order that it corresponds to the final step of MasKGIT.
D.2 Partial caching
In the partial caching algorithm we described in Section 4.1 we have degree of freedom in dividing the selected index set into A and B (where we laveAUB and A B 0).
Let us explain our implementation. Let us use the notation of Jn and In introduced in the previous section. In the n-th step, suppose we decompose In into A, and Bu, where An is the set of indices unmasked in the intermediate step of partial caching. If we let Jn-1/2 == Jn-1 U An for 2 1, then adopted the canonical extension of the scheduler in Appendix D.1
Cosine schedule: |Jn-1/2l = round(cos DD1 --1/2)))
Uniform schedule: |J 1/2| round(D. --1/2).
Thus, the cardinality of A, was determined by |A,| |Jn-1/21 e| - |Ja- depending on the sampling schedule use. Since each of our sampling algorithm outputs ordering of masked indices (from which determine In), we simply determine An as the top-k of the ordered indices (k = |An| |Jn-1/21 [Jn-1)).
D.2.1 Computational efficiency of partial caching
The total cost of attention computation in partial caching is / times the original full attention computation (in terms of the notations in Section 4.1 However, it costs more computation in caching the vectors and other CPU/GPU operations. Indeed, while caching shows some performance gain with A6000 latency in the language experiment (Figure 5 Right), it vanishes when we use H100 GPU (Figure g Right). It would be caused by the faster attention computation of H100, reducing its weight among the overall computational cost and making other computational overhcad apparent.
D.3 Additional details on image modeling experiments
The MAGE ViT-B model (Li et al. 2023), which we used in the experiments, can be regarded as masked diffusion model on the space of pretrained VQGAN tokenizers (Lsssrr et al. 2021). It was trained on the ImageNet 256 x 256 dataset (Deng et al.) 2009). The codebook size is given by [S] = 1024, and the length of each token sequence (corresponding to single image) is D 256. Fach experiment with MAGE was conducted with single A6000 GPU with a minibatch size of 64. Based on 50000 unconditional images generated from each sampler, mcasured FID and IS against the ImageNet dataset by using torch-fidelity" following the description of the repository of MAGE
httpp::/gtthub.co//oohsse///ee
30

---

Published in Transactions on Machine Learning Research (04/2026)
MaskGIT
a=3.0
Random
120
Moment
a-6.0
120
Random+ Cache
Generative Perplexity
Temp
a-9.0
Generative Perplexity
Hybrid
100
Random
o=12.0
100
Hybrid+Cache
80
60
40
60
20
40
2
2
2
2
Steps
Sampling Time (seconds)
Figure 8: Additional experimental results. (Left) Generative Perplexity of various samplers with lempera-
ture sampling. (Right) Generative Perplexity of our proposed samplers against sampling time per batch H100 GPU.
D.4 Additional details on language modeling experiments
We used the SDTT model
DDechenaux & Gulcehre 2025 which is a masked diffusion model over a GPT-2
tokenizers (Radford et al. 2019). was trained on the OpenWebText datasct (Gokaslan & Cohen 2019)
The codebook size is |8| 50257 and the token sequence length is given by D 1024. Most experiments
were conducted on single H100 GPU, while the preliminary experiments, the ones in Appendix D.4.1 and the latency computation in Figure Right) were conducted on single A6000 GPU. Each plot was computed by 1024 samples generated with minibatch size of 16, using the following performance metrics.
Generative Perplexity. was measured against the GPT-2 large model (Radford et al. 2019 and averaged over 1024 samples. We used the implementation of Deschenaux & Gulcehre (2025).
Entropy. Following the existing work (Gat al. 2024 Zheng al. 2025 we measured the sentence-wise entropy for checking the diversity of generated sentences. In implementation (following the description of (Zheng et al. 2025 for sequence of tokens x (xI,..., ED), define the sentence entropy as
#liE [D] x; =}] log # [D] x =
Σ sESna
D
D
where S x is the set of tokens appearing in æ. Tts average over 1024 samples was plotted.
Figure Left) shows the omitted results for Generative Perplexity. While the methods with lower tempera- ture apparently attain better the generation quality, lowering the temperature extremely harms the diversity in reality (Figure Left).
D.4.1 Numerical precision
Zheng et al. 2025) pointed out that the sampling from masked diffusion models with low numerical precision (32-bit) can lead to errors in categorical sampling, showing lower (better) Generative Perplexity at the cost of lower (worse) Entropy. Because of this, they suggest using 64-bit computation for a fair evaluation of masked diffusion models.
However, they also note in Section J.2.2 of their paper, it is primarily because the lower numerical precision results in biased positional selection rather than the sampling distribution shift at each position. In our case, since fix the token positions in most samplers (except MaskGIT), we do not suffer from this problem though we use 32-bit precision. Indeed, Table 2 shows that, while the vanilla sampler in discrete diffusion suffer from the difference numerical precision, Fixed, which corresponds to Random in the main text, exhibits similar results in both precision settings. 4Tt was loaded by
student (loss="kld", round=7) in the repository htttps///iithbb htttshuub.cco//eae jttp://ccc/ccccc//eeteeeee
31

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: Comparison of different numerical precision in Vanilla and Fixed samplers. Vanilla is a standard sampler for discrete diffusion, where it independently determines whether not unmasking certain position.
Fixed is sampler that pre-determines the number of unmasked positions at each step, and it determines which positions to unmask uniformly at random. Both follows the uniform schedule (Appendix D.1 in expectation.
steps
32 steps
128 steps
Sampler
Precision
Gen. PPL
Entropy
Gen. PPL
Entropy
Gen. PPL
Entropy
32-bit
125.62
5.40
41.91
5.31
27.63
5.17
Vanilla
64-bit
137.95
5.42
46.57
5.35
33.80
5.28
Fixed
32-bit
131.01
5.41
45.10
5.32
33.15
5.26
(= Random)
64-bit
130.76
5.41
46.66
5.35
34.29
5.29
D.4.2 Hybrid algorithm
Let us explain the details of the Hybrid sampler in the language experiments, where we mergod the Halton and U-Moment samplers. Let consider the n-th sampling step out of N total steps and let Jn-1 be the set of indices already unmasked at this stage. The ordering of first. k = [Tl (where I Jn Jn-1 is from Appendix D.1 positions from each sampler is given follows:
Halton: We consider the one-dimensional Halton sequence of indices (with base 2), i.e., rearrange- ment of [D], and let i (41,..., ig) be its first k entries that are also in [D] Jn-1-
U-Moment: As in Algorithm 2 we define the ordering j = (J.,..., by
j argtop
log Σ P + t
where the exponent 1 + 1/a is determined by the temperature parameter a, and & is an independently sampled standard Gumbel noise for each j E [D] Jn-1.
We then merge i and j into as described in Section 4.2 parameter m m,, controlling how many indices we take from i, is scheduled as m, (ound((1nnNV]AA) Intuitively, it means that we basically start from Halton, whose exploration works at the initial stages, and gradually move to U-Moment, which conducts exploitation-basee index selection. In the implementation of Hybrid+Cache, we just apply the caching procedure as explained in Appendix D.2 to the above merged ordering k.
obtain an ordering for Hybrid, where merging
E
Further experimental results
E.1
Unconditional image generation with MAGE
we give the detailed FID values of Random, MaskGIT, Moment, Temp at each temperature in Table 3 Based on these numbers, we also present the relative differences between MaskGIT and the other samplers Table in the main body, which validates our theory on Moment approximating MaskGIT with the same temperature.
In addition to the raw data presented in Figure 3
E.2 Class-conditional image generation with MaskGIT-PyTorch
To confirm the approximation ability of Moment conditional image generation, we used the MaskGIT- PyTorrh (Besnier Chen 2023) model pretrained on ImageNet 256 x 256. We adopted the arccos sampling scheduler (see also Appendix D.1 for unmasking size, and the Gumbel temperature schedule followed the
32

---

Published in Transactions on Machine Learning Research (04/2026)
Table 3: ImageNet FIDs for various methods in unconditional experiment with MAGE.
MaskGIT
Moment
Temp
U-Moment
Steps
Random
3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
8
49.86
18.95
19.21
27.18
19.47
17.42
24.26
20.83
18.74
25.26
48.42
48.75
48.92
16
31.85
19.36
11.29
13.32
20.71
11.52
12.73
19.86
12.09
13.25
29.41
30.21
30.99
32
22.55
24.61
11.45
9.84
25.76
11.77
9.73
23.83
11.73
10.00
20.71
21.53
22.24
64
18.40
27.23
12.36
9.03
27.80
12.77
9.13
25.32
12.33
9.27
16.79
17.52
18.29
default implementation (same as that in Appendix D.1 The classifier-free guidance coefficient was set to 3.0 throughout the experiment.
Table 4 shows the FID values of several samplers and Table 5 shows their relative differences to MaskGIT samplers (corresponding to Table 3 and Table [ in the unconditional experiment, respectively). We can confirm again that Moment samplers consistently achieve the FID within 10% with MaskGIT of corresponding temperature. This approximation can also be visibly confirmed by the plot in Figure g
In the experiment with MaskGIT-PyTorch, while U-Moment improves when increasing steps, docs not nee- essarily out performs Random fewer steps, in contrast to the experiment with MAGE Table 3 We attribute this the of classifier-free guidance, which extrapolates logits between conditional and unconditional predictions; since this extrapolation does not yield normalized probabilities, the assumptions underlying Proposition 3 no longer hold in this setting.
Table 4: FIDs for various methods and steps in class-conditional experiment with MaskGIT- MaskGIT-PPTTonn Porch..
MaskGIT
Moment
Temp
U-Moment
Steps
Random
3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
4
30.11
14.34
19.17
23.83
12.85
17.92
23.01
17.65
22.82
25.97
31.96
31.89
31.65
8
12.92
6.96
6.60
7.67
7.26
6.41
7.21
7.02
8.20
9.89
13.53
13.54
13.40
16
7.85
8.65
6.79
6.08
9.43
7.39
6.12
7.24
6.18
6.41
7.90
7.84
7.87
6.37
10.28
6.72
11.15
7.09
8.43
6.45
5.91
6.32
6.26
6.26
Table 5: Mean relative difference (%) of ImgeNet FIDs against reference MaskGIT samplers across 4, 16, 32 steps in class-conditional experiment with JaskGrT-PyTorch Mean relative difference was computed in the same way as Table 1 temperature.
Bolded under 10%. Moment consistently approximates MaskGIT with the same
Ref. MaskGIT temperature
MaskGIT
Moment
Temp
Random
a 3.0
6.0
12.0
3.0
6.0
12.0
3.0
6.0
12.0
3.0
60.7
20.4
35.2
8.1
15.1
31.1
14.5
35.7
47.9
6.0
47.5
21.3
17.0
29.9
7.1
12.9
6.3
18.2
29.5
12.0
32.4
36.1
16.4
43.1
23.9
3.9
19.7
4.2
13.9
E.3 Language experiments with different models
To oblain further empirical evidence of the efficiency of our methods in Section we additionally lested them against a larger model [Dechenaux Gulcehre 2025 which we refer to as SDTTT-argee and a model with a modified architecture (Hayakawa et al.] [2025] which we refer Di4C). Each datapoint is basod on 256 generated sequences of 1024 tokens, with to 256 sampling steps. each of these two experiments, Hybrid used Gumbel coefficient of a 6.0 as in the main body.
33

---

Published in Transactions on Machine Learning Research (04/2026)
We also conduct experiments with the LLaDA-8B base model (Nie et al. 2025 to validate findings in even larger scale in Section F.3.3
E.3.1 SaTT-large model with Hybrid and Partial caching
In the experiment with larger model, used SarT-large, a pretrained transformer model scaled to 863M parameters, in contrast to SDTT with 169M parameters used in Section 5.2 Since SDTT-large is larger than the GPT-2 large model (774M parameters) in scale, we used Llama3-8B Dubey et al. 2024 for computing generative perplexities.
Figure 10 shows the results for SDTT-large. We can see a similar trade-off improvement of Hybrid over Random and consistent speedups of Hybrid and Random+Cache over Random. We also plotted a comparison of SDTT-large and SDTT with Random and Hybrid samplers in Figure 12 While the model difference becomes dominant as increase sampling steps, the use of Hybrid is effective for moth models, especially in the few-step regime.
E.3.2 Hybrid sampler with mixture modeling
We applied Hybrid to Di4C, model obtained from SDTT after two rounds of DiAC-fmetuing (Hayakawa et al. 2025 This Di4C finetuning includes a slight model architecture modification, which adds random latent variable to capture dimensional correlations between different positions/ tokens (caching not applica- ble because of this modified modeling). Figure 13 shows the comparison including SDTT results. Combining Hybrid and the Di4C finet fneeunnng (Di4C+Hybrid) sums up individual performance gains of Hybrid and Di4C from the original result of Random; this is especially remarkable in the few-step regime.
E.3.3 Experiments with an 8B-scale diffusion language model
In the experiment with LLaDA-SB (Nic al. 2025). we tested its unconditional generation of 256-token sequences, where we kept the Entropy and Generative Perplexity measured by Llama3-8B (Dubey et al. 20244) primary evaluation metrics. While large models like LLaDA conventionally evaluated by conditional- generation metrics, but those metrics usually just measure the alignment between query and answer, rather than the distributions of output. Our results show that, when low-temperature MaskGIT samplers are applied to LLaDA, sentence entropy in unconditional generation particularly captures the output's 's diversily loss, which is not measured in conditional tasks.
Figure 13 shows the results for LLaDA-8B. We can see that the MaskGIT's implicit low- low-tempeatture sampling be approximated by the moment sampler and leads to extremetly reduced Entropy (Figure [3 Left). In Figure Rightt, the trade-off of exploration-exploiee Hybrid sampler still interpolates between the two samplers and perform better than the vanilla sampler in terms of both trade-off and efficiency.
is shifted bit in favor of exploration, while the
E.4
Further ablation of Hybrid modeling
E.4.1
Asymmetry of Hybrid modeling
Hybrid asymmotric regarding the order of two mergod methods detailed in Appendix D.4.2 where Halton determines the indices to unmask and then U-Moment determines the rest of unmasking. Thus, we also consider s-Hybrid, where we just swap the order of two methods in each step, while the number of indices allocated to each method is unchanged. We show the results of s e-Hybrid (256 samples per datapoint) with SDTT among other samplers in Table 6 While s s-Hybrid shows slightly worse performance compared to Hybrid, the overall behavior similar, especially in the few-step regime. The potential source of the difference is that, since the Halton sequence is deterministic, its low-discrepancy effect can be reduced when we already select some positions. To mitigate this effect, adaptive methods such as determinantal point processes (uleezza al. 2012) as a replacement for Halton can be promising in future work.
5 "sdtt7-dsdc2.ccpt http:://aendb..rr//eeerr///////
34

---

Published in Transactions on Machine Learning Research (04/2026)
30
MaskGIT
0=3.0
Moment
a=6.0
Random
a=12.0
25
20
FID
15
10
10
15
20
25
30
Steps
Figure 9: FID against number of steps for various samplers with MaskGIT-PyTTrch. FID was computed with 50,000 (50 per class) class-conditional generation results.
160
160
Random
Random
Random+Cache
Random+Cachn
Generative Perplexity
Hybrid Hache
Generative Perplexity
Hybrid
120
120
Hybrid Cache
100
80
80
60
60
40
20
20
5.05
5.10
5.15
5.20
5.25
2
2
Entropy
Sampling Time (seconds)
Experiments with SDTT-large. Generative Perplexity was measured by Llama3-8B. (Left) Tradc-off between Generative Perplexity and Entropy, (Right) Generative Perplexity against Sampling Time
Figure 10:
on A6000.
160
Large Large+Randdm
140
Random
Generative Perplexity
Large Hybrid
120
Hybrid
100
80
2
2
2
2
Steps
Figure 11: Efficiency comparison SDTT-large and SDTT with Random and Hybrid samplers. 'Large+" means SDTT-large; otherwise it shows the results of SDTT. Generative Perplexity was measured by Llamaa- 8B for both models.
35

---

Published in Transactions on Machine Learning Research (04/2026)
Large+Random
140
Random
Generative Perplexity
Large Hybrid
120
Hybrid
100
80
60
40
20
2
2
2
2
2
Steps
Figure 12: Efficiency comparison of SDTT-large and SDTT with Random and Hybrid samplers. 'Large+'
means SDTT-large; otherwise it shows the results of SDTT. Generative Perplexity measured by Llama3- 8B for both models.
Di4C+ Random
Di4C+Random
120
120
Random
Random
Generative Perplexity
HloC+H
Generative Perplexity
Di4C+Hybrid
100
Hybrid
100
Hybrid
80
80
60
60
40
5.20
5.25
5.30
5.35
5.40
Sampling Time (seconds)
Entropy
Figure 13:
Efficiency comparison of Di4C and SDTT with Random and Hybrid samplers. Plots without "D4C+* show the results of SDTT. Generative Perplexity was measured by GPT-2 large for both models.
MaskGIT a=1.0
Random
Generative Perplexity
10a
Moment a=3.0
Random a=6.0
Generative Perplexity
10a
U-Moment (c=1.0)
U-Moment (x=33.0)
U-Moment a-12.0
Hybrid (a-1.0)
(.03
10a
10a
10a
1.0
1.5
2.0
2.5
3.0
3.5
4.5
5.0
4.4
4.5
4.6
4.7
4.8
4.9
5.0
Entropy
Entropy
Figure 14: Experiments with LLaDA-SB, Each plot was computed by 256 generated sentences with 256 tokens with number of steps in (8, 16, 32, 64, 128, 256). (Left) Generative Perplexitics (f) against Entropy for lemperature-based methods. We can again confirm that lemperature sampling (MaskGIT, Moment) sig- nificantly harms Entropy, while their unbiased counterpart, (U-Moment keep moderate entropy). (Right) Trade-off between Generative Perplexity and Entropy. Hybrid samplers show improvement Random similarly to Figure 5( while Halton is also competitive in this experimental regime.
36

---

Published in Transactions on Machine Learning Research (04/2026)
Table 6: Generative Perplexity and Entropy of s s-Hybrid and other samplers applied SDTT.
Steps
Metric
Method
8
16
32
64
128
256
Random
126.36
64.31
45.45
37.67
33.78
31.67
U-Moment
107.04
55.36
38.95
31.96
28.40
26.62
Gen. PPL
Hybrid 73.02
46.73
37.55
34.09
31.87
29.64
s s-Hybrid 73.46
48.76
40.25
35.30
31.97
30.84
Halton 55.09
42.53
37.41
35.76
34.32
33.13
Random 5.399
5.381
5.349
5.313
5.278
5.250
U-Moment 5.354
5.333
5.298
5.263
5.228
5.199
Entropy
Hybrid 5.399
5.369
5.337
5.299
5.273
5.234
8 s-Hybrid
5.382
5.344
5.318
5.278
5.229
5.198
Halton 5.426
5.385
5.351
5.320
5.293
5.263
E.4.2 Comparison of various schedules in Hybrid modeling
As ablation study regarding the merging schedule in the Hybrid sampler, compared the following schedulers in the SDTT experiment, where Thalton determines the ratio of indices allocated to Halton in the merging rule in Section D.4.2 in the n-th sampling step:
Linear: Thalton
Linear inv: Thalton = N
N
Cosine: Phalton cos
2N
Cosine inv: Thalton sin 2N
Sigmoid: Thalton = σ (12 2)
Sigmoid inv: Thalton = a (12 ( 2))
Half: Thalton
2
Here, Ite-z the sigmoid function. The actual number of indices for Halton in the experiment is computed as Whatton round @ rhaltou). Figure E.4.2 shows the results, where each datapoint was computed using 256 samples. We can see that our default scheduler (Linear) is overall a reasonable choice, but the optimal merging strategy appears to highly depend on the number of steps, and so ultimately we should adaptively change the merging schedule depending on the total sampling steps or equivalently the budget for generating samples.
Linear
100
Casine
Sigmold
90
Linear Inv
AR Perplexity
80
Sigmaid
Half
70
60
50
40
30
5.250
5.275
5.300
5.325
5.350
5.375
5.400 5.425
Sentence Entropy
Figure 15: Comparison of various merging schedulers applied to Hybrid sampler of the SDTT model.
37

---

Published in Transactions on Machine Learning Research (04/2026)
CONTARRYS
(a) NaskGIT, 6.0, 16 steps (FID: 11.64).
(b) Nonent, 6.0, 16 steps (FID: 11.69).
UOTTY GIT
(c) MaskGIT, 12.0, 64 steps (FID: 9.12).
(d) Moment, a 12.0, 64 steps (FID: 9.1.).
Figure 16: Comparison of generated samples by MAGE.
E.5 Generated samples
E.5.1 MAGE with MaskGIT and Moment samplers
We plot the generated samples of MAGE using these samplers with 16 steps and 64 steps in Figure 16 Note that, although they are generated from the same seed, samples do not correspond to each other between MaskGIT and Moment because of the different uses of random nunbers as in Figures 6 and
E.5.2 SDTT with MaskGIT and Hybrid samplers
To see the actual examples of entropy reduction and its mitigation by our methods, we put the 8-step generation results of MaskGIT and Hybrid with 6.0. We see the repetitive of the same characters in MaskGIT samplers, which harms the Entropy, while Hybrid results look more natural with diverse use of words, even with higher (worse) Generative Perplexities.
E.5.3 LLaDA-8B with MaskGIT and Hybrid samplers
Similarly to the previous section, we put 256-token unconditional generation results of LLaDA-8B in Fig- ures E.5.2 & E.5.2 We observe that the implicit low-entropy sampling reduces the diversity in the MaskGIT sampler, while our Hybrid counterparts does not suffer from it.
38

---

Published in Transactions on Machine Learning Research (04/2026)
</endoftext(>>i and for free speech. And to I. however, the other night, and to 'indicate' his com- ments, he argued that of the.e. did not say, and prove to the woman he did not speak the following, and that is not true.
I. he said that, that and that. and that I. and I and, that "I have. and, I. and, in the case of I. and. and. and her,.. and and. and her," and he reports that the public has been framed in manner that is reported to him.
I. he connected I. to the communication of the.e to the.e, and to the case of the.e. and. with the case of in the. and. I. and I. II, and, I., and,, that was the case of the.e. and, did not decide, and. and I. and. and. the case of T. the. and. I." he about a, that and. I, and I., and
Figure 17: Generated by SDTT with MaskGIT, 3.0 (Gen. PPL: 13.88, Entropy: 3.320).
</endoftext/> and get to show off your help. Don't 't worry about that side of the screen, the beta version gets released on June 5, 2015.
[U+2620]]:
What's available here is free download set up for variety of machines, naturally; specifically to vDB, MooseX, and the UE6 Server. It is prided itself the recent release of new GUI server, and the game is well over there, as there are yet to play loo.
Unofficial demos, which focuses on the development, does not release on CODAs as Saturday 3, 2015. The the current release is the server release. The New Descripten was created late and relcased 2012, and several developers have since been involved, working on various different games over a year or so.
In addition, the community has already begun beta testing on the server and client in an attempt to polish them and ability to show them through the client publicly. Once that can play out, potential releases kaizen out this summer, and beyond this fall, it is enough time for the human capital of release to remain in place. To ensure the delay, however, group of senior officials, Republicans, and
Generated by SDTT with Hybrid, a = 3.0 (Gen. PPL: 67.98, Entropy: 5.360). [U+26200 is actually a single Unicode character of crossbones.
Figure 18:
<lendoftext/> and unworthy of free speech. According to Judge Sitton, the FBI, had listened to respond to said individuals' their comments, and argued against it. But the judge,..t., refused to respond the motion and did not respond, on the grounds that it not confidentiality.
"The FBI does not, and doesn.. have a right stopping this," " he wrote. "This matter, therefore, can't be mentioned in the reported briefs."
However,..t. wrote that "there are number of reports that appear to have been contained in the case that turns up.
So, perhaps considered the possibility taking action on. or on that, and reviewing it. While this was in time with the.A. and.A. it occurred to me that Mr.. A. v. A. is an important part of the history of the FBI and, until now, not very much known. had talked to Mr. Sullivan, and in conversation with Barry, the AFR," he about year ago, indicated today, and today is about 10:30 to 11:
Figure 19: Generated by SDTT with MaskGIT, a 6.0 (Gen. PPT: 31.89, Entropy: 4.465).
39

---

Published in Transactions on Machine Learning Research (04/2026)
</endoftext|> and get show of your game. Don't worry about right side of the screen, the physical version gets certified April 5, 2015.
Now read:
We've read here about a game download set to ship perfect three months, offering various changes to game quality in Moosehead, by the time PMabouts. Microsoft has Guided on the most recent relcasc the new development game, and that product well too available, as we are excited test some.
The gameplay demos will be shown Steam, which does not release CMA1 as Saturday, October 12.
The the ship will be the first release of The Invaders, Raiden, the Ocean and Star Fox games, and will ship on the same date, but on a different console about a year after arrival.
So far, six games have already been confirmed for testing on this ship, and we will attempt to get begin to publicly ship with them during the summer (because the year can a difficult time to try them out). Please thank you for your game, as soon it arrives in our capital, bear in mind each other. To get the latest information and set your sights at least, you find
Figure 20: Generated by SDTT with Hybrid, 6.0 (Gen. PPL: 73.02, Entropy: 5.399).
32 steps: 0000 and 1000000
The the values, all the value differences closest to 10000.
Here is breakdown of the solution steps:
1. Calculate the absolute values of x and 10000: a. For 100000 < x < 10000: 100000 10000
100000 = 10000 For 10000 < x < 1000000: 100000 - 10000 < < 100000 = 10000
2. Calculate the differences: a. For 10000 < 1000000: 10000 = 10000 For 10000 < 1000000: 10000
-2000 2000
3. Determine the final
256 steps: 00000000000000000000000000000000
0U1TT111111100000000000000000000
00000111110000000000000000000000
00000000000000000000000000000000
Figure 21: Generated by LLaDA-8B with MaskGIT, 1.0, with 32-step and 256-step sampling.
40

---

Published in Transactions on Machine Learning Research (04/2026)
32 steps:
// retreive information
SaveString the string myQuery,
SaveString strDir mull;
1/
1/ Fetch the ODBCQL sub request
String myQuery: String QueryList; InitQueryCaret()
//---------- Check Initial Directory
1/ <-> check the root directory exists for time profile ATUMS <<<
*/
if ( CheckQuery aoyoo,, myQuery)
return(); //Query found
Treck(uery(sttDir,,myy myDir, myQuery.) ):
//refresh
writer. writer.write( www.e(tieee ('Treefpp3 characters] strDir
">' myquery. '<(n));
intNextDir : false;
Set procedure
256 steps: adiham in rhythm between 1891 and 1899.(1] Although the Allitt building was demolished in the 1970s, a reproduction of its function as The Southern Cross appears in Four Winds & Compass, the museum in Fylde Way as at 2013.[1] [1] As of 2013, the widow recovers [21][1][A] a replica first mate's uniform, worn by the 19th sourdough captain of the Southwind,[1] on display at Fylde Deco Hotel.[]] Arts
In 1898, the year of the appearance of the widow, the Needlecraft Hall was just built; some believe was one of the first bonds between classes represented, said to have been formed.[1] Marriage
Wilson at work, at the G gensoe textile factory in 1902
The wife initially apports, from the reproduction of seahorses shot, in order buy his own smithy. [1] Wilson, who had auctioned a number of goods based on the experience of Dutchmen, had his erstwhile
Figure 22: Generated by LLaDA-SB with Hybrid, 1.0, with 32-step and 256-step sampling. In the result of 32-step generation, we used the verbatim package of TATEX to properly display the spacings. [3 characters] are the Unicode characters [0+6399|[455002]1555522
41


Reviews:
[{"review_id": "DRcEII9tvK", "reviewer": "Reviewer_j9iu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies efficient sampling for masked diffusion models, with a particular focus on understanding and improving the widely used MaskGIT sampler. Specifically, the authors note that the original “sample-then-choose” procedure in the MaskGIT sampler is theoretically opaque and algorithmically restrictive. The authors then reformulates the sampler via a \"choose-then-sample\" (CTS) procedure, which resolves the issues and enables improvements regarding efficiency (This is the main contribution of this paper). Both theoretical analysis and empirical validations of the MaskGIT sampler based on the CTS formulation are provided to justify the findings of this paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The major claims of the paper are supported by a mixture of theoretical and empirical results. Theoretically, one of the main results has shown that MaskGIT is asymptotically equivalent to a moment sampler of CTS type, which is technically nontrivial and well motivated. Empirically, Empirically, experiments on both image and language generation are provided to justify the validity of the proposed method. Also, the reviewers personally finds the caching-based algorithm to be practically interesting and useful.", "audience_yn": "Yes", "audience_explanation": "To the best of the reviewer's knowledge, both theoretical and empirical studies of masked discrete diffusion models and their variants, such as diffusion language model, have become a popular research topic in modern generative AI community. Therefore, most TMLR audience will definitely find the results of this paper  to be interesting.", "requested_changes": "On the one hand, the reviewer thinks that quality of the manuscript could be possibly improved by making a clearer connection between the theoretical results proved in the paper and the experimental settings. For instance, would it be possible for the authors to provide a brief discussion on how the assumption $N \\gg k^2$ in Theorem 2 relates to the settings in practice? \n\nOn the other hand, the reviewer finds that authors might be missing a (long) list of related work on the theory of discrete diffusion models, which is highly related to the masked discrete diffusion model and MaskGIT sampler studied in this paper. An incomplete list of missing work is given by [1,2,3,4,5,6,7,8,9,10.11]. Also, there might be some intrinsic connection between the techniques used in these papers and the lemmas proved in the current manuscript. For instance, how does Proposition 4 relate to the Girsanov transform between two continuous-time markov chains studied in [2]? (It seems that Proposition 4 might be a discretized version of the Girsanov transform?) Therefore, the authors are strongly encouraged to include a detailed discussion on all related work from the theory community.", "broader_impact_concerns": "Since this paper is mainly about (masked) discrete diffusion model and generative AI, the reviewer doesn't think there is any related concern on the work's ethical implications.", "additional_comments": "Overall, the reviewer believes that this paper addresses an important problem, but it requires minor revision before being accepted by TMLR. The authors are strongly encouraged to carefully incorporate suggestions raised by all reviewers, discuss the issues in detail, and include all missing references listed below.\n\nReferences:\n\n[1] Zhang, Zikun, Zixiang Chen, and Quanquan Gu. \"Convergence of score-based discrete diffusion models: A discrete-time analysis.\" arXiv preprint arXiv:2410.02321 (2024).\n\n[2] Ren, Yinuo, Haoxuan Chen, Grant M. Rotskoff, and Lexing Ying. \"How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework.\" arXiv preprint arXiv:2410.03601 (2024).\n\n[3] Liang, Yuchen, Renxiang Huang, Lifeng Lai, Ness Shroff, and Yingbin Liang. \"Absorb and converge: Provable convergence guarantee for absorbing discrete diffusion models.\" arXiv preprint arXiv:2506.02318 (2025).\n\n[4] Huang, Xunpeng, Yingyu Lin, Nikki Lijing Kuang, Hanze Dong, Difan Zou, Yian Ma, and Tong Zhang. \"Almost Linear Convergence under Minimal Score Assumptions: Quantized Transition Diffusion.\" arXiv preprint arXiv:2505.21892 (2025).\n\n[5] Srikanth, Aadithya, Mudit Gaur, and Vaneet Aggarwal. \"Discrete State Diffusion Models: A Sample Complexity Perspective.\" arXiv preprint arXiv:2510.10854 (2025).\n\n[6] Wan, Zhengyan, Yidong Ouyang, Qiang Yao, Liyan Xie, Fang Fang, Hongyuan Zha, and Guang Cheng. \"Error Analysis of Discrete Flow with Generator Matching.\" arXiv preprint arXiv:2509.21906 (2025).\n\n[7] Liang, Yuchen, Yingbin Liang, Lifeng Lai, and Ness Shroff. \"Discrete Diffusion Models: Novel Analysis and New Sampler Guarantees.\" arXiv preprint arXiv:2509.16756 (2025).\n\n[8] Liang, Yuchen, Zhiheng Tan, Ness Shroff, and Yingbin Liang. \"Sharp Convergence Rates for Masked Diffusion Models.\" arXiv preprint arXiv:2602.22505 (2026).\n\n[9] Dmitriev, Daniil, Zhihan Huang, and Yuting Wei. \"Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees.\" arXiv preprint arXiv:2602.15008 (2026).\n\n[10] Conforti, Giovanni, Alain Durmus, Le-Tuyet-Nhi Pham, and Gael Raoul. \"Non-Asymptotic Convergence of Discrete Diffusion Models: Masked and Random Walk dynamics.\" arXiv preprint arXiv:2512.00580 (2025).\n\n[11] Huang, Xunpeng, Yingyu Lin, Nishant Jain, Kaibo Wang, Difan Zou, Yian Ma, and Tong Zhang. \"On the Complexity Theory of Masked Discrete Diffusion: From $\\mathrm {poly}(1/\\epsilon) $ to Nearly $\\epsilon $-Free.\" arXiv preprint arXiv:2509.21835 (2025).", "extra_scores": {}, "extra_text": {}}, {"review_id": "gEXrLaYc6f", "reviewer": "Reviewer_THAk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel MaskGIT sampler for masked diffusion models (MDM) with three main contributions: 1) they propose a theoretical analysis showing that MaskGIT is asymptotically equivalent to a choose-then-sample (CTS) procedure and instantiated as the moment sampler, which decouples index selection from token sampling; 2) they also show that MaskGIT implicitly performs low-temperature sampling and that removing temperature yields an unbiased CTS procedure in the one-by-one limit under accurate marginals. 3) They improve the CTS-based algorithm via a partial caching scheme and a hybrid index-selection strategy. The experiments on image and language generation validate the theoretical interpretation and improved quality/diversity–efficiency trade-offs.\n\nMain strengths: solid theoretical analysis with an explicit total-variation bound and strong empirical support\n\nMain limitations: it seems the experiment's gains are more significant in language than in images; the authors may want to provide more concrete analysis of such observations and possibly conduct the experiment on a larger scale.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. On the theoretical side, the authors provide a concrete approximation result with an explicit total variation bound, along with detailed derivations and proofs, and they also establish an unbiasedness result for CTS without temperature in the one-by-one setting. Empirically, the experiments are generally well-designed and provide meaningful evidence for the paper’s main contributions.\nThat said, the support is not equally strong across all claims. For instance, the caching results are encouraging, but the paper also makes clear that the gains are somewhat modest and may depend on the hardware setup. This does not hurt the central contributions of the paper, but they do slightly narrow their scope.", "audience_yn": "Yes", "audience_explanation": "Yes. I believe this paper would be of interest to readers working on diffusion models, particularly in discrete generative modeling. As masked language and image generative models such as LLaDA, Gemini Diffusion, and MMaDA have recently attracted growing attention, MaskGIT-style sampling has become an influential and widely adopted component in the design of new methods. This paper makes a useful contribution to that line of work by offering conceptual clarification, formal analysis, and empirical validation.", "requested_changes": "1. Clarify evaluation choices in the language setting: The paper uses GPT-2-based generative perplexity and entropy as main evaluation metrics, so a brief discussion of how these metrics relate to human-perceived quality and whether using GPT-2 as the evaluator could introduce bias into the results.\n\n2. More experiments at scale: It seems the experiment gains are more significant in language than in images; the authors may want to give more concrete analysis on such observations and possibly conduct experiments at larger scales to see if this trend holds.", "broader_impact_concerns": "The paper already includes a broader impact statement, and it addresses the main concerns at an appropriate level for this type of work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jLWNBEGR15", "reviewer": "Reviewer_Qcq3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\n\n    This paper aim to interpret heuristic MaskGIT sampler in theoretical sense. The authors found out the samplers' problem such as lack of diversity or performance degradation arise from biased sampling thorugh the view of CTS reformulation. Based on this interpretation, the authors propose several other samplers (such as hybrid) that can lead to performance gain, and also partial-caching strategy. While the interpretation of MaskGIT sampler is novel, performance of proposed samplers or effectiveness of caching strategy remains questionable in generalized (or bigger experiment settings). \n\n### Strengths\n**1. Clear separation of token sampling and index selection opens a useful algorithmic design space.**  \n\n     Beyond the asymptotic reinterpretation itself, the CTS view makes MaskGIT’s entangled sampler modular, which enables discussion of unbiased token sampling and new ordering strategies that are difficult to express in the original formulation.\n\n**2. The empirical section (Image domian) is well aligned with the paper’s main claims.**  \n\n     The ideas related to CTS and MaskGIT are well empirically supported in the Image domain.\n\n### Weakness\n**1. The writing is somewhat not clear, especially the notations.**\n\n**2. Bound in Theorem2 seems not tight or often trivial**\n\n    This paper provides a compelling theoretical reinterpretation of the previously heuristic MaskGIT sampler by connecting it to the Gumbel top-k trick. More precisely, Theorem 2 show that MaskGIT is asymptotically approximated by a CTS style Moment sampler with that induced bias, which offers a principled explanation for degradation at large step counts (or lack of diversity issue). \n\n    This paper provides a compelling theoretical reinterpretation of the previously heuristic MaskGIT sampler by connecting it to the Gumbel top-k trick. More precisely, Theorem 2 shows that MaskGIT is asymptotically approximated by a CTS style Moment sampler with an induced bias, offering a principled explanation for degradation at large step counts, including reduced diversity.\n\n    However, this approximation relies on a strong regime such as $N \\gg k^2$, and the bound can become loose or even trivial in realistic fast-sampling settings, which are directly tied to efficiency, or in large-vocabulary language tasks with large $|\\mathcal S|$. As a result, although the theory is insightful, a noticeable gap remains between the asymptotic analysis and the practical usefulness of the proposed samplers, especially in language settings.\n\n\n**3. Merging rule in Hybrid sampler seems non-trivial**\n\n    For Section 4.2, I agree with the motivation of balancing exploration and exploitation, and the KL-based discussion is intuitively appealing. However, the actual Hybrid construction remains highly heuristic. In particular, the merged ordering depends nontrivially on whether Halton or U-Moment is applied first, and the mixing schedule is also manually designed rather. This makes the practical success of Hybrid interesting, but its design is not yet fully principled. I recognize that authors put \"changed order\" sampler (s-Hybrid) at Appendix, However, still the rule itself seems heuristic.\n\n**4. Limitation of the partial-caching strategy**\n\n    The partial-caching method still relies on heuristic design choices, since the partition into A and B depends on the token-selection ordering and can therefore affect performance. Its latency benefit also appears hardware-dependent: the paper reports gains on A6000, but not on H100. In addition, although the appendix includes results beyond the smallest model, the overall study remains limited in scale, leaving it unclear whether the same benefit would hold reliably for substantially larger modern models.\n\n**5. Limitation of experimental scope**\n\n     In the image domain, the paper explicitly positions its experiments as validation of the CTS interpretation of MaskGIT rather than as a demonstration of practical improvement, while the practical gains are mainly argued in the language domain. However, the language experiments still have limited scope. The comparison set is centered on MaskGIT related samplers and the paper’s own CTS variants, without broader comparisons to more modern adaptive scheduling or parallel sampling methods. In addition, the evaluation relies mostly on the perplexity-entropy trade-off rather than downstream task quality, which leaves open questions about broader practical effectiveness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly yes. The paper presents a clear and persuasive theoretical reinterpretation of the previously heuristic MaskGIT sampler, and the image-domain experiments are well aligned with this goal. However, the strongest evidence is asymptotic rather than uniformly practical: the key approximation behind Theorem 2 relies on a strong regime, and the bound can become loose or even trivial in fast-sampling or large-vocabulary language settings. In addition, the language experiments are narrower in scope, focusing mostly on MaskGIT-related samplers and perplexity-entropy trade-offs rather than broader adaptive baselines or downstream task quality. As a result, the paper provides convincing evidence for its conceptual and asymptotic claims, but somewhat weaker evidence for broad practical superiority.", "audience_yn": "Yes", "audience_explanation": "I think yes. Researchers working on discrete diffusion would likely find this paper interesting, especially because it offers a rare theoretical perspective on the widely used but previously heuristic MaskGIT sampler. The CTS reformulation and the analysis of implicit temperature provide useful conceptual tools for understanding and designing various samplers. Even if some practical claims remain limited in scope, the paper is still likely to be of interest to readers who care about the theory and algorithms.", "requested_changes": "**1. Please strengthen the empirical justification of the theoretical regime behind Theorem 2.**\n\n    The paper argues that the approximation is meaningful when many positions remain masked, but the image experiments do not clearly show how the relevant quantities $N, k, |\\mathcal S|$ behave under each experimental setting, or whether the practical setups actually fall into the regime where the bound is informative. It would strengthen the paper to explicitly report these quantities, or otherwise clarify when the asymptotic approximation should be expected to be reliable in practice.\n\n**2. Please better justify the design of the Hybrid sampler (or change it)**\n\n    While the motivation of balancing exploration and exploitation is intuitive, the actual merging rule still appears fairly heuristic. In particular, the fact that the selected indices depend on the merge order between Halton and U-Moment suggests that the construction is nontrivial (as in weakness). It would help to provide either a stronger conceptual justification for the merge rule, or an alternative strategy.\n\n**3. Please provide stronger support for the linear exploration-to-exploitation schedule used in Hybrid.** \n\n     The current choice is intuitively reasonable, but it is unclear whether this schedule is empirically well justified. Could you show some empirical results in language experiments (e.g., the values of each KL component).", "broader_impact_concerns": "The authors wrote it, and I think it is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QMTqxAMV7I", "reviewer": "Reviewer_C5Hr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nWhile most masked diffusion models rely on the heuristic, confidence-based MaskGIT sampler, its theoretical foundations have remained unclear. This paper provides a theoretical analysis, proving that MaskGIT's sample-then-choose mechanism is asymptotically equivalent to a choose-then-sample (CTS) moment sampler. This equivalence reveals that MaskGIT inherently suffers from an implicit low-temperature bias. To resolve this, they propose the unbiased U-Moment sampler. Furthermore, leveraging the decoupled nature of the CTS framework, the paper introduces two advanced techniques: partial caching approximation for computational efficiency and a hybrid approach that balances exploration and exploitation.\n\n**Strengths**\n\n1. Clear theoretical demystification of MaskGIT: The paper rigorously proves the inherent low-temperature bias in the MaskGIT sampler. Many researchers in the field have intuitively suspected the flaws in MaskGIT's mechanism (i.e., sampling based on confidence and then reusing those confidence values to select unmasking locations), and this paper resolves these suspicions with formal proofs. I consider the mathematical demonstration, that MaskGIT's sample-then-choose mechanism is asymptotically equivalent to a choose-then-sample (CTS) moment sampler, to be the most substantial contribution of this work.\n2. Bridging theory and practical improvements: The authors go beyond theoretical analysis to propose practical improvements based on their findings, such as the U-Moment sampler, partial caching approximation, and the hybrid approach. While there are some weaknesses and open questions regarding these proposed methods (which will be discussed below), the attempt to connect theoretical foundations with practical engineering techniques makes this paper compelling.\n\n**Weaknesses**\n\n1. Regarding the claim: MaskGIT sampler often shows degraded performance when increasing sampling steps.\n    1. Missing U-Moment results in the image domain: The paper attributes MaskGIT's performance degradation at higher sampling steps to its implicit low-temperature bias, proposing the U-Moment sampler as a solution. However, Section 5.1 omits the performance evaluation of U-Moment for image generation. I recommend adding U-Moment experimental results in the image domain.\n    2. Contradictory trends in the language domain: The premise that MaskGIT sampler degrades with increasing steps contradicts recent empirical successes in the language domain. State-of-the-art masked diffusion language models (e.g., LLaDA, Dream) heavily rely on MaskGIT sampler or its threshold-based variants, and their performance actually improves as sampling steps increase. Furthermore, while this paper demonstrates U-Moment's effectiveness using the SDTT model, it is questionable whether SDTT fully represents general masked diffusion language models. Providing experimental results on representative models like LLaDA or Dream would significantly strengthen the paper's claims and generalize its contributions.\n2. Unclear practical advantages over existing CTS algorithms: As acknowledged in the paper, several choose-then-sample (CTS) algorithms based on metrics like entropy or margin already exist. While reinterpreting the MaskGIT sampler as a CTS Moment sampler is a meaningful theoretical contribution, it remains unclear what specific empirical or theoretical advantages the Moment sampler offers over these existing CTS alternatives. If the proposed Moment sampler does not demonstrate a clear comparative advantage against prior CTS baselines, the paper's practical impact may be limited to providing a post-hoc theoretical analysis of MaskGIT.\n3. Limited exclusivity of the proposed techniques: The proposed partial caching and hybrid approach appear broadly applicable to other existing CTS methods and even adaptable to the original MaskGIT sampler without significant modifications. Additionally, the paper lacks empirical comparison between the proposed partial caching and existing methods like dKV-cache. I encourage the authors to clarify why these techniques uniquely benefit from the Moment sampler and to provide a comparative analysis against existing caching baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As stated above, the core theoretical finding, that MaskGIT's sample-then-choose is asymptotically equivalent to a CTS moment sampler, is the most substantial contribution and is well supported by rigorous analysis. However, the subsequent practical contributions (U-Moment, partial caching, and hybrid approach) lack sufficient empirical evidence. The authors should either provide more comprehensive experiments across relevant domains and baselines, or explicitly discuss current limitations and position broader applications as future work.", "audience_yn": "Yes", "audience_explanation": "Although there are concerns regarding the proposed techniques, the core theoretical findings alone are valuable to the TMLR audience. The mathematical demystification of the MaskGIT sampler provides meaningful insights into the underlying mechanics of masked diffusion models. Even if this paper falls short of delivering fully validated practical methods, its theoretical framework will serve as a good starting point and a guiding foundation for researchers aiming to develop new samplers and advanced techniques in this field.", "requested_changes": "Please refer to the Weaknesses section above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)